Question on sending Apache Logs from Linux

[gimeb]

Initially I ran the following commands to send access and error logs
Ran first time:
[root@x3musaw80 log]# curl -s -O http://136.133.236.12/nagioslogserver/s ... p-linux.sh
[root@x3musaw80 log]# bash setup-linux.sh -s 136.133.236.12 -p 5544

Ran second time:
curl -s -O http://136.133.236.12/nagioslogserver/s ... p-linux.sh
bash setup-linux.sh -s 136.133.236.12 -p 5544 -f "/opt/IBM/HTTPServer85/logs/error_log" -t apache_error
bash setup-linux.sh -s 136.133.236.12 -p 5544 -f "/opt/IBM/HTTPServer85/logs/access_log" -t apache_access

1. In the above commands I was tagging the file as “apache_access”. Is there a way to change the tag to ibm-logs since default apache filter is not parsing them correctly?
2. Would execution of the first time commands and second time could cause any issues with sending the logs?

[rkennedy]

You should be able to look at the /etc/rsyslog.d/ directory for the corresponding config file that relates to error_log, or access_log and delete it. Then, rerun the bash command with the -t you're looking for.

1. When these are inputted into NLS, it will match the parameter with the -t to the 'program'. You'll need to setup a grok filter accordingly.
2. Nope - it will generate two different config files. The first one represents the syslog configuration file, and the remaining two would create their own. Here's an example -

Code: Select all

[root@suse11 rsyslog.d]# ls -l
total 12
-rw-r--r-- 1 root root 726 Mar 11 14:06 90-nagioslogserver_var_log_httpd_access_log.conf
-rw-r--r-- 1 root root 736 Mar 11 13:53 90-nagioslogserver_var_log_httpd_error_log.conf
-rw-r--r-- 1 root root 899 Mar 11 13:51 99-nagioslogserver.conf
[root@suse11 rsyslog.d]#

As you can see, each one has their own configuration file. When editing / removing any of them, make sure you run a service rsyslog restart.

[gimeb]

1. Therefore, please let me know if I can safely execute the following commands:

curl -s -O http://136.133.236.12/nagioslogserver/s ... p-linux.sh
bash setup-linux.sh -s 136.133.236.12 -p 5544 -f "/opt/IBM/HTTPServer85/logs/error_log" -t ibm-logs
bash setup-linux.sh -s 136.133.236.12 -p 5544 -f "/opt/IBM/HTTPServer85/logs/access_log" -t ibm-logs

2. Do I need to execute the curl again of bash commands be sufficient

3. What would happen with logs which was previously tags as following. Would the apache-error and apache-access tags be now overwritten with ibm-logs tag or I need to explicitly remove previous tags

bash setup-linux.sh -s 136.133.236.12 -p 5544 -f "/opt/IBM/HTTPServer85/logs/error_log" -t apache_error
bash setup-linux.sh -s 136.133.236.12 -p 5544 -f "/opt/IBM/HTTPServer85/logs/access_log" -t apache_access

[rkennedy]

1. Please remove the previous file created first, otherwise you will have duplicate logs going in with a different 'program'. It will be located in /etc/rsyslog.d/

2. Nope, the file is staying the same. All that is changing is the variables we are feeding it.

3. They will not be removed, they will have the old 'program'. Moving forward, your logs will now have 'ibm-logs' as the 'program'. The old ones will stay as they were.

Does that make sense?

[gimeb]

Let summarize my tasks. Please validate them for accuracy before I proceed.

1. Currently I have the following in the /etc/rsyslod.d.
[root@x3musaw80 rsyslog.d]# ls
90-nagioslogserver_opt_IBM_HTTPServer85_logs_access_log.conf 99-nagioslogserver.conf
90-nagioslogserver_opt_IBM_HTTPServer85_logs_error_log.conf
[root@x3musaw80 rsyslog.d]#

I should delete 90-nagioslogserver_opt_IBM_HTTPServer85_logs_access_log.conf and 90-nagioslogserver_opt_IBM_HTTPServer85_logs_error_log.conf. Please confirm

2. Run the following commands
bash setup-linux.sh -s 136.133.236.12 -p 5544 -f "/opt/IBM/HTTPServer85/logs/error_log" -t ibm-logs
bash setup-linux.sh -s 136.133.236.12 -p 5544 -f "/opt/IBM/HTTPServer85/logs/access_log" -t ibm-logs

3. run service rsyslog restart

Now I should see new logs getting into the NLS with ibm-logs tag

[rkennedy]

Yes, that is correct.

[gimeb]

I have executed all the outlined steps. However, I do not see the most recent log data. I have attached the LinuxLogsNotShow_Commands.docx which shows all the commands I have executed. As you can see that the most resent logs for 3/22 for some reason are not listed after execution of setup-linux.sh. I have also included the screen print of the dashboard (LinuxLogNoShow_Dashboard.docx). As you can see from the screens that the most current apache logs are not getting processed. The timestamp on the most recent message shows 02/Mar/2016:08:32:29. However, there are more recent data. Please advise

[hsmith]

Our theory here is that when you deleted the input, and added it back, the rsyslog daemon reparsed the log. How big is this log file? It might just be playing catch-up right now.

[gimeb]

I seems that syslogs are getting processed. See below
2016-03-22T21:13:45.228-07:00 136.133.231.211 syslog <77>Mar 22 21:01:01 x3musaw80 run-parts(/etc/cron.hourly)[2914 finished 0anacron

However, the most resent entry from of ibm-logs is still show the old data from 10/Mar/2016. See below
2016-03-22T11:19:43.542-07:00 136.133.231.211 ibm-logs <133>Mar 22 11:07:01 x3musaw80 ibm-logs: 136.133.96.66 - - [10/Mar/2016:09:59:01 -0800] "GET /favicon.ico HTTP/1.1" 404 19205

The actual logs are showing the data from 3/22. See snippet below.
136.133.96.66 - - [22/Mar/2016:21:57:10 -0700] "POST /MusaWeb/calculatePayment.action HTTP/1.1" 200 242958
136.133.96.66 - - [22/Mar/2016:21:57:10 -0700] "GET /MusaWeb/javascript/config_snapshot.js HTTP/1.1" 304 -
136.133.96.66 - - [22/Mar/2016:21:57:10 -0700] "GET /MusaWeb/musa2/js/analytics/omniture/s.code.dev.js HTTP/1.1" 304 -
136.133.96.66 - - [22/Mar/2016:21:57:10 -0700] "GET /MusaWeb/musa2/js/lib/ui/effects.blind.js HTTP/1.1" 304 -
136.133.96.66 - - [22/Mar/2016:21:57:10 -0700] "GET /MusaWeb/musa2/js/lib/jquery_plugins/musa_paymentestimator.js HTTP/1.1" 304 -
136.133.96.66 - - [22/Mar/2016:21:57:11 -0700] "GET /MusaWeb/vehicleETCRestful.action?code=M3S&year=2015&format=json HTTP/1.1" 200 175
136.133.96.66 - - [22/Mar/2016:21:57:11 -0700] "GET /webservices/mx/incentivesV2ByZipJSON/92620 HTTP/1.1" 200 120

Please advise on the next step

[chito]

Hello,

Does the file name contain date(YYYYMMDD)?

If so, I think you need to upgrade rsyslog v8.5 or later to use wild card.
http://www.slideshare.net/rainergerhard ... tor-imfile

Also, "$InputFileName" of rsyslog doesn't support wild card(*) in polling mode.
You have to use inotify mode. (You can not use script setup method.)