Page 1 of 3
Question on sending Apache Logs from Linux
Posted: Thu Mar 10, 2016 6:38 pm
by gimeb
Initially I ran the following commands to send access and error logs
Ran first time:
[root@x3musaw80 log]# curl -s -O
http://136.133.236.12/nagioslogserver/s ... p-linux.sh
[root@x3musaw80 log]# bash setup-linux.sh -s 136.133.236.12 -p 5544
Ran second time:
curl -s -O
http://136.133.236.12/nagioslogserver/s ... p-linux.sh
bash setup-linux.sh -s 136.133.236.12 -p 5544 -f "/opt/IBM/HTTPServer85/logs/error_log" -t apache_error
bash setup-linux.sh -s 136.133.236.12 -p 5544 -f "/opt/IBM/HTTPServer85/logs/access_log" -t apache_access
1. In the above commands I was tagging the file as “apache_access”. Is there a way to change the tag to ibm-logs since default apache filter is not parsing them correctly?
2. Would execution of the first time commands and second time could cause any issues with sending the logs?
Re: Question on sending Apache Logs from Linux
Posted: Fri Mar 11, 2016 3:17 pm
by rkennedy
You should be able to look at the /etc/rsyslog.d/ directory for the corresponding config file that relates to error_log, or access_log and delete it. Then, rerun the bash command with the -t you're looking for.
1. When these are inputted into NLS, it will match the parameter with the -t to the 'program'. You'll need to setup a grok filter accordingly.
2. Nope - it will generate two different config files. The first one represents the syslog configuration file, and the remaining two would create their own. Here's an example -
Code: Select all
[root@suse11 rsyslog.d]# ls -l
total 12
-rw-r--r-- 1 root root 726 Mar 11 14:06 90-nagioslogserver_var_log_httpd_access_log.conf
-rw-r--r-- 1 root root 736 Mar 11 13:53 90-nagioslogserver_var_log_httpd_error_log.conf
-rw-r--r-- 1 root root 899 Mar 11 13:51 99-nagioslogserver.conf
[root@suse11 rsyslog.d]#
As you can see, each one has their own configuration file. When editing / removing any of them, make sure you run a
service rsyslog restart.
Re: Question on sending Apache Logs from Linux
Posted: Mon Mar 14, 2016 1:55 pm
by gimeb
1. Therefore, please let me know if I can safely execute the following commands:
curl -s -O
http://136.133.236.12/nagioslogserver/s ... p-linux.sh
bash setup-linux.sh -s 136.133.236.12 -p 5544 -f "/opt/IBM/HTTPServer85/logs/error_log" -t ibm-logs
bash setup-linux.sh -s 136.133.236.12 -p 5544 -f "/opt/IBM/HTTPServer85/logs/access_log" -t ibm-logs
2. Do I need to execute the curl again of bash commands be sufficient
3. What would happen with logs which was previously tags as following. Would the apache-error and apache-access tags be now overwritten with ibm-logs tag or I need to explicitly remove previous tags
bash setup-linux.sh -s 136.133.236.12 -p 5544 -f "/opt/IBM/HTTPServer85/logs/error_log" -t apache_error
bash setup-linux.sh -s 136.133.236.12 -p 5544 -f "/opt/IBM/HTTPServer85/logs/access_log" -t apache_access
Re: Question on sending Apache Logs from Linux
Posted: Mon Mar 14, 2016 2:17 pm
by rkennedy
1. Please remove the previous file created first, otherwise you will have duplicate logs going in with a different 'program'. It will be located in /etc/rsyslog.d/
2. Nope, the file is staying the same. All that is changing is the variables we are feeding it.
3. They will not be removed, they will have the old 'program'. Moving forward, your logs will now have 'ibm-logs' as the 'program'. The old ones will stay as they were.
Does that make sense?
Re: Question on sending Apache Logs from Linux
Posted: Mon Mar 14, 2016 5:32 pm
by gimeb
Let summarize my tasks. Please validate them for accuracy before I proceed.
1. Currently I have the following in the /etc/rsyslod.d.
[root@x3musaw80 rsyslog.d]# ls
90-nagioslogserver_opt_IBM_HTTPServer85_logs_access_log.conf 99-nagioslogserver.conf
90-nagioslogserver_opt_IBM_HTTPServer85_logs_error_log.conf
[root@x3musaw80 rsyslog.d]#
I should delete 90-nagioslogserver_opt_IBM_HTTPServer85_logs_access_log.conf and 90-nagioslogserver_opt_IBM_HTTPServer85_logs_error_log.conf. Please confirm
2. Run the following commands
bash setup-linux.sh -s 136.133.236.12 -p 5544 -f "/opt/IBM/HTTPServer85/logs/error_log" -t ibm-logs
bash setup-linux.sh -s 136.133.236.12 -p 5544 -f "/opt/IBM/HTTPServer85/logs/access_log" -t ibm-logs
3. run service rsyslog restart
Now I should see new logs getting into the NLS with ibm-logs tag
Re: Question on sending Apache Logs from Linux
Posted: Tue Mar 15, 2016 12:40 pm
by rkennedy
Yes, that is correct.
Re: Question on sending Apache Logs from Linux
Posted: Tue Mar 22, 2016 4:03 pm
by gimeb
I have executed all the outlined steps. However, I do not see the most recent log data. I have attached the LinuxLogsNotShow_Commands.docx which shows all the commands I have executed. As you can see that the most resent logs for 3/22 for some reason are not listed after execution of setup-linux.sh. I have also included the screen print of the dashboard (LinuxLogNoShow_Dashboard.docx). As you can see from the screens that the most current apache logs are not getting processed. The timestamp on the most recent message shows 02/Mar/2016:08:32:29. However, there are more recent data. Please advise
Re: Question on sending Apache Logs from Linux
Posted: Tue Mar 22, 2016 4:27 pm
by hsmith
Our theory here is that when you deleted the input, and added it back, the rsyslog daemon reparsed the log. How big is this log file? It might just be playing catch-up right now.
Re: Question on sending Apache Logs from Linux
Posted: Wed Mar 23, 2016 12:18 am
by gimeb
I seems that syslogs are getting processed. See below
2016-03-22T21:13:45.228-07:00 136.133.231.211 syslog <77>Mar 22 21:01:01 x3musaw80 run-parts(/etc/cron.hourly)[2914 finished 0anacron
However, the most resent entry from of ibm-logs is still show the old data from 10/Mar/2016. See below
2016-03-22T11:19:43.542-07:00 136.133.231.211 ibm-logs <133>Mar 22 11:07:01 x3musaw80 ibm-logs: 136.133.96.66 - - [10/Mar/2016:09:59:01 -0800] "GET /favicon.ico HTTP/1.1" 404 19205
The actual logs are showing the data from 3/22. See snippet below.
136.133.96.66 - - [22/Mar/2016:21:57:10 -0700] "POST /MusaWeb/calculatePayment.action HTTP/1.1" 200 242958
136.133.96.66 - - [22/Mar/2016:21:57:10 -0700] "GET /MusaWeb/javascript/config_snapshot.js HTTP/1.1" 304 -
136.133.96.66 - - [22/Mar/2016:21:57:10 -0700] "GET /MusaWeb/musa2/js/analytics/omniture/s.code.dev.js HTTP/1.1" 304 -
136.133.96.66 - - [22/Mar/2016:21:57:10 -0700] "GET /MusaWeb/musa2/js/lib/ui/effects.blind.js HTTP/1.1" 304 -
136.133.96.66 - - [22/Mar/2016:21:57:10 -0700] "GET /MusaWeb/musa2/js/lib/jquery_plugins/musa_paymentestimator.js HTTP/1.1" 304 -
136.133.96.66 - - [22/Mar/2016:21:57:11 -0700] "GET /MusaWeb/vehicleETCRestful.action?code=M3S&year=2015&format=json HTTP/1.1" 200 175
136.133.96.66 - - [22/Mar/2016:21:57:11 -0700] "GET /webservices/mx/incentivesV2ByZipJSON/92620 HTTP/1.1" 200 120
Please advise on the next step
Re: Question on sending Apache Logs from Linux
Posted: Wed Mar 23, 2016 12:44 am
by chito
Hello,
Does the file name contain date(YYYYMMDD)?
If so, I think you need to upgrade rsyslog v8.5 or later to use wild card.
http://www.slideshare.net/rainergerhard ... tor-imfile
Also, "$InputFileName" of rsyslog doesn't support wild card(*) in polling mode.
You have to use inotify mode. (You can not use script setup method.)