Question on sending Apache Logs from Linux

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
rkennedy
Posts: 6579
Joined: Mon Oct 05, 2015 11:45 am

Re: Question on sending Apache Logs from Linux

Post by rkennedy »

gimeb wrote:I seems that syslogs are getting processed. See below
2016-03-22T21:13:45.228-07:00 136.133.231.211 syslog <77>Mar 22 21:01:01 x3musaw80 run-parts(/etc/cron.hourly)[2914 finished 0anacron

However, the most resent entry from of ibm-logs is still show the old data from 10/Mar/2016. See below
2016-03-22T11:19:43.542-07:00 136.133.231.211 ibm-logs <133>Mar 22 11:07:01 x3musaw80 ibm-logs: 136.133.96.66 - - [10/Mar/2016:09:59:01 -0800] "GET /favicon.ico HTTP/1.1" 404 1920

Please advise on the next step
How large are your log files? Looking at the original file posted it looks like they were at the 2nd of march, your post here is now dated march 10th. As @hsmith mentioned, they may still be processing which will just take time.
The actual logs are showing the data from 3/22. See snippet below.
136.133.96.66 - - [22/Mar/2016:21:57:10 -0700] "POST /MusaWeb/calculatePayment.action HTTP/1.1" 200 242958
136.133.96.66 - - [22/Mar/2016:21:57:10 -0700] "GET /MusaWeb/javascript/config_snapshot.js HTTP/1.1" 304 -
136.133.96.66 - - [22/Mar/2016:21:57:10 -0700] "GET /MusaWeb/musa2/js/analytics/omniture/s.code.dev.js HTTP/1.1" 304 -
136.133.96.66 - - [22/Mar/2016:21:57:10 -0700] "GET /MusaWeb/musa2/js/lib/ui/effects.blind.js HTTP/1.1" 304 -
136.133.96.66 - - [22/Mar/2016:21:57:10 -0700] "GET /MusaWeb/musa2/js/lib/jquery_plugins/musa_paymentestimator.js HTTP/1.1" 304 -
136.133.96.66 - - [22/Mar/2016:21:57:11 -0700] "GET /MusaWeb/vehicleETCRestful.action?code=M3S&year=2015&format=json HTTP/1.1" 200 175
136.133.96.66 - - [22/Mar/2016:21:57:11 -0700] "GET /webservices/mx/incentivesV2ByZipJSON/92620 HTTP/1.1" 200 120
Just to clarify - are these logs appearing in NLS already, or only your access_log?
Former Nagios Employee
gimeb
Posts: 93
Joined: Mon Jan 25, 2016 1:56 pm

Re: Question on sending Apache Logs from Linux

Post by gimeb »

When I select 7 day range I see the following on the top line
2016-03-22T11:19:43.542-07:00 136.133.231.211 ibm-logs <133>Mar 22 11:07:01 x3musaw80 ibm-logs: 136.133.96.66 - - [10/Mar/2016:09:59:01 -0800] "GET /favicon.ico HTTP/1.1" 404 19205

When I select 30 days I actually see the logs are getting processes and the top line is now starts with the following:
2016-03-22T11:19:43.542-07:00 136.133.231.211 ibm-logs <133>Mar 22 11:07:01 x3musaw80 ibm-logs: 136.133.96.66 - - [02/Mar/2016:08:32:29 -0800] "POST /MusaWeb/calculatePayment.action HTTP/1.1" 200 245769

There is no data from 3/22 [22/Mar/2016:21:57:10 -0700]

All my logs are having the following format: access_20160323.log. The new logs is getting generated every day with new time stamp
Gary
rkennedy
Posts: 6579
Joined: Mon Oct 05, 2015 11:45 am

Re: Question on sending Apache Logs from Linux

Post by rkennedy »

chito wrote:Hello,

Does the file name contain date(YYYYMMDD)?

If so, I think you need to upgrade rsyslog v8.5 or later to use wild card.
http://www.slideshare.net/rainergerhard ... tor-imfile

Also, "$InputFileName" of rsyslog doesn't support wild card(*) in polling mode.
You have to use inotify mode. (You can not use script setup method.)
Please take a look at this response, you'll need to use a wildcard as the file names are always different.
Former Nagios Employee
gimeb
Posts: 93
Joined: Mon Jan 25, 2016 1:56 pm

Re: Question on sending Apache Logs from Linux

Post by gimeb »

Please let me know the following.
1. My understanding based on the out of the command I have executed (see LinuxLogsNotShow_Commands.docx in my prior communication) the version I have is 5.8.10. Please confirm
2. Please let me know how to upgrade to V8.5 and would it cause any issues with NLS and logs it is already collecting from other Linus machines.
3. You have indicated that "You have to use inotify mode". Please let me know where I need to do this changes and the exact syntax
Gary
User avatar
hsmith
Agent Smith
Posts: 3539
Joined: Thu Jul 30, 2015 11:09 am
Location: 127.0.0.1
Contact:

Re: Question on sending Apache Logs from Linux

Post by hsmith »

gimeb wrote:1. My understanding based on the out of the command I have executed (see LinuxLogsNotShow_Commands.docx in my prior communication) the version I have is 5.8.10. Please confirm
Looks correct.
gimeb wrote:2. Please let me know how to upgrade to V8.5 and would it cause any issues with NLS and logs it is already collecting from other Linus machines.
http://www.rsyslog.com/doc/v8-stable/in ... kages.html. I have not seen any issues with it, I have it running on one of my test systems.
gimeb wrote:3. You have indicated that "You have to use inotify mode". Please let me know where I need to do this changes and the exact syntax
Take a look at this: http://www.slideshare.net/rainergerhard ... tor-imfile
Former Nagios Employee.
me.
gimeb
Posts: 93
Joined: Mon Jan 25, 2016 1:56 pm

Re: Question on sending Apache Logs from Linux

Post by gimeb »

I have upgraded the rsyslog to V8. The output after the update - rsyslog.x86_64 0:8.17.0-1.el6

However, I have reviewed the link and I am still not clear of what I need to do to implement inotify mode and where. I am not Linux admin and any help would be appreciated.
Gary
User avatar
chito
Posts: 5
Joined: Thu Jun 02, 2011 9:38 pm

Re: Question on sending Apache Logs from Linux

Post by chito »

Hello,

I am not Linux admin and a support person. So, I cannot be responsible....
Here is my conf file in my test environment.
----------------------------------------------------------------------------
/etc/rsyslog.conf

Add the following line:

Code: Select all

global(workDirectory="/var/lib/rsyslog")
/etc/rsyslog.d/nagioslogserver_xxxxxxxxx.conf
* Create a new conf file.

Code: Select all

# Default Settings
$PrivDropToGroup adm

# Load Modules
module(load="imfile")

# rsyslog ruleset
ruleset(name="nagiosls") {
 action(type="omfwd"
           target="xxx.xxx.xxx.xxx" # NLS Server IP Address
           port="5544"
           protocol="tcp" )

}

# rsyslog Input Modules
input(type="imfile"
            tag="xxxx:"                  # TAG info
            file="/xxx/xxx/xxx.*"   # File Path (you can use wild card(*))
            persistStateInterval="20000" # Please tune the interval
            severity="notice"     # Please tune the level
            ruleset="nagiosls")
------------------------------------------------------------------
I'm not sure whether the above sample is proper in your environment.
Please read rsyslog documentation.
http://www.rsyslog.com/doc/v8-stable/co ... mfile.html

I hope everything works out...
User avatar
hsmith
Agent Smith
Posts: 3539
Joined: Thu Jul 30, 2015 11:09 am
Location: 127.0.0.1
Contact:

Re: Question on sending Apache Logs from Linux

Post by hsmith »

Thank you Chitose :)
Former Nagios Employee.
me.
gimeb
Posts: 93
Joined: Mon Jan 25, 2016 1:56 pm

Re: Question on sending Apache Logs from Linux

Post by gimeb »

I have conf files in many directories and I do not know which ones I need to modify. I have attached them.

The /etc/rsyslog.d contains the files in attached 90-nagioslogserver_opt_IBM_HTTPServer85_logs_access.zip

The /etc/ contains rsyslog.conf and this file has the following line " #$WorkDirectory /var/lib/rsyslog # where to place spool files". Should I just un-comment it. However, the one you have provided is "global(workDirectory="/var/lib/rsyslog")"

The /var/lib/rsyslog contains files attached in nls-state-opt_IBM_HTTPServer85_logs_access_20160229.zip

Please let me know how to proceed with the changes
You do not have the required permissions to view the files attached to this post.
Gary
jolson
Attack Rabbit
Posts: 2560
Joined: Thu Feb 12, 2015 12:40 pm

Re: Question on sending Apache Logs from Linux

Post by jolson »

I have PM'd you, please check your inbox and respond when you have the time - thank you!
Twits Blog
Show me a man who lives alone and has a perpetually clean kitchen, and 8 times out of 9 I'll show you a man with detestable spiritual qualities.
Locked