Page 1 of 1
Logging application logs with Nagios log server
Posted: Wed Oct 05, 2016 11:20 am
by jshill103
I have nagios log server running and am not great at dev ops type things.
We have some passenger-node logs inside of /var/logs/nginx and I wanted to get those sent over to my log server. I made and attempt at adding these logs to the config like this:
Code: Select all
passenger.* ~/var/log/nginx/passenger.log
access.* ~/var/log/nginx/access.log
nginx.* ~/var/log/nginx/error.log
But the logs are not coming into the log server. I am not sure what needs to be done. Any help would be much appreciated.
Re: Logging application logs with Nagios log server
Posted: Wed Oct 05, 2016 4:19 pm
by rkennedy
Could you post your entire rsyslog.conf and all of the files in your /etc/rsyslog.d/ folder? This should help to identify what's going on.
You might just be missing something like *.* @@ip.for.nls.here:5544 (replace ip.for.nls.here:5544 as needed)
Re: Logging application logs with Nagios log server
Posted: Wed Oct 05, 2016 5:39 pm
by jshill103
21-cloudinit.conf 99-nagioslogserver.conf are my only files in /etc/rsyslog.d/
and here is my config:
Code: Select all
#### RULES ####
###############
#
# First some standard log files. Log by facility.
#
auth,authpriv.* /var/log/auth.log
*.*;auth,authpriv.none -/var/log/syslog
cron.* /var/log/cron.log
daemon.* -/var/log/daemon.log
kern.* -/var/log/kern.log
lpr.* -/var/log/lpr.log
mail.* -/var/log/mail.log
user.* -/var/log/user.log
passenger.* ~/var/log/nginx/passenger.log
access.* ~/var/log/nginx/access.log
nginx.* ~/var/log/nginx/error.log
#
# Logging for the mail system. Split it up so that
# it is easy to write scripts to parse these files.
#
mail.info -/var/log/mail.info
mail.warn -/var/log/mail.warn
mail.err /var/log/mail.err
#
# Logging for INN news system.
#
news.crit /var/log/news/news.crit
news.err /var/log/news/news.err
news.notice -/var/log/news/news.notice
#
# Some "catch-all" log files.
#
*.=debug;\
auth,authpriv.none;\
news.none;mail.none -/var/log/debug
*.=info;*.=notice;*.=warn;\
auth,authpriv.none;\
cron,daemon.none;\
mail,news.none -/var/log/messages
#
# Emergencies are sent to everybody logged in.
#
*.emerg :omusrmsg:*
#
# I like to have messages displayed on the console, but only on a virtual
# console I usually leave idle.
#
#daemon,mail.*;\
# news.=crit;news.=err;news.=notice;\
# *.=debug;*.=info;\
# *.=notice;*.=warn /dev/tty8
# The named pipe /dev/xconsole is for the `xconsole' utility. To use it,
# you must invoke `xconsole' with the `-file' option:
#
# $ xconsole -file /dev/xconsole [...]
#
# NOTE: adjust the list below, or you'll go crazy if you have a reasonably
# busy site..
#
daemon.*;mail.*;\
news.err;\
*.=debug;*.=info;\
*.=notice;*.=warn |/dev/xconsole
Re: Logging application logs with Nagios log server
Posted: Thu Oct 06, 2016 3:21 pm
by rkennedy
Could you also post 21-cloudinit.conf 99-nagioslogserver.conf? Every configuration file matters so we'll need to see how these are setup too.
Re: Logging application logs with Nagios log server
Posted: Thu Oct 06, 2016 3:25 pm
by jshill103
99:
Code: Select all
### Begin forwarding rule for Nagios Log Server NAGIOSLOGSERVER
$WorkDirectory /var/spool/rsyslog # Where spool files will live NAGIOSLOGSERVER
$ActionQueueFileName nlsFwdRule0 # Unique name prefix for spool files NAGIOSLOGSERVER
$ActionQueueMaxDiskSpace 1g # 1GB space limit (use as much as possible) NAGIOSLOGSERVER
$ActionQueueSaveOnShutdown on # Save messages to disk on shutdown NAGIOSLOGSERVER
$ActionQueueType LinkedList # Use asynchronous processing NAGIOSLOGSERVER
$ActionResumeRetryCount -1 # Infinite retries if host is down NAGIOSLOGSERVER
# Remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional NAGIOSLOGSERVER
*.* @@54.208.102.46:5544 # NAGIOSLOGSERVER
### End of Nagios Log Server forwarding rule NAGIOSLOGSERVER
and 21:
Code: Select all
# Log cloudinit generated log messages to file
:syslogtag, isequal, "[CLOUDINIT]" /var/log/cloud-init.log
# comment out the following line to allow CLOUDINIT messages through.
# Doing so means you'll also get CLOUDINIT messages in /var/log/syslog
& ~
Re: Logging application logs with Nagios log server
Posted: Thu Oct 06, 2016 4:40 pm
by avandemore
Did you restart rsyslog after adding the new directive?
Re: Logging application logs with Nagios log server
Posted: Thu Oct 06, 2016 4:49 pm
by rkennedy
In addition to what @avandemore mentioned, you may need to create another configuration file. For testing, try this -
create 98-nginx-passenger.conf in the /etc/rsyslog.d/ folder.
Code: Select all
$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/spool/rsyslog
# Input for FILE_PATH
$InputFileName /var/log/nginx/passenger.log
$InputFileTag nginxpassenger:
$InputFileStateFile nls-state-1 # Must be unique for each file being polled
# Uncomment the folowing line to override the default severity for messages
# from this file.
#$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor
# Forward to Nagios Logserver and then discard.
if $programname == 'FILE_TAG' then @@54.208.102.46:5544
if $programname == 'FILE_TAG' then ~
This should forward your passenger logs. After adding it, make sure to restart rsyslogd. If that works, then you'll just want to create two more files for the error / access logs. The important variables to modify are going to be -
Code: Select all
# Input for FILE_PATH
$InputFileName /var/log/nginx/passenger.log
$InputFileTag nginxpassenger:
$InputFileStateFile nls-state-1 # Must be unique for each file being polled
Re: Logging application logs with Nagios log server
Posted: Thu Oct 06, 2016 5:59 pm
by jshill103
I did what rkennedy said to do in his comments. made a new config with your code, did the rsyslog restart and now my log server is taking a really long time to load the dashboard page. So I guess it is working based on the load speed, but I don't see any new log sources coming into the dashboard yet.
Re: Logging application logs with Nagios log server
Posted: Fri Oct 07, 2016 9:30 am
by rkennedy
Has the initial surge stopped at all, or what are you seeing at this point?