Logging application logs with Nagios log server

[jshill103]

I have nagios log server running and am not great at dev ops type things.
We have some passenger-node logs inside of /var/logs/nginx and I wanted to get those sent over to my log server. I made and attempt at adding these logs to the config like this:

Code: Select all

passenger.*                     ~/var/log/nginx/passenger.log
access.*                        ~/var/log/nginx/access.log
nginx.*                         ~/var/log/nginx/error.log

But the logs are not coming into the log server. I am not sure what needs to be done. Any help would be much appreciated.

[rkennedy]

Could you post your entire rsyslog.conf and all of the files in your /etc/rsyslog.d/ folder? This should help to identify what's going on.

You might just be missing something like *.* @@ip.for.nls.here:5544 (replace ip.for.nls.here:5544 as needed)

[jshill103]

21-cloudinit.conf 99-nagioslogserver.conf are my only files in /etc/rsyslog.d/

and here is my config:

Code: Select all

#### RULES ####
###############

#
# First some standard log files.  Log by facility.
#
auth,authpriv.*                 /var/log/auth.log
*.*;auth,authpriv.none          -/var/log/syslog
cron.*                          /var/log/cron.log
daemon.*                        -/var/log/daemon.log
kern.*                          -/var/log/kern.log
lpr.*                           -/var/log/lpr.log
mail.*                          -/var/log/mail.log
user.*                          -/var/log/user.log
passenger.*                     ~/var/log/nginx/passenger.log
access.*                        ~/var/log/nginx/access.log
nginx.*                         ~/var/log/nginx/error.log

#
# Logging for the mail system.  Split it up so that
# it is easy to write scripts to parse these files.
#
mail.info                       -/var/log/mail.info
mail.warn                       -/var/log/mail.warn
mail.err                        /var/log/mail.err

#
# Logging for INN news system.
#
news.crit                       /var/log/news/news.crit
news.err                        /var/log/news/news.err
news.notice                     -/var/log/news/news.notice

#
# Some "catch-all" log files.
#
*.=debug;\
        auth,authpriv.none;\
        news.none;mail.none     -/var/log/debug
*.=info;*.=notice;*.=warn;\
        auth,authpriv.none;\
        cron,daemon.none;\
        mail,news.none          -/var/log/messages

#
# Emergencies are sent to everybody logged in.
#
*.emerg                         :omusrmsg:*

#
# I like to have messages displayed on the console, but only on a virtual
# console I usually leave idle.
#
#daemon,mail.*;\
#       news.=crit;news.=err;news.=notice;\
#       *.=debug;*.=info;\
#       *.=notice;*.=warn       /dev/tty8

# The named pipe /dev/xconsole is for the `xconsole' utility.  To use it,
# you must invoke `xconsole' with the `-file' option:
#
#    $ xconsole -file /dev/xconsole [...]
#
# NOTE: adjust the list below, or you'll go crazy if you have a reasonably
#      busy site..
#
daemon.*;mail.*;\
        news.err;\
        *.=debug;*.=info;\
        *.=notice;*.=warn       |/dev/xconsole

[rkennedy]

Could you also post 21-cloudinit.conf 99-nagioslogserver.conf? Every configuration file matters so we'll need to see how these are setup too.

[jshill103]

99:

Code: Select all

### Begin forwarding rule for Nagios Log Server                           NAGIOSLOGSERVER
$WorkDirectory /var/spool/rsyslog # Where spool files will live             NAGIOSLOGSERVER
$ActionQueueFileName nlsFwdRule0 # Unique name prefix for spool files     NAGIOSLOGSERVER
$ActionQueueMaxDiskSpace 1g   # 1GB space limit (use as much as possible) NAGIOSLOGSERVER
$ActionQueueSaveOnShutdown on # Save messages to disk on shutdown         NAGIOSLOGSERVER
$ActionQueueType LinkedList   # Use asynchronous processing               NAGIOSLOGSERVER
$ActionResumeRetryCount -1    # Infinite retries if host is down          NAGIOSLOGSERVER
# Remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional       NAGIOSLOGSERVER
*.* @@54.208.102.46:5544                                               # NAGIOSLOGSERVER
### End of Nagios Log Server forwarding rule                              NAGIOSLOGSERVER

and 21:

Code: Select all

# Log cloudinit generated log messages to file
:syslogtag, isequal, "[CLOUDINIT]" /var/log/cloud-init.log

# comment out the following line to allow CLOUDINIT messages through.
# Doing so means you'll also get CLOUDINIT messages in /var/log/syslog
& ~

[avandemore]

Did you restart rsyslog after adding the new directive?

[rkennedy]

In addition to what @avandemore mentioned, you may need to create another configuration file. For testing, try this -

create 98-nginx-passenger.conf in the /etc/rsyslog.d/ folder.

Code: Select all

    $ModLoad imfile
    $InputFilePollInterval 10
    $PrivDropToGroup adm
    $WorkDirectory /var/spool/rsyslog
     
    # Input for FILE_PATH
    $InputFileName /var/log/nginx/passenger.log
    $InputFileTag nginxpassenger:
    $InputFileStateFile nls-state-1 # Must be unique for each file being polled
    # Uncomment the folowing line to override the default severity for messages
    # from this file.
    #$InputFileSeverity info
    $InputFilePersistStateInterval 20000
    $InputRunFileMonitor
     
    # Forward to Nagios Logserver and then discard.
    if $programname == 'FILE_TAG' then @@54.208.102.46:5544  
    if $programname == 'FILE_TAG' then ~

This should forward your passenger logs. After adding it, make sure to restart rsyslogd. If that works, then you'll just want to create two more files for the error / access logs. The important variables to modify are going to be -

Code: Select all

    # Input for FILE_PATH
    $InputFileName /var/log/nginx/passenger.log
    $InputFileTag nginxpassenger:
    $InputFileStateFile nls-state-1 # Must be unique for each file being polled

[jshill103]

I did what rkennedy said to do in his comments. made a new config with your code, did the rsyslog restart and now my log server is taking a really long time to load the dashboard page. So I guess it is working based on the load speed, but I don't see any new log sources coming into the dashboard yet.

[rkennedy]

Has the initial surge stopped at all, or what are you seeing at this point?