Nagios Support Forum

Posted: **Thu May 21, 2015 4:30 pm**

I am trying to setup NLS to pickup mailman logs via rsyslog. I used the apache /etc/rsyslog.d/file as an example. HEre is what I tried.

Code: Select all

$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/lib/rsyslog

# Input for mailman_post
$InputFileName /var/log/mailman/post
$InputFileTag mailman_post:
$InputFileStateFile nls-state-var_log_mailman_post_log # Must be unique for each file being polled
# Uncomment the folowing line to override the default severity for messages
# from this file.
#$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor

# Forward to Nagios Log Server and then discard, otherwise these messages
# will end up in the syslog file (/var/log/messages) unless there are other
# overriding rules.
if $programname == 'mailman_post' then @@nagios-log1.HOSTNAMEHERE.com:5544
if $programname == 'mailman_post' then ~

After making the change I restarted rsyslog on that host and looked for some results on the NLS server. Unfortunately nothing came through. It continues to show the other logs (including apache) for that device but not seeing anything from this.

Am I going at this the wrong way?

Posted: **Thu May 21, 2015 4:33 pm**

Please post your /etc/rsyslog.conf file - you may be missing the following directive:

Code: Select all

$IncludeConfig /etc/rsyslog.d/*.conf

Otherwise, everything looks fine to me.

Additional Troubleshooting-
What version of rsyslog are you using?

Code: Select all

rsyslogd -v

Can you see the packets coming in from the Log Server side of things?

Code: Select all

tcpdump -n dst port 5544 and host 192.168.x.x

Posted: **Thu May 21, 2015 4:58 pm**

jolson wrote:Please post your /etc/rsyslog.conf file - you may be missing the following directive:
Code: Select all
$IncludeConfig /etc/rsyslog.d/*.conf
Otherwise, everything looks fine to me.

Yeap, I have it. KEep in mind I am already using this for apache in the same way and getting log output.

jolson wrote: Additional Troubleshooting-
What version of rsyslog are you using?
Code: Select all
rsyslogd -v
Can you see the packets coming in from the Log Server side of things?
Code: Select all
tcpdump -n dst port 5544 and host 192.168.x.x

Code: Select all

rsyslogd -v
rsyslogd 5.8.10, compiled with:
        FEATURE_REGEXP:                         Yes
        FEATURE_LARGEFILE:                      No
        GSSAPI Kerberos 5 support:              Yes
        FEATURE_DEBUG (debug build, slow code): No
        32bit Atomic operations supported:      Yes
        64bit Atomic operations supported:      Yes
        Runtime Instrumentation (slow code):    No

tcpdump does show data going out and as I said I can see the normal rsyslog plus apache entries as well. I just come up with nothing when I search for "mailman". I am also monitoring the mailman/post log and can see new log entries in the actual file. I would assume mailman_* would find the entry since that's the "program" name I used in the config file.

I'll poke around and see what I can find. Knowing the rsyslog config looks ok helps. I'll see if I can figure out a way to debug rsyslog for the mailmain_* portion.

Posted: **Fri May 22, 2015 9:20 am**

Everything looks proper and should work. Please ensure that the hostname you have in the rsyslog configuration is proper (nagios-log1.HOSTNAMEHERE.com). If that looks good, check /var/log/mailman/post to ensure that new logs are being generated - what happens if you manually add a log to the bottom? The file should be polled every 10 seconds and a log send when one is found.

I just come up with nothing when I search for "mailman".

Take a look at the logs in /var/log/mailman/post and search for a specific string located in one of those logs. Any luck?

You could try switching the output port to a port other than 5544. Try setting up a 'tcp' input on port 9001, for instance:

2015-05-22 09_19_40-Instance Configuration • Nagios Log Server.png

After setting that up, simply change your rsyslog config to send to that port:

Code: Select all

then @@nagios-log1.HOSTNAMEHERE.com:9001

Once this is complete, you could do a tcpdump on Nagios Log Server to verify logs are being received on the proper port:

Code: Select all

tcpdump -n dst port 9001

Posted: **Fri May 22, 2015 9:39 am**

I did not think of switching to another port like that to isolate it. I went with 9002.

Code: Select all

/etc/rsyslog.d/80-nagioslogserver_var_log_mailman_post

Code: Select all

$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/lib/rsyslog

# Input for mailman_post
$InputFileName /var/log/mailman/post
$InputFileTag mailman_post:
$InputFileStateFile nls-state-var_log_mailman_post_log # Must be unique for each file being polled
# Uncomment the folowing line to override the default severity for messages
# from this file.
#$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor

# Forward to Nagios Log Server and then discard, otherwise these messages
# will end up in the syslog file (/var/log/messages) unless there are other
# overriding rules.
if $programname == 'mailman_post' then @@nagios-log1.HOSTNAMEHERE:9002
if $programname == 'mailman_post' then ~

The config in NLS:

Code: Select all

tcp {
type => 'mailman'
port => 9002
}

Applied config on NLS.

Restarted rsyslog, then "tail -f /var/log/mailman/post &" and then "tcpdump -n dst port 9002" and watched. Sent test messages through mailman and I saw the /var/log/mailman/post logs entries pop up but nothing in tcpdump.

I double checked again that the other information configured under port 5544 was still going out via tcpdump and it was...

I am including my entire rsyslog.conf, maybe I am doing something else that causes this specific one to fail?

Code: Select all

#### MODULES ####

$ModLoad imuxsock.so    # provides support for local system logging (e.g. via logger command)
$ModLoad imklog.so      # provides kernel logging support (previously done by rklogd)
$ModLoad immark.so      # provides --MARK-- message capability


# Provides TCP syslog reception
#$ModLoad imtcp.so
#$InputTCPServerRun 514

#### GLOBAL DIRECTIVES ####

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$template MalformedMsgFormatter,"%timegenerated% %fromhost% %rawmsg:::drop-last-lf%\n"

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

## logs be world readable
$umask 0000
$FileCreateMode 0644
$DirCreateMode 0755


# log most critical and kernel messages to the console
# kern.debug is omitted so it can be used by iptables

*.crit;kern.info                /dev/console

# Exclude most other stuff from logging to the console.

mail,news,lpr,cron,user.none    /dev/console
local0,local1,local2.none       /dev/console
local3,local4,local5.none       /dev/console
local6,local7.none              /dev/console

# The standard stuff that goes to /var/adm/messages.
# RHEL / rsyslog adds an "authpriv" facility that by default
# is written to a "secure" file.

# The authpriv file has restricted access.
authpriv.*                                                      /var/log/secure
*.err;daemon,auth.notice;mail.crit;news,user.none;mark.debug    /var/log/messages

# This exclusion is arguably superfluous sinc admin2/loghost runs
# syslog-ng with its own config file.  We will need to be careful to
# update this to reflect new/changed loghost(s).  Below is a second
# stream sent to lab3 for testing
# central logging server
*.debug                         @10.2.XX.XX
& @10.1.XX.XX

# ---------------------
# send all 'auth' stuff to the authorization log

auth,mark.debug         /var/log/auth/debug

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# ---------------------
# local facilities were once assigned to different overlay groups
#
# local0        - currently unused (except for scd-dns*?)
# local1        - veriodns
# local2        - it provisioning
# local3        - it provisioning
# local4        - it provisioning
# local5        - currently unused (avoid this one - see below)
# local6        - Net-SNMP snmpd
# local7        - currently unused

# --------------------
# Local.Info

local0.info;mark.debug  /var/log/local0/info
local1.info;mark.debug  /var/log/local1/info
local2.info;mark.debug  /var/log/local2/info
local3.info;mark.debug  /var/log/local3/info
local4.info;mark.debug  /var/log/local4/info
local5.info;mark.debug  /var/log/local5/info
local6.info;mark.debug  /var/log/local6/info
local7.info;mark.debug  /var/log/local7/info

# --------------------
# Local.Debug

local0,mark.debug       /var/log/local0/debug
local1,mark.debug       /var/log/local1/debug
local2,mark.debug       /var/log/local2/debug
local3,mark.debug       /var/log/local3/debug
local4,mark.debug       /var/log/local4/debug
local6,mark.debug       /var/log/local6/debug
local5,mark.debug       /var/log/local5/debug
local7,mark.debug       /var/log/local7/debug

# ---------------------
# various daemons, such as sshd and ftpd (if ever running)
# includes log entries from alert.pl as well now, msw 10/2007
daemon,mark.debug       /var/log/daemon/debug

# ---------------------
# RHEL default rsyslog has
# mail.*                                                  -/var/log/maillog
# The - indicates that the file should not be sync'd after each message,
# saving time and disk I/O for files that get a lot of messages, but at
# the potential risk of message loss if the host crashes.
mail.notice;mark.debug  /var/log/mail/notice
mail,mark.debug         /var/log/mail/debug

# ---------------------
# General stuff
kern,mark.debug         /var/log/kern/debug
lpr,mark.debug          /var/log/lpd/debug
syslog,mark.debug       /var/log/syslogd/debug
user,mark.debug         /var/log/user/debug
uucp,mark.debug         /var/log/uucp/debug

# Catch cron info
cron.info;mark.debug    /var/log/cron/info

# Notify the operator, if he's logged in.
*.alert;kern.err;daemon.err;user.none           operator
*.alert;user.none                               root

# Send emergency messages to everyone that is currently logged in.
*.emerg                         *


# For 'user' facility messages that aren't quite emergencies but are
# important.
user.err                                        /dev/console
user.err                                        /var/adm/messages
user.alert                                      root, operator

# rsyslog foo:
# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/spppl/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList   # run asynchronously
#$ActionResumeRetryCount -1    # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

As a comparison here is my working apache_access version of this.

Code: Select all

$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/lib/rsyslog

# Input for apache_access
$InputFileName /var/log/httpd/access_log
$InputFileTag apache_access:
$InputFileStateFile nls-state-var_log_httpd_access_log # Must be unique for each file being polled
# Uncomment the folowing line to override the default severity for messages
# from this file.
#$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor

# Forward to Nagios Log Server and then discard, otherwise these messages
# will end up in the syslog file (/var/log/messages) unless there are other
# overriding rules.
if $programname == 'apache_access' then @@nagios-log1.HOSTNAMEHERE:5544
if $programname == 'apache_access' then ~

Posted: **Fri May 22, 2015 9:49 am**

Any chance that port 9002 is closed on the NLS firewall?

Posted: **Fri May 22, 2015 9:55 am**

jolson wrote:Any chance that port 9002 is closed on the NLS firewall?

To make sure it was not "firewall" issues I shutdown iptables (service iptables stop) and re-ran the above test. No joy ;(

With iptables back up I was able to telnet to port 9002 and get the

Code: Select all

Escape character is '^]'.

Tried the same for 9001 and 5544 and all responded the same.

Posted: **Fri May 22, 2015 11:00 am**

Yikes. Could you verify that Logstash is listening properly?

Code: Select all

netstat -na | egrep "9002|5544|9001"

If Logstash is listening, than the only reasonable assumption is that rsyslog isn't sending the logs over in the first place.

Let's try running a few rsyslog troubleshooting commands on your client:

Code: Select all

service rsyslog stop

Code: Select all

rsyslogd -f /etc/rsyslog.d/80-nagioslogserver_var_log_mailman_post -N1

Code: Select all

/sbin/rsyslogd -c5 -dn > /root/rsyslog.debug

Let the above run for a few minutes, and kill it with CTRL+C. The resulting debug log should give us a whole bunch of information to work with. Restart rsyslog when you're done with troubleshooting:

Code: Select all

service rsyslog start

I still see nothing wrong with your config file, but the above is worth checking to make sure.

Posted: **Fri May 22, 2015 12:32 pm**

Code: Select all

tcp        0      0 :::5544                     :::*                        LISTEN
tcp        0      0 :::9001                     :::*                        LISTEN
tcp        0      0 :::9002                     :::*                        LISTEN

Code: Select all

rsyslogd -f /etc/rsyslog.d/80-nagioslogserver_var_log_mailman_post -N1
rsyslogd: version 5.8.10, config validation run (level 1), master config /etc/rsyslog.d/80-nagioslogserver_var_log_mailman_post
rsyslogd: WARNING: rsyslogd is running in compatibility mode. Automatically generated config directives may interfer with your rsyslog.conf settings. We suggest upgrading your config and adding -c5 as the first rsyslogd option.
rsyslogd: Warning: backward compatibility layer added to following directive to rsyslog.conf: ModLoad immark
rsyslogd: Warning: backward compatibility layer added to following directive to rsyslog.conf: MarkMessagePeriod 1200
rsyslogd: Warning: backward compatibility layer added to following directive to rsyslog.conf: ModLoad imuxsock
rsyslogd: End of config validation run. Bye.

Attached the debug file tho I didn't immediately see anything wrong yet.

Posted: **Fri May 22, 2015 1:05 pm**

I looked over this post and noticed something - kicking myself for not picking up on this sooner.

Code: Select all

mv /etc/rsyslog.d/80-nagioslogserver_var_log_mailman_post /etc/rsyslog.d/80-nagioslogserver_var_log_mailman_post.conf

Code: Select all

service rsyslog restart

Hopefully it's just that simple.

Nagios Support Forum

rsyslog config for mailman test

rsyslog config for mailman test

Re: rsyslog config for mailman test

Re: rsyslog config for mailman test

Re: rsyslog config for mailman test

Re: rsyslog config for mailman test

Re: rsyslog config for mailman test

Re: rsyslog config for mailman test

Re: rsyslog config for mailman test

Re: rsyslog config for mailman test

Re: rsyslog config for mailman test