Reading /var/log/messages with NLS

[polarbear1]

Hi Guys,

Probably a stupid question but my Google-fu is failing me. I am interested in picking up /var/log/messages with NLS so I did what I'd do with any other log...

Code: Select all

bash setup-linux.sh -s nagiosbox -p 5544 -f /var/log/messages -t messages

and then I spammed some unique test words using logger while watching the dashboard - and nothing.

So - Can I do it? How do I get about doing it?

Thanks.

[jolson]

According to what you've posted above, this should work fine. Are you receiving any other logs on NLS from the server you're using?

It is important to keep in mind that only logs with a severity of info or higher will get parsed by rsyslog properly - if logger is not adding a severity, they will not send.

Could you post your rsyslog configurations?

Code: Select all

cat /etc/rsyslog.conf
cat /etc/rsyslog.d/*.conf

[polarbear1]

I partially eat my words - 6 out of my 8 boxes are working, but 2 are not. The configurations are identical, or at least they should be. A diff on the 2 files below between a working and a non-working server confirms the files are identical, so something else is probably the culprit here.

I am not monitoring any other file on the 2 boxes that aren't working for me. The ones that are working, are also monitoring other files.

/etc/rsyslog.d/*.conf

Code: Select all

$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/lib/rsyslog

# Input for messages
$InputFileName /var/log/messages
$InputFileTag messages:
$InputFileStateFile nls-state-var_log_messages # Must be unique for each file being polled
# Uncomment the folowing line to override the default severity for messages
# from this file.
#$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor

# Forward to Nagios Log Server and then discard, otherwise these messages
# will end up in the syslog file (/var/log/messages) unless there are other
# overriding rules.
if $programname == 'messages' then @@schpnag1:5544
if $programname == 'messages' then ~

rsyslog.conf

Code: Select all

# rsyslog v5 configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imklog   # provides kernel logging support (previously done by rklogd)
#$ModLoad immark  # provides --MARK-- message capability

# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514

# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514


#### GLOBAL DIRECTIVES ####

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf


#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog


# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 *

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log


# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/lib/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList   # run asynchronously
#$ActionResumeRetryCount -1    # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###

# A template to for higher precision timestamps + severity logging
$template SpiceTmpl,"%TIMESTAMP%.%TIMESTAMP:::date-subseconds% %syslogtag% %syslogseverity-text%:%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"

:programname, startswith, "spice-vdagent"	/var/log/spice-vdagent.log;SpiceTmpl

[jolson]

I am not monitoring any other file on the 2 boxes that aren't working for me

Is SELinux running on either of the two non-working boxes? It would also be advisable to ensure connectivity to the NLS box. From your client machines:

Code: Select all

telnet schpnag1 5544

Code: Select all

nmap -p 5544 schpnag1

Once you've verified that you can reach NLS, let's verify that you're receiving data from the client boxes. From your client machines:

Code: Select all

tcpdump -n dst port 5544

Code: Select all

service rsyslog restart

Does anything show up in the tcpdump? If so, let's take it one step further. From your NLS box:

Code: Select all

tcpdump -n dst port 5544 and host 192.168.x.x

Where 192.168.x.x is one of your client machines. Are the logs coming in?

Once you've tested the above, let me know.

[polarbear1]

None (working and non-working) of the servers are aware of the schpnag1 hostname, so doing it by IP.

192.168.1.175 - schpnag1 -- NLS server.
192.168.1.46 - schtwb06 -- a server that isn't reporting.

selinux:

• [root@schtwb06 ~]# sestatus
  SELinux status: disabled

telnet:

Code: Select all

[root@schtwb06 ~]# telnet 192.168.1.175 5544
Trying 192.168.1.175...
Connected to 192.168.1.175.
Escape character is '^]'.

nmap [xxx = edited for privacy]

Code: Select all

Starting Nmap 5.51 ( http://nmap.org ) at 2015-05-26 14:39 CDT
Nmap scan report for schpnag1.xxx.com (192.168.1.175)
Host is up (0.00018s latency).
PORT     STATE SERVICE
5544/tcp open  unknown
MAC Address: xxx

Nmap done: 1 IP address (1 host up) scanned in 0.12 seconds

tcpdump tests aren't picking anything up between the 2 servers. To verify, performed the same tcpdump test on a known working server and there is clear activity being detected.

[jolson]

The question then is why rsyslog isn't working with the configuration you have specified. Can you show me the contents of your rsyslog.d directory on the client server?

Code: Select all

ls -l /etc/rsyslog.d/

[polarbear1]

On the client end on these 2 boxes, there's just the 1 file as mentioned in the earlier reply...

Code: Select all

[root@schtwb06 ~]# ls /etc/rsyslog.d/
90-nagioslogserver_var_log_messages.conf

On a related note, the NLS box is --

Code: Select all

[root@schpnag1 ~]# ls /etc/rsyslog.d/
nagioslogserver.conf

which is....

Code: Select all

[root@schpnag1 ~]# more /etc/rsyslog.d/nagioslogserver.conf
# ### begin forwarding rule ###
#
# NAGIOS LOG SERVER
#
$WorkDirectory /var/lib/rsyslog    # where to place spool files
$ActionQueueFileName fwdRule1      # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g        # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on      # save messages to disk on shutdown
$ActionQueueType LinkedList        # run asynchronously
$ActionResumeRetryCount -1         # infinite retries if host is down
*.* @@localhost:5544
#
# ### end of the forwarding rule ###

[jolson]

Try adding another rsyslog configuration to the mix. Call it 99-nagioslogserver.conf.

Code: Select all

touch /etc/rsyslog.d/99-nagioslogserver.conf

Add the following to it:

Code: Select all

### Begin forwarding rule for Nagios Log Server                           NAGIOSLOGSERVER
$WorkDirectory /var/lib/rsyslog # Where spool files will live             NAGIOSLOGSERVER
$ActionQueueFileName nlsFwdRule0 # Unique name prefix for spool files     NAGIOSLOGSERVER
$ActionQueueMaxDiskSpace 1g   # 1GB space limit (use as much as possible) NAGIOSLOGSERVER
$ActionQueueSaveOnShutdown on # Save messages to disk on shutdown         NAGIOSLOGSERVER
$ActionQueueType LinkedList   # Use asynchronous processing               NAGIOSLOGSERVER
$ActionResumeRetryCount -1    # Infinite retries if host is down          NAGIOSLOGSERVER
# Remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional       NAGIOSLOGSERVER
*.* @@192.168.x.x:5544                                               # NAGIOSLOGSERVER
### End of Nagios Log Server forwarding rule                              NAGIOSLOGSERVER

Replace 192.168.x.x with your NLS IP address. After the above is done, restart rsyslog:

Code: Select all

service rsyslog restart

By default, the above config will parse through console, messages, secure, maillog, cron, spooler, and boot.log and send those messages over to NLS. Let me know if you have better luck when you don't specify the /var/log/messages file directly.

[polarbear1]

FIXED - So you were right earlier. Basically it had to do with resolving the hosts.

Solution 1 - Add the NLS server to the hosts file

Solution 2 - Run the linux-setup.sh script specifying the IP of the NLS box instead of the host name.

Thanks.

[jolson]

No problem - glad we got this worked out. I'll lock the thread as resolved. Feel free to open another if you have more questions or issues. Thanks!