Page 2 of 3

Re: Linux host not sending logs

Posted: Mon Mar 23, 2015 11:52 am
by Gabriel_Barba
Ok,

The server doesn't have rsyslogd but rsyslog as a service, here's the output:

Code: Select all

[root@fpS241ch rsyslog.d]#     service rsyslogd status
rsyslogd: unrecognized service
[root@fpS241ch rsyslog.d]#     service rsyslog status
rsyslogd (pid  30335) is running...

Code: Select all

[root@fpS241ch rsyslog.d]# getenforce
Disabled

Code: Select all

[root@fpS241ch rsyslog.d]# cat /etc/rsyslog.d/*.conf
$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/spool/rsyslog

# Input for maillog_108.241
$InputFileName /var/log/maillog
$InputFileTag maillog_108.241:
$InputFileStateFile nls-state-var_log_maillog # Must be unique for each file being polled
# Uncomment the folowing line to override the default severity for messages
# from this file.
$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor

# Forward to Nagios Log Server and then discard, otherwise these messages
# will end up in the syslog file (/var/log/messages) unless there are other
# overriding rules.
if $programname == 'maillog_108.241' then @@10.218.108.216:5544
if $programname == 'maillog_108.241' then ~
### Begin forwarding rule for Nagios Log Server                           NAGIOSLOGSERVER
$WorkDirectory /var/spool/rsyslog # Where spool files will live             NAGIOSLOGSERVER
$ActionQueueFileName nlsFwdRule0 # Unique name prefix for spool files     NAGIOSLOGSERVER
$ActionQueueMaxDiskSpace 1g   # 1GB space limit (use as much as possible) NAGIOSLOGSERVER
$ActionQueueSaveOnShutdown on # Save messages to disk on shutdown         NAGIOSLOGSERVER
$ActionQueueType LinkedList   # Use asynchronous processing               NAGIOSLOGSERVER
$ActionResumeRetryCount -1    # Infinite retries if host is down          NAGIOSLOGSERVER
# Remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional       NAGIOSLOGSERVER
*.* @@10.218.108.216:5544                                               # NAGIOSLOGSERVER
### End of Nagios Log Server forwarding rule                              NAGIOSLOGSERVER

Code: Select all

[root@fpS241ch rsyslog.d]# cat /etc/rsyslog.conf
#rsyslog v3 config file

# if you experience problems, check
# http://www.rsyslog.com/troubleshoot for assistance

#### MODULES ####

$ModLoad imuxsock.so	# provides support for local system logging (e.g. via logger command)
$ModLoad imklog.so	# provides kernel logging support (previously done by rklogd)
#$ModLoad immark.so	# provides --MARK-- message capability

# Provides UDP syslog reception
#$ModLoad imudp.so
#$UDPServerRun 514

# Provides TCP syslog reception
#$ModLoad imtcp.so  
#$InputTCPServerRun 514


#### GLOBAL DIRECTIVES ####

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required, 
# not useful and an extreme performance hit
#$ActionFileEnableSync on


#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog


# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 *

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log



# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/spppl/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList   # run asynchronously
#$ActionResumeRetryCount -1    # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###
#
#

$WorkDirectory /var/spool/rsyslog


Re: Linux host not sending logs

Posted: Mon Mar 23, 2015 12:07 pm
by jolson
Please add the following to your rsyslog.conf:

Code: Select all

echo "\$IncludeConfig /etc/rsyslog.d/*.conf" >> /etc/rsyslog.conf
And restart rsyslog:

Code: Select all

service rsyslog restart
Let me know if that works. Thanks!

Re: Linux host not sending logs

Posted: Mon Mar 23, 2015 12:28 pm
by Gabriel_Barba
After applying the change in the conf file, the tcpdump started sending a huge list of info:

Code: Select all

[root@fpS241ch etc]# tcpdump -n dst port 5544 -v
tcpdump: listening on eth4, link-type EN10MB (Ethernet), capture size 65535 bytes


11:22:02.606122 IP (tos 0x0, ttl 64, id 28977, offset 0, flags [DF], proto TCP (6), length 60)
    10.218.108.241.14402 > 10.218.108.216.5544: Flags [S], cksum 0x494d (correct), seq 159366684, win 14600, options [mss 1460,sackOK,TS val 4051291949 ecr 0,nop,wscale 7], length 0
11:22:02.606146 IP (tos 0x0, ttl 64, id 55484, offset 0, flags [DF], proto TCP (6), length 60)
    10.218.108.241.14403 > 10.218.108.216.5544: Flags [S], cksum 0xb021 (correct), seq 1228216209, win 14600, options [mss 1460,sackOK,TS val 4051291949 ecr 0,nop,wscale 7], length 0
11:22:02.606403 IP (tos 0x0, ttl 64, id 28978, offset 0, flags [DF], proto TCP (6), length 52)
    10.218.108.241.14402 > 10.218.108.216.5544: Flags [.], cksum 0x2cf3 (correct), ack 2387045106, win 115, options [nop,nop,TS val 4051291951 ecr 3047806149], length 0
11:22:02.606487 IP (tos 0x0, ttl 64, id 28979, offset 0, flags [DF], proto TCP (6), length 135)
    10.218.108.241.14402 > 10.218.108.216.5544: Flags [P.], cksum 0xeff6 (incorrect -> 0x6d1d), seq 0:83, ack 1, win 115, options [nop,nop,TS val 4051291951 ecr 3047806149], length 83
11:22:02.606497 IP (tos 0x0, ttl 64, id 55485, offset 0, flags [DF], proto TCP (6), length 52)
    10.218.108.241.14403 > 10.218.108.216.5544: Flags [.], cksum 0x65d5 (correct), ack 973794593, win 115, options [nop,nop,TS val 4051291951 ecr 3047806149], length 0
11:22:02.606790 IP (tos 0x0, ttl 64, id 28980, offset 0, flags [DF], proto TCP (6), length 595)
    10.218.108.241.14402 > 10.218.108.216.5544: Flags [P.], cksum 0xf1c2 (incorrect -> 0x3537), seq 83:626, ack 1, win 115, options [nop,nop,TS val 4051291952 ecr 3047806150], length 543
however I still don't see any change in NLS...

Re: Linux host not sending logs

Posted: Mon Mar 23, 2015 1:02 pm
by Gabriel_Barba
I've made a test, first on the server to be tracked (241):

Code: Select all

[root@fpS241ch etc]# tcpdump -n dst port 5544 

I can see a lot of communication flowing when the maillog file gets changed

tracking the NLS I ran the command

Code: Select all

[root@fps216ch ~]# tcpdump src host 10.218.108.241 and udp dst port 5544 and dst host 10.218.108.216
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
But there's no change at all...

Re: Linux host not sending logs

Posted: Mon Mar 23, 2015 1:05 pm
by Gabriel_Barba
And another test in the NLS:

Code: Select all

[root@fps216ch ~]# tcpdump -n dst port 5544
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
12:03:14.230366 IP 10.218.108.241.14402 > 10.218.108.216.5544: Flags [P.], seq 159505845:159506108, ack 2387045106, win 115, options [nop,nop,TS val 4053763228 ecr 3050260800], length 263
12:03:14.320886 IP 10.218.108.241.14402 > 10.218.108.216.5544: Flags [P.], seq 263:607, ack 1, win 115, options [nop,nop,TS val 4053763319 ecr 3050277429], length 344
So messages are comming to the NLS, but those aren't showed up in the webpage...

Re: Linux host not sending logs

Posted: Mon Mar 23, 2015 1:48 pm
by jolson
Now that we know the messages are sending and arriving, my bet is that your logstash filter isn't setup properly.

From the logstash documentation:
Filters

Filters are used as intermediary processing devices in the Logstash chain. They are often combined with conditionals in order to perform a certain action on an event, if it matches particular criteria. Some useful filters:

grok: parses arbitrary text and structure it. Grok is currently the best way in Logstash to parse unstructured log data into something structured and queryable. With 120 patterns shipped built-in to Logstash, it’s more than likely you’ll find one that meets your needs!
You will need to create a filter to tag your maillog data properly. Right now, the data is hitting your server however logstash doesn't know what to do with the data because it doesn't match any of the default filters. You will need to define your own. Please read up on the grok filter here:
http://logstash.net/docs/1.4.2/filters/grok

Re: Linux host not sending logs

Posted: Mon Mar 23, 2015 2:10 pm
by md70758420
Have you checked the Nagios Log Server's timezone? I felt like an idiot when I realized it wasn't set correctly after I got everything else working. I went through pretty much everything you've done here before that.

Re: Linux host not sending logs

Posted: Mon Mar 23, 2015 2:26 pm
by jolson
Please follow the suggestions outlined by myself and md70758420 above and let us know how things work out. Thanks!

Re: Linux host not sending logs

Posted: Mon Mar 23, 2015 2:55 pm
by Gabriel_Barba
Ok, I'm reading the documentation and doing some tests because this grok is new to me, I've found some patterns ready for sendmail however I'm not 100% positive how to implement them in the filters.

Re: Linux host not sending logs

Posted: Mon Mar 23, 2015 3:41 pm
by jolson
No problem - you can think of filters as a way to parse random btis of text that inputs take in. Grok is one of the most powerful filters, and so you have to configure it to parse your maillog properly. Filters can be configured from the Administration -> Global Configuration -> Filters area.

Please let me know if you have any additional questions.