Linux host not sending logs

[Gabriel_Barba]

Hello,

I've a problem with linux hosts, since they aren't sending logs to log server, I ran the instalation instructions:

[root@fpS241ch work]# bash /root/setup-linux.sh -s 10.218.108.216 -p 5544
Detected rsyslog 4.6.2
Detected rsyslog work directory /var/spool/rsyslog
Destination Log Server: 10.218.108.216:5544
Creating /etc/rsyslog.d/99-nagioslogserver.conf...
SELinux is disabled.
rsyslog configuration check passed.
Restarting rsyslog service with 'service'...
Shutting down system logger: [ OK ]
Starting system logger: [ OK ]
Okay.
rsyslog is running with the new configuration.
Visit your Nagios Log Server dashboard to verify that logs are being received.

and then created an specific file to be sent
[root@fpS241ch ~]# bash setup-linux.sh -s 10.218.108.216 -p 5544 -f /var/log/maillog -t maillog_108.241
Detected rsyslog 4.6.2
Detected rsyslog work directory /var/spool/rsyslog
Destination Log Server: 10.218.108.216:5544
Creating /etc/rsyslog.d/90-nagioslogserver_var_log_maillog.conf...
SELinux is disabled.
rsyslog configuration check passed.
Restarting rsyslog service with 'service'...
Shutting down system logger: [ OK ]
Starting system logger: [ OK ]
Okay.
rsyslog is running with the new configuration.
Visit your Nagios Log Server dashboard to verify that logs are being received.


however if I go to log server, there's no information about this host, we are able to see other hosts (not one of them is linux host just network devices).

Can you please help me to find out what I'm missing?

Regards,
Gabriel

[Box293]

Are you seeing the traffic hit the Nagios Log Server?

Run this on your Nagios Log Server:

Code: Select all

tcpdump src host xxx.xxx.xxx.xxx and tcp dst port 5544 and dst host 10.218.108.216

Change xxx.xxx.xxx.xxx to be the IP Address of the server sending the logs.


Or if it's UDP traffic:

Code: Select all

tcpdump src host xxx.xxx.xxx.xxx and udp dst port 5544 and dst host 10.218.108.216

[Gabriel_Barba]

Hello,

I've ran the command for 10 minutes each, and can't see any traffic on the Log Server, I see at the server I want to collect logs rsyslog is running:

[root@fpS241ch ~]# service rsyslog status
rsyslogd (pid 31676) is running...

and the file is generating logs:
[root@fpS241ch log]# ll maillog
-rw------- 1 root root 4815851 Mar 23 08:12 maillog
[root@fpS241ch log]# ll maillog
-rw------- 1 root root 4816899 Mar 23 08:14 maillog

I'm missing something?

Regards,
Gabriel

[jolson]

What distribution of linux are you running this script on?

Code: Select all

cat /etc/*release

I trust that pings work fine from the linux host to the log server?

Code: Select all

ping 10.218.108.216

Are there any firewalls in between that you would need to create an ACL in?

On your Nagios Log Server, is the Logstash port open and listening?

Code: Select all

netstat -na |grep 5544
iptables -L

Let us know. Thanks!

[Gabriel_Barba]

Hello,

Here are the outputs:

on the client where we are running the script:

Code: Select all

[root@fpS241ch rsyslog.d]# cat /etc/*release
Red Hat Enterprise Linux Server release 6.2 (Santiago)
Red Hat Enterprise Linux Server release 6.2 (Santiago)

Code: Select all

[root@fpS241ch rsyslog.d]# ping 10.218.108.216
PING 10.218.108.216 (10.218.108.216) 56(84) bytes of data.
64 bytes from 10.218.108.216: icmp_seq=1 ttl=64 time=0.372 ms

On the log server:

Code: Select all

[root@fps216ch ~]# netstat -na |grep 5544
tcp        0      0 :::5544                     :::*                        LISTEN      
tcp        0      0 ::ffff:10.218.108.216:5544  ::ffff:10.218.108.167:51283 ESTABLISHED 
tcp        0      0 ::ffff:10.218.108.216:5544  ::ffff:10.218.108.167:55438 ESTABLISHED 
tcp        0      0 ::ffff:10.218.108.216:5544  ::ffff:10.218.108.167:55364 ESTABLISHED 
tcp        0      0 ::ffff:10.218.108.216:5544  ::ffff:10.218.108.167:58272 ESTABLISHED 
tcp        0      0 ::ffff:10.218.108.216:5544  ::ffff:10.218.108.201:33074 ESTABLISHED 
tcp        0      0 ::ffff:10.218.108.216:5544  ::ffff:10.218.108.167:38372 ESTABLISHED 
tcp        0      0 ::ffff:10.218.108.216:5544  ::ffff:10.218.108.167:51282 ESTABLISHED 
tcp        0      0 ::ffff:10.218.108.216:5544  ::ffff:10.218.108.167:58275 ESTABLISHED 
tcp        0      0 ::ffff:10.218.108.216:5544  ::ffff:10.218.108.201:33075 ESTABLISHED 
tcp        0      0 ::ffff:10.218.108.216:5544  ::ffff:10.218.108.167:38371 ESTABLISHED 
udp        0      0 :::5544                     :::*                                    
[root@fps216ch ~]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination    

Code: Select all

[root@fps216ch ~]# cat /etc/*release
CentOS release 6.6 (Final)
CentOS release 6.6 (Final)
CentOS release 6.6 (Final)

[jolson]

Interesting - it doesn't look like iptables is listing any rules in its chain. Could you try turning iptables off on the Nagios Log Server temporarily?

service iptables stop

See if the logs start coming in now...

If the logs do not, please turn iptables back on, and let us know the output of the following:

Code: Select all

tail /var/log/logstash/logstash.log

Thank you!

[Gabriel_Barba]

so... I turned off iptables, and the logs still not in the log server...

Code: Select all

[root@fps216ch ~]# service iptables stop
iptables: Setting chains to policy ACCEPT: filter          [  OK  ]
iptables: Flushing firewall rules:                         [  OK  ]
iptables: Unloading modules:                               [  OK  ]

went checking to the log server, and nothing is comming (previously waited to see if maillog is creating some data)

Code: Select all

[root@fps216ch ~]# service iptables start
iptables: Applying firewall rules:                         [  OK  ]
[root@fps216ch ~]# tail /var/log/logstash/logstash.log
[root@fps216ch ~]# 

[jolson]

We ran a TCPdump and saw that your NLS was not getting any data from log sources. After that, we turned your firewall off and verified that the log data is still not transmitted. This being the case, the logs are either not being sent from your box, or they are not getting through the network properly.

Just to make certain logstash is fine, let's check that it's up and running properly:

Code: Select all

service logstash status

Code: Select all

cat /usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf

Code: Select all

cat /usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf

Code: Select all

cat /etc/sysconfig/logstash

Code: Select all

ps -ef |grep logstash

After that, let's go back to your linux box that is sending the logs, and run the following:

Code: Select all

tcpdump -n dst port 5544

If you see traffic being sent, please let us know.

Thank you.

[Gabriel_Barba]

Ok, for the first set of instructions here is the output:

Code: Select all

[root@fps216ch ~]#     service logstash status
Logstash Daemon (pid  1294) is running...
[root@fps216ch ~]#     cat /usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf
# 
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Mon, 01 Dec 2014 10:37:57 -0500
#

#
# Global inputs
#

input {
    syslog {
        type => 'syslog'
        port => 5544
    }
    tcp {
        type => 'eventlog'
        port => 3515
        codec => json {
            charset => 'CP1252'
        }
    }
    tcp {
        type => 'import_raw'
        tags => 'import_raw'
        port => 2056
    }
    tcp {
        type => 'import_json'
        tags => 'import_json'
        port => 2057
        codec => json
    }
}

#
# Local inputs
#


[root@fps216ch ~]#     cat /usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf
# 
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Mon, 01 Dec 2014 10:37:57 -0500
#

#
# Global filters
#

filter {
    if [program] == 'apache_access' {
        grok {
            match => [ 'message', '%{COMBINEDAPACHELOG}']
        }
        date {
            match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z' ]
        }
        mutate {
            replace => [ 'type', 'apache_access' ]
             convert => [ 'bytes', 'integer' ]
             convert => [ 'response', 'integer' ]
        }
    }
     
    if [program] == 'apache_error' {
        grok {
            match => [ 'message', '\[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\] \[%{WORD:class}\] \[%{WORD:originator} %{IP:clientip}\] %{GREEDYDATA:errmsg}']
        }
        mutate {
            replace => [ 'type', 'apache_error' ]
        }
    }
}

#
# Local filters
#


[root@fps216ch ~]#     cat /etc/sysconfig/logstash
###############################
# Default settings for logstash
###############################

# Override Java location
#JAVACMD=/usr/bin/java

# Set a home directory
APP_DIR=/usr/local/nagioslogserver
LS_HOME="$APP_DIR/logstash"

# set ES_CLUSTER
ES_CLUSTER=$(cat $APP_DIR/var/cluster_uuid)

# Arguments to pass to java
#LS_HEAP_SIZE="256m"
LS_JAVA_OPTS="-Djava.io.tmpdir=$APP_DIR/tmp"

# Logstash filter worker threads
#LS_WORKER_THREADS=1

# pidfiles aren't used for upstart; this is for sysv users.
#LS_PIDFILE=/var/run/logstash.pid

# user id to be invoked as; for upstart: edit /etc/init/logstash.conf
LS_USER=nagios
LS_GROUP=nagios

# logstash logging
#LS_LOG_FILE=/var/log/logstash/logstash.log
#LS_USE_GC_LOGGING="true"

# logstash configuration directory
LS_CONF_DIR="$LS_HOME/etc/conf.d"

# Open file limit; cannot be overridden in upstart
#LS_OPEN_FILES=2048

# Nice level
#LS_NICE=0

# Increase Filter workers to 4 threads
LS_OPTS=" -w 4"

if [ "x$1" == "xstart" -o "x$1" == "xrestart" -o "x$1" == "xreload" ];then
	GET_LOGSTASH_CONFIG_MESSAGE=$( php /usr/local/nagioslogserver/scripts/get_logstash_config.php )
	GET_LOGSTASH_CONFIG_RETURN=$?
	if [ "$GET_LOGSTASH_CONFIG_RETURN" != "0" ]; then
		echo $GET_LOGSTASH_CONFIG_MESSAGE
		exit 1
	fi
fi[root@fps216ch ~]#     ps -ef |grep logstash
root      1292     1  0  2014 ?        00:00:00 runuser -s /bin/sh -c exec /usr/local/nagioslogserver/logstash/bin/logstash agent -f /usr/local/nagioslogserver/logstash/etc/conf.d -l /var/log/logstash/logstash.log  -w 4 nagios
nagios    1294  1292  0  2014 ?        02:02:10 /usr/bin/java -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -Xmx500m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -jar /usr/local/nagioslogserver/logstash/vendor/jar/jruby-complete-1.7.11.jar -I/usr/local/nagioslogserver/logstash/lib /usr/local/nagioslogserver/logstash/lib/logstash/runner.rb agent -f /usr/local/nagioslogserver/logstash/etc/conf.d -l /var/log/logstash/logstash.log -w 4
root     54596 48728  0 09:56 pts/1    00:00:00 grep logstash

on the server that's supposed to send the logs:

Code: Select all

[root@fpS241ch rsyslog.d]#     tcpdump -n dst port 5544
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth4, link-type EN10MB (Ethernet), capture size 65535 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel

[jolson]

So far we have determined that nothing is wrong on the NLS side of things. On your Linux host, please check the following settings...

Code: Select all

service rsyslogd status

Code: Select all

getenforce

Code: Select all

cat /etc/rsyslog.d/*.conf

Code: Select all

cat /etc/rsyslog.conf