rsyslog can't connect to Nagios Log Server cluster node

[stecino]

Hi all,

I have centos server that I am trying to add to Nagios Log Server

Rsyslog is running, but for some reason it wont connect to Nagios Log Server cluster node


[root@sbur2tl1 ~]# netstat -nap | grep rsyslog
unix 5 [ ] DGRAM 273448 27073/rsyslogd /dev/log

Connectivity is there

[root@sbur2tl1 ~]# telnet xx.xx.1.147 5544
Trying xx.xx.1.147...
Connected to xx.xx.1.147.
Escape character is '^]'.
^]
telnet> quit

This is my nagios log server conf

[root@sbur2tl1 ~]# cat /etc/rsyslog.d/99-nagioslogserver.conf
### Begin forwarding rule for Nagios Log Server NAGIOSLOGSERVER
$WorkDirectory /var/lib/rsyslog # Where spool files will live NAGIOSLOGSERVER
$ActionQueueFileName nlsFwdRule0 # Unique name prefix for spool files NAGIOSLOGSERVER
$ActionQueueMaxDiskSpace 1g # 1GB space limit (use as much as possible) NAGIOSLOGSERVER
$ActionQueueSaveOnShutdown on # Save messages to disk on shutdown NAGIOSLOGSERVER
$ActionQueueType LinkedList # Use asynchronous processing NAGIOSLOGSERVER
$ActionResumeRetryCount -1 # Infinite retries if host is down NAGIOSLOGSERVER
# Remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional NAGIOSLOGSERVER
*.* @@xx.xx.1.147:5544 # NAGIOSLOGSERVER
### End of Nagios Log Server forwarding rule NAGIOSLOGSERVER

I checked and rsyslog is capturing the events

# ### end of the forwarding rule ###
[root@sbur2tl1 ~]# logger 'this is a test'
[root@sbur2tl1 ~]# cat /var/log/messages | grep 'this is a test'
Dec 19 14:55:02 sbur2tl1 root: this is a test

[stecino]

Had to explicitly define the rsyslog.conf file. For things to work.
It's weird because on the other ones, it just works. I will have to look at the startup script for rsyslog

[tgriep]

Are you still having problems with sending the logs from your Centos server to the Nagios Logserver?

[stecino]

No it got resolved

[tgriep]

Thanks for getting back to us. I'll close the ticket.

[stecino]

Actually, I am running into similar issues with Ubuntu. This time it shows that it is connected, but it's not transporting logs, as I don't see logsource being added to the Nagios log server

root@sbur2api01:~# netstat -nap | grep rsyslog
tcp 0 0 10.xx.xx.135:36921 10.xx.x.147:5544 ESTABLISHED 30536/rsyslogd
unix 2 [ ] DGRAM 49501156 30536/rsyslogd /var/spool/postfix/dev/log
unix 2 [ ] DGRAM 49501154 30536/rsyslogd /dev/log

root@sbur2api01:~# logger mytest
root@sbur2api01:~# cat /var/log/syslog | grep mytest
Dec 30 22:00:20 sbur2api01 root: mytest

root@sbur2api01:~# ps -ef | grep rsyslog
syslog 30536 1 0 21:17 ? 00:00:00 rsyslogd -c5 -f /etc/rsyslog.conf

root@sbur2api01:~# cat /etc/rsyslog.d/99-nagioslogserver.conf
### Begin forwarding rule for Nagios Log Server NAGIOSLOGSERVER
$WorkDirectory /var/spool/rsyslog # Where spool files will live NAGIOSLOGSERVER
$ActionQueueFileName nlsFwdRule0 # Unique name prefix for spool files NAGIOSLOGSERVER
$ActionQueueMaxDiskSpace 1g # 1GB space limit (use as much as possible) NAGIOSLOGSERVER
$ActionQueueSaveOnShutdown on # Save messages to disk on shutdown NAGIOSLOGSERVER
$ActionQueueType LinkedList # Use asynchronous processing NAGIOSLOGSERVER
$ActionResumeRetryCount -1 # Infinite retries if host is down NAGIOSLOGSERVER
# Remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional NAGIOSLOGSERVER
*.* @@10.xx.x.147:5544 # NAGIOSLOGSERVER
### End of Nagios Log Server forwarding rule NAGIOSLOGSERVER

I noticed it's using UTC, vs PST, does it matter?

[stecino]

Actually, I am running into similar issues with Ubuntu. This time it shows that it is connected, but it's not transporting logs, as I don't see logsource being added to the Nagios log server

root@sbur2api01:~# netstat -nap | grep rsyslog
tcp 0 0 10.xx.xx.135:36921 10.xx.x.147:5544 ESTABLISHED 30536/rsyslogd
unix 2 [ ] DGRAM 49501156 30536/rsyslogd /var/spool/postfix/dev/log
unix 2 [ ] DGRAM 49501154 30536/rsyslogd /dev/log

root@sbur2api01:~# logger mytest
root@sbur2api01:~# cat /var/log/syslog | grep mytest
Dec 30 22:00:20 sbur2api01 root: mytest

root@sbur2api01:~# ps -ef | grep rsyslog
syslog 30536 1 0 21:17 ? 00:00:00 rsyslogd -c5 -f /etc/rsyslog.conf

root@sbur2api01:~# cat /etc/rsyslog.d/99-nagioslogserver.conf
### Begin forwarding rule for Nagios Log Server NAGIOSLOGSERVER
$WorkDirectory /var/spool/rsyslog # Where spool files will live NAGIOSLOGSERVER
$ActionQueueFileName nlsFwdRule0 # Unique name prefix for spool files NAGIOSLOGSERVER
$ActionQueueMaxDiskSpace 1g # 1GB space limit (use as much as possible) NAGIOSLOGSERVER
$ActionQueueSaveOnShutdown on # Save messages to disk on shutdown NAGIOSLOGSERVER
$ActionQueueType LinkedList # Use asynchronous processing NAGIOSLOGSERVER
$ActionResumeRetryCount -1 # Infinite retries if host is down NAGIOSLOGSERVER
# Remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional NAGIOSLOGSERVER
*.* @@10.xx.x.147:5544 # NAGIOSLOGSERVER
### End of Nagios Log Server forwarding rule NAGIOSLOGSERVER

I noticed it's using UTC, vs PST, does it matter?

I needed to restart logstash on the cluster instances a few times, then logsources appeared