Server not sending all logs as configure.

[GhostRider2110]

Same setup as in my other thread: https://support.nagios.com/forum/viewto ... 71#p301982

Nagios Log Server Cluster:

iganagioslog - CentOS release 6.10 (Final)
This is the first system setup when we started using NLS, I believe it was a VMware image from Nagios.
NLS 2.1.3

iganagioslog01 - Red Hat Enterprise Linux Server release 7.7 (Maipo)
Install from downloaded tar file.
I have a server which I can't get the http logs to go. The syslog files are and I'm getting the http files from other systems with same setup.[root@igapubwebcache01 rsyslog.d]# ls -al
total 32
drwxr-xr-x. 2 root root 230 Jan 22 10:23 .
drwxr-xr-x. 101 root root 8192 Jan 22 10:09 ..
-rw-r--r-- 1 root root 741 Jan 22 10:23 89-nagioslogserver_var_log_httpd_error_log.conf
-rw-r--r-- 1 root root 732 Jan 22 10:23 90-nagioslogserver_var_log_httpd_access_log.conf
-rw-r--r-- 1 root root 736 Jan 21 08:55 90-nagioslogserver_var_log_varnish_varnishncsa.log.conf
-rw-r--r-- 1 root root 1083 Jan 17 13:34 99-nagioslogserver.conf

[root@igapubwebcache01 rsyslog.d]# cat *-nagioslog*
$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/lib/rsyslog

# Input for apache_error
$InputFileName /var/log/httpd/error_log
$InputFileTag apache_error:
$InputFileStateFile nls-state-var_log_httpd_error_log # Must be unique for each file being polled
# Uncomment the folowing line to override the default severity for messages
# from this file.
#$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor

# Forward to Nagios Log Server and then discard, otherwise these messages
# will end up in the syslog file (/var/log/messages) unless there are other
# overriding rules.
if $programname == 'apache_error' then @@iganagiosls01:5582
if $programname == 'apache_error' then stop

$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/lib/rsyslog

# Input for apache_access
$InputFileName /var/log/httpd/access_log
$InputFileTag apache_access:
$InputFileStateFile nls-state-var_log_httpd_access_log # Must be unique for each file being polled
# Uncomment the folowing line to override the default severity for messages
# from this file.
#$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor

# Forward to Nagios Log Server and then discard, otherwise these messages
# will end up in the syslog file (/var/log/messages) unless there are other
# overriding rules.
if $programname == 'apache_access' then @@iganagiosls01:5581
if $programname == 'apache_access' then stop

$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/lib/rsyslog

# Input for varnishncsa
$InputFileName /var/log/varnish/varnishncsa.log
$InputFileTag varnishncsa:
$InputFileStateFile nls-state-var_log_varnish_varnishncsa.log # Must be unique for each file being polled
# Uncomment the folowing line to override the default severity for messages
# from this file.
#$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor

# Forward to Nagios Log Server and then discard, otherwise these messages
# will end up in the syslog file (/var/log/messages) unless there are other
# overriding rules.
if $programname == 'varnishncsa' then @@iganagiosls01:5584
if $programname == 'varnishncsa' then stop
### Begin forwarding rule for Nagios Log Server NAGIOSLOGSERVER
$WorkDirectory /var/lib/rsyslog # Where spool files will live NAGIOSLOGSERVER
$ActionQueueFileName nlsFwdRule0 # Unique name prefix for spool files NAGIOSLOGSERVER
$ActionQueueHighWaterMark 8000 # NAGIOSLOGSERVER
$ActionQueueLowWaterMark 2000 # NAGIOSLOGSERVER
$ActionQueueMaxDiskSpace 1g # 1GB space limit (use as much as possible) NAGIOSLOGSERVER
$ActionQueueSaveOnShutdown on # Save messages to disk on shutdown NAGIOSLOGSERVER
$ActionQueueType LinkedList # Use asynchronous processing NAGIOSLOGSERVER
$ActionResumeRetryCount -1 # Infinite retries if host is down NAGIOSLOGSERVER
# Remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional NAGIOSLOGSERVER
*.* @@iganagiosls01:5544 # NAGIOSLOGSERVER
### End of Nagios Log Server forwarding rule NAGIOSLOGSERVER

rsyslog.conf:

[root@igapubwebcache01 rsyslog.d]# cat ../rsyslog.conf
# rsyslog configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

# The imjournal module bellow is now used as a message source instead of imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
#$ModLoad imklog # reads kernel messages (the same are read from journald)
#$ModLoad immark # provides --MARK-- message capability

# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514

# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514


#### GLOBAL DIRECTIVES ####

# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

# Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
$OmitLocalLogging on

# File to store the position in the journal
$IMJournalStateFile imjournal.state


#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages

# The authpriv file has restricted access.
authpriv.* /var/log/secure

# Log all the mail messages in one place.
mail.* -/var/log/maillog


# Log cron stuff
cron.* /var/log/cron

# Everybody gets emergency messages
*.emerg :omusrmsg:*

# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler

# Save boot messages also to boot.log
local7.* /var/log/boot.log


# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###

Using tcpdump I'm only seeing the packets out from the system via port 5544, nothing out the others. Yet I know this config should work since I am getting them from other systems, on the same subnet as this system and the Vm's are off the same template for deployment. I'm just a little baffled right now.

Thank
Mitch

[GhostRider2110]

Update, I can start nc on the client and hit the ports on the logserver that apache_access and apache_error are supposed to be using and the logserver is seeing them and characters are being processed by the log server. So I guess it's down to debugging why rsyslogd is not sending those two? I guess I could remove the configs and run the scripts again just to see.

[GhostRider2110]

Removed the configs, used the setup script to recreate them for apache_access and apache_error. Still no luck. syslog and the varnish config's are working...
No errors I can see in /var/log/messages related to this.


Thanks
Mitch

[cdienger]

I ran the command "rsyslogd -N 1" to look for errors in the confiruation that was provided and it complained that there was a missing module in the config for teh access_log. Try adding this to the top of 90-nagioslogserver_var_log_httpd_access_log.conf:

Code: Select all

$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/lib/rsyslog

and restart the service and check it for errors:

Code: Select all

service rsyslog restart
rsyslogd -N 1

[GhostRider2110]

I had that in all three files, 89-* and the two 90-* config files, and I would get two warnings that it could not load imfile again since the module was already loaded, from 89-*

Just FYI that is how the files are created by the script. The first apache one I run is given the 89- prefix the rest the 90-prefix. The 89- file had the imfile modload line in it and the rest do not. This is a change from an older setup script, where the imfile modload line was put in every file.

I have run tcpdump listening on both the logserver and the system for traffic to the log server from the system via the port and nothing. If I run nc to that port on the logserver I can see traffic.

Code: Select all

[root@iganagiosls01 elasticsearch]# tcpdump src host 172.16.254.10 and tcp dst port 5582 and dst host 10.100.49.91 -i ens192
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens192, link-type EN10MB (Ethernet), capture size 262144 bytes
16:23:41.224453 IP igapubwebcache01-v.iga.local.57434 > iganagiosls01.iga.local.fac-restore: Flags [S], seq 2746987945, win 29200, options [mss 1460,sackOK,TS val 3632082983 ecr 0,nop,wscale 7], length 0
16:23:41.242557 IP igapubwebcache01-v.iga.local.57434 > iganagiosls01.iga.local.fac-restore: Flags [.], ack 711440459, win 229, options [nop,nop,TS val 3632083001 ecr 88492788], length 0
16:23:43.950031 IP igapubwebcache01-v.iga.local.57434 > iganagiosls01.iga.local.fac-restore: Flags [P.], seq 0:1, ack 1, win 229, options [nop,nop,TS val 3632085709 ecr 88492788], length 1
16:23:44.441951 IP igapubwebcache01-v.iga.local.57434 > iganagiosls01.iga.local.fac-restore: Flags [P.], seq 1:2, ack 1, win 229, options [nop,nop,TS val 3632086201 ecr 88495513], length 1
16:23:45.023239 IP igapubwebcache01-v.iga.local.57434 > iganagiosls01.iga.local.fac-restore: Flags [P.], seq 2:3, ack 1, win 229, options [nop,nop,TS val 3632086782 ecr 88496005], length 1
16:23:46.556419 IP igapubwebcache01-v.iga.local.57434 > iganagiosls01.iga.local.fac-restore: Flags [F.], seq 3, ack 1, win 229, options [nop,nop,TS val 3632088315 ecr 88496586], length 0
16:23:46.574505 IP igapubwebcache01-v.iga.local.57434 > iganagiosls01.iga.local.fac-restore: Flags [.], ack 2, win 229, options [nop,nop,TS val 3632088333 ecr 88498120], length 0

Code: Select all

[root@igapubwebcache01 rsyslog.d]# tcpdump src host 172.16.254.10 and tcp dst port 5582 and dst host 10.100.49.91 -i ens192
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens192, link-type EN10MB (Ethernet), capture size 262144 bytes
16:22:09.071832 IP igapubwebcache01.57038 > iganagiosls01.fac-restore: Flags [S], seq 3917093883, win 29200, options [mss 1460,sackOK,TS val 3631990838 ecr 0,nop,wscale 7], length 0
16:22:09.089852 IP igapubwebcache01.57038 > iganagiosls01.fac-restore: Flags [.], ack 2814552913, win 229, options [nop,nop,TS val 3631990856 ecr 88400643], length 0
16:22:09.089989 IP igapubwebcache01.57038 > iganagiosls01.fac-restore: Flags [P.], seq 0:104, ack 1, win 229, options [nop,nop,TS val 3631990856 ecr 88400643], length 104
16:23:41.216550 IP igapubwebcache01.57434 > iganagiosls01.fac-restore: Flags [S], seq 2746987945, win 29200, options [mss 1460,sackOK,TS val 3632082983 ecr 0,nop,wscale 7], length 0
16:23:41.234641 IP igapubwebcache01.57434 > iganagiosls01.fac-restore: Flags [.], ack 711440459, win 229, options [nop,nop,TS val 3632083001 ecr 88492788], length 0
16:23:43.942171 IP igapubwebcache01.57434 > iganagiosls01.fac-restore: Flags [P.], seq 0:1, ack 1, win 229, options [nop,nop,TS val 3632085709 ecr 88492788], length 1
16:23:44.434092 IP igapubwebcache01.57434 > iganagiosls01.fac-restore: Flags [P.], seq 1:2, ack 1, win 229, options [nop,nop,TS val 3632086201 ecr 88495513], length 1
16:23:45.015329 IP igapubwebcache01.57434 > iganagiosls01.fac-restore: Flags [P.], seq 2:3, ack 1, win 229, options [nop,nop,TS val 3632086782 ecr 88496005], length 1
16:23:46.548555 IP igapubwebcache01.57434 > iganagiosls01.fac-restore: Flags [F.], seq 3, ack 1, win 229, options [nop,nop,TS val 3632088315 ecr 88496586], length 0
16:23:46.566638 IP igapubwebcache01.57434 > iganagiosls01.fac-restore: Flags [.], ack 2, win 229, options [nop,nop,TS val 3632088333 ecr 88498120], length 0

Code: Select all

[root@igapubwebcache01 rsyslog.d]# nc iganagiosls01 5582



^C
[root@igapubwebcache01 rsyslog.d]# 

At this point I'm willing to try anything. Only thing I have not done is reboot the system. Which if I can get approval may be next. Just to correct something, on this system the vanrnishncsa config is not working either.

This is one of 3 varnish cache servers, and of the 3 only 1 is actually sending all the logs as it should. igapubwebcache02 is sending all but the varnishncsa but igapubwebcache03 seems to be fine. I have compared rsyslog.conf and the individual files in rsyslog.d, even to the point of scp'ing them over to one of the other systems.. Still no go...

Will let you know what happens on this next test...


Thanks

Mitch

[GhostRider2110]

Added and here is what I get from the log file:

Code: Select all

Jan 22 16:34:20 igapubwebcache01 rsyslogd: [origin software="rsyslogd" swVersion="8.24.0-41.el7_7.2" x-pid="7149" x-info="http://www.rsyslog.com"] start
Jan 22 16:34:20 igapubwebcache01 rsyslogd: module 'imfile' already in this config, cannot be added  [v8.24.0-41.el7_7.2 try http://www.rsyslog.com/e/2221 ]
Jan 22 16:34:20 igapubwebcache01 systemd: Started System Logging Service.
Jan 22 16:34:20 igapubwebcache01 rsyslogd: rsyslogd's groupid changed to 4

But nothing being pumped out to the logserver...

[GhostRider2110]

Added the following to /etc/rsyslog.conf

### Debugging ###
syslog.* /var/log/syslog.debug;RSYSLOG_DebugFormat
$DebugFile /var/log/syslog.debug
$DebugLevel 2


Attached it the debug file.

Thanks
Mitch

[Box293]

At this point I'm willing to try anything. Only thing I have not done is reboot the system. Which if I can get approval may be next. Just to correct something, on this system the vanrnishncsa config is not working either.

This is one of 3 varnish cache servers, and of the 3 only 1 is actually sending all the logs as it should. igapubwebcache02 is sending all but the varnishncsa but igapubwebcache03 seems to be fine. I have compared rsyslog.conf and the individual files in rsyslog.d, even to the point of scp'ing them over to one of the other systems.. Still no go...

Honestly a full reboot would be a worthwhile test seeing as you have identical configs on separate systems.

[GhostRider2110]

Well, rebooted, got some logs from apache_access, then they stopped... Really not sure where to go from here....

[GhostRider2110]

This is interesting, looking at what was sent to the NLS after the reboot, it looks like the apache logs from around 1630 Jan 22 until log rotation at 0345 Jan 23? Any ideas?