Server not sending all logs as configure.

[cdienger]

No worries. You know where we'll be

[GhostRider2110]

Have not been able to give this the attention it needs, but I was able to make a change and test.

I move the apache_access logs config to the 5544 port, and that works. Not really sure that that tells me, since other systems, clones of that one are working fine sending to port 5581. Which is the config I have been using for several years. Stay tuned for more info..

Thanks.

[Box293]

Thanks for the update.

[GhostRider2110]

Started back this morning on this. I rebooted the system igapubwebcache01. Nothing but the normal syslog. One thing I had noticed is the file in /var/lib/rsyslog. On other systems, when things are working right, there are about 4 files there, depending on what is configured in /etc/rsyslog.d/ and that makes sense. But there were many more than that in there. So I stopped rsyslog, removed the files from /var/lib/rsyslog and restarted. That created one file in /var/lib/rsyslog, the imjournal.state file. I had to restart rsyslog at least once more for the other 3 files, all imfile-state.##### files to be created. At this point, I am getting the logs from that system into NLS.

As a test I did make a change to one of the rsyslog.d files, changing the server name to the other server in the cluster. Stopped and started rsyslog, and I am still getting logs into NLS.

It's all very strange. I do believe some of this is related to the update to NLS 2x. I didn't have any of these problems before that update.

I want to replace the older NLS server, which is a VM image from Nagios, to a manual install on our own RHEL 7 configured server. I have the system built and ready to add. I'll open another thread on this to make sure I am covering all I need to removing the old server and adding in the new. I have to documentation and have been going over it. But I digress. Where I am leading to is I'm worried any change or restart will put me back where I was with this system that had stopped sending to NLS. Doing an audit if the systems that should be sending logs there are several having the same problem. I am going to go through and try the same thing I did with pubwebcache01 and see what happens. But why would this happen is my real question? Has this been reported before?

Thanks
Mitch

[GhostRider2110]

Another update to this. I went to the next webcache server, which has the same exact rsyslog.d/ config files, and I am getting logs from /var/log/messages, the httpd/access_log (labled apache_access), but two of the other logs are not getting there. Also if the file list in /var/lib/rsyslog looks like this:

Code: Select all

[root@igapubwebcache02 ~]# cd /var/lib/rsyslog/
[root@igapubwebcache02 rsyslog]# ls -al
total 1072
drwx------.  2 root root  32768 Feb 25 09:40 .
drwxr-xr-x. 40 root root   4096 Dec 11 14:17 ..
-rw-------   1 root adm     120 Jan 24 03:13 imfile-state:2785
-rw-------   1 root adm     118 Jan  4 03:28 imfile-state:2814
-rw-------   1 root adm     118 Dec 16 03:23 imfile-state:2830
-rw-------   1 root adm     118 Dec 29 03:15 imfile-state:2844
-rw-------   1 root adm     118 Dec 14 03:30 imfile-state:4682
-rw-------   1 root adm     118 Jan  2 03:49 imfile-state:4733
-rw-------   1 root adm     120 Jan 22 03:26 imfile-state:680
-rw-------   1 root adm     120 Feb 25 03:34 imfile-state:686
-rw-------   1 root adm     120 Feb 24 03:23 imfile-state:688
-rw-------   1 root adm     120 Jan 30 03:19 imfile-state:692
-rw-------   1 root adm     118 Dec 25 03:07 imfile-state:696
-rw-------   1 root adm     118 Dec 31 03:45 imfile-state:699
-rw-------   1 root adm     118 Dec 12 03:26 imfile-state:701
-rw-------   1 root adm     120 Jan 28 03:50 imfile-state:708
-rw-------   1 root adm     118 Jan 26 03:08 imfile-state:713
-rw-------   1 root adm     113 Feb 21 03:31 imfile-state:8389729
-rw-------   1 root adm     104 Feb 21 03:31 imfile-state:8389730
-rw-------   1 root adm     113 Jan 14 03:25 imfile-state:8389733
-rw-------   1 root adm     111 Feb 20 03:30 imfile-state:8389734
-rw-------   1 root adm     106 Feb 20 03:30 imfile-state:8389735
-rw-------   1 root adm     111 Jan 16 03:33 imfile-state:8389736
-rw-------   1 root adm     111 Feb 22 03:47 imfile-state:8389737
-rw-------   1 root adm     104 Feb 22 03:47 imfile-state:8389738
-rw-------   1 root adm     111 Feb 23 03:08 imfile-state:8389739
-rw-------   1 root adm     106 Feb 23 03:08 imfile-state:8389740
-rw-------   1 root adm     111 Dec 13 03:16 imfile-state:8389742
-rw-------   1 root adm     113 Feb 24 03:23 imfile-state:8389743
-rw-------   1 root adm     110 Feb 24 03:23 imfile-state:8389744
-rw-------   1 root adm     113 Feb 25 03:34 imfile-state:8389745
-rw-------   1 root adm     110 Feb 25 03:34 imfile-state:8389746
-rw-------   1 root adm     111 Jan 20 03:24 imfile-state:8389747
-rw-------   1 root adm     111 Feb 25 09:30 imfile-state:8389748
-rw-------   1 root adm     111 Jan 22 03:26 imfile-state:8389749
-rw-------   1 root adm     111 Jan 19 03:47 imfile-state:8389750
-rw-------   1 root adm     102 Jan 19 03:47 imfile-state:8389751
-rw-------   1 root adm     111 Dec 16 03:23 imfile-state:8389752
-rw-------   1 root adm     111 Jan 21 03:28 imfile-state:8389753
-rw-------   1 root adm     104 Jan 22 03:26 imfile-state:8389754
-rw-------   1 root adm     111 Dec 17 03:27 imfile-state:8389755
-rw-------   1 root adm     113 Jan 24 03:13 imfile-state:8389756
-rw-------   1 root adm     102 Jan 21 03:28 imfile-state:8389757
-rw-------   1 root adm     113 Dec 18 03:21 imfile-state:8389758
-rw-------   1 root adm     113 Jan 23 03:13 imfile-state:8389759
-rw-------   1 root adm     104 Jan 24 03:13 imfile-state:8389760
-rw-------   1 root adm     113 Dec 21 03:38 imfile-state:8389761
-rw-------   1 root adm     111 Jan 26 03:08 imfile-state:8389762
-rw-------   1 root adm     102 Jan 23 03:13 imfile-state:8389763
-rw-------   1 root adm     111 Dec 20 03:27 imfile-state:8389764
-rw-------   1 root adm     113 Jan 25 03:24 imfile-state:8389765
-rw-------   1 root adm     102 Jan 26 03:08 imfile-state:8389766
-rw-------   1 root adm     113 Dec 23 03:30 imfile-state:8389767
-rw-------   1 root adm     106 Jan 25 03:24 imfile-state:8389768
-rw-------   1 root adm     111 Dec 22 03:43 imfile-state:8389769
-rw-------   1 root adm     111 Jan 27 03:33 imfile-state:8389770
-rw-------   1 root adm     113 Jan 28 03:50 imfile-state:8389771
-rw-------   1 root adm     104 Jan 27 03:33 imfile-state:8389772
-rw-------   1 root adm     113 Dec 24 03:39 imfile-state:8389773
-rw-------   1 root adm     113 Jan 29 03:21 imfile-state:8389774
-rw-------   1 root adm     104 Jan 28 03:50 imfile-state:8389775
-rw-------   1 root adm     111 Dec 25 03:07 imfile-state:8389776
-rw-------   1 root adm     111 Jan 30 03:19 imfile-state:8389777
-rw-------   1 root adm     104 Jan 29 03:21 imfile-state:8389778
-rw-------   1 root adm     111 Dec 26 03:16 imfile-state:8389779
-rw-------   1 root adm     111 Jan 31 03:35 imfile-state:8389780
-rw-------   1 root adm     102 Jan 30 03:19 imfile-state:8389781
-rw-------   1 root adm     110 Jan 31 03:35 imfile-state:8389782
-rw-------   1 root adm     104 Dec 27 03:36 imfile-state:8389783
-rw-------   1 root adm     113 Feb  1 03:50 imfile-state:8389784
-rw-------   1 root adm     112 Feb  1 03:50 imfile-state:8389785
-rw-------   1 root adm     104 Dec 28 03:43 imfile-state:8389786
-rw-------   1 root adm     111 Feb  2 03:35 imfile-state:8389787
-rw-------   1 root adm     104 Feb  2 03:35 imfile-state:8389788
-rw-------   1 root adm     111 Dec 30 03:49 imfile-state:8392736
-rw-------   1 root adm     113 Feb  4 03:29 imfile-state:8392739
-rw-------   1 root adm     111 Dec 29 03:15 imfile-state:8392741
-rw-------   1 root adm     104 Dec 29 03:15 imfile-state:8392742
-rw-------   1 root adm     111 Feb  3 03:12 imfile-state:8392744
-rw-------   1 root adm     104 Feb  3 03:12 imfile-state:8392745
-rw-------   1 root adm     113 Feb  5 03:25 imfile-state:8392748
-rw-------   1 root adm     104 Dec 30 03:49 imfile-state:8392749
-rw-------   1 root adm     110 Feb  4 03:29 imfile-state:8392751
-rw-------   1 root adm     104 Jan  1 03:41 imfile-state:8392752
-rw-------   1 root adm     113 Feb  6 03:18 imfile-state:8392754
-rw-------   1 root adm     110 Feb  5 03:25 imfile-state:8392755
-rw-------   1 root adm     111 Feb  7 03:28 imfile-state:8392757
-rw-------   1 root adm     113 Jan  3 03:34 imfile-state:8392758
-rw-------   1 root adm     104 Feb  7 03:28 imfile-state:8392759
-rw-------   1 root adm     104 Feb  6 03:18 imfile-state:8392760
-rw-------   1 root adm     111 Feb  8 03:36 imfile-state:8392762
-rw-------   1 root adm     113 Jan  4 03:28 imfile-state:8392763
-rw-------   1 root adm     104 Feb  8 03:36 imfile-state:8392764
-rw-------   1 root adm     111 Feb  9 03:45 imfile-state:8392765
-rw-------   1 root adm     111 Jan  5 03:46 imfile-state:8392766
-rw-------   1 root adm     104 Feb  9 03:45 imfile-state:8392767
-rw-------   1 root adm     111 Feb 10 03:17 imfile-state:8392768
-rw-------   1 root adm     111 Jan  6 03:39 imfile-state:8392769
-rw-------   1 root adm     111 Feb 11 03:08 imfile-state:8392770
-rw-------   1 root adm     104 Feb 10 03:17 imfile-state:8392771
-rw-------   1 root adm     104 Feb 11 03:08 imfile-state:8392772
-rw-------   1 root adm     113 Feb 12 03:31 imfile-state:8392774
-rw-------   1 root adm     106 Feb 12 03:31 imfile-state:8392775
-rw-------   1 root adm     113 Feb 13 03:30 imfile-state:8392777
-rw-------   1 root adm     104 Feb 13 03:30 imfile-state:8392778
-rw-------   1 root adm     111 Feb 14 03:12 imfile-state:8392780
-rw-------   1 root adm     112 Feb 14 03:12 imfile-state:8392781
-rw-------   1 root adm     113 Feb 15 03:14 imfile-state:8392783
-rw-------   1 root adm     104 Feb 15 03:14 imfile-state:8392784
-rw-------   1 root adm     104 Jan  9 03:18 imfile-state:8392785
-rw-------   1 root adm     111 Feb 16 03:20 imfile-state:8392786
-rw-------   1 root adm     112 Feb 16 03:20 imfile-state:8392787
-rw-------   1 root adm     106 Jan 11 03:08 imfile-state:8392788
-rw-------   1 root adm     111 Feb 17 03:26 imfile-state:8392789
-rw-------   1 root adm     110 Feb 17 03:26 imfile-state:8392790
-rw-------   1 root adm     104 Jan 12 03:10 imfile-state:8392791
-rw-------   1 root adm     111 Feb 18 03:11 imfile-state:8392792
-rw-------   1 root adm     104 Feb 18 03:11 imfile-state:8392793
-rw-------   1 root adm     111 Jan 13 03:38 imfile-state:8392794
-rw-------   1 root adm     113 Feb 19 03:46 imfile-state:8392795
-rw-------   1 root adm     106 Feb 19 03:46 imfile-state:8392796
-rw-------   1 root adm     104 Jan 13 03:38 imfile-state:8392798
-rw-------   1 root adm     127 Feb 25 09:40 imjournal.state
-rw-------   1 root adm  548154 Feb 15 08:51 nlsFwdRule0.00000066
-rw-------   1 root adm     577 Feb 15 08:51 nlsFwdRule0.qi
-rw-------   1 root adm     245 Dec 11 14:14 nls-state-var_log_httpd_access_log
-rw-------   1 root adm     239 Dec 11 14:14 nls-state-var_log_httpd_error_log
-rw-------   1 root adm     248 Dec 11 14:14 nls-state-var_log_varnish_varnishncsa.log

I shut down rsyslog, removed all the file in /var/lib/rsyslog, restart and it took a little bit, but now /var/lib/rsyslog looks like this:

Code: Select all

[root@igapubwebcache02 rsyslog]# ls -al
total 920
drwx------.  2 root root    131 Feb 25 09:51 .
drwxr-xr-x. 40 root root   4096 Dec 11 14:17 ..
-rw-------   1 root adm     118 Feb 25 09:47 imfile-state:688
-rw-------   1 root adm     111 Feb 25 09:48 imfile-state:8389748
-rw-------   1 root adm     127 Feb 25 09:51 imjournal.state
-rw-------   1 root adm  920821 Feb 25 09:48 nlsFwdRule0.00000005
-rw-------   1 root adm     575 Feb 25 09:48 nlsFwdRule0.qi

Which is even different from igapubwebcache01, shich does not have the nlsFwdRule0* files. I'm a little confused.

Thanks
Mitch

[Box293]

I think some troubleshooting needs to occur to determine the following:

Are the logs reaching Nagios Log Server?

Is syslog sending the logs?

In think these questions will be answered if you enabled debug logging for the rsyslog service itself. Then you need to watch to see if the logs are being sent. I would personally diagnose each log type one at a time.

[cdienger]

Maybe I missed it somewhere, but what OS is igapubwebcache01 and igapubwebcache02 running and what is the version of rsyslog is on those? I don't know if this is exactly the issue, but I've run into issues with bugs like https://github.com/rsyslog/rsyslog/issues/2659 before. Are you able to update the rsyslog services?

[GhostRider2110]

I think some troubleshooting needs to occur to determine the following:

Are the logs reaching Nagios Log Server?

Is syslog sending the logs?

In think these questions will be answered if you enabled debug logging for the rsyslog service itself. Then you need to watch to see if the logs are being sent. I would personally diagnose each log type one at a time.

Thanks Box293,

In short, sometimes and it's not consistant and it's across a many systems. When things are not working, some of the logs is being sent to the NLS and some are not. tcpdump shows this. The missing logs are not going out of the client to the NLS. The only consistancy is that the journal logs (syslog) are going and being received. I have run rsyslog in debug mode but can't really determine the error.

[GhostRider2110]

Maybe I missed it somewhere, but what OS is igapubwebcache01 and igapubwebcache02 running and what is the version of rsyslog is on those? I don't know if this is exactly the issue, but I've run into issues with bugs like https://github.com/rsyslog/rsyslog/issues/2659 before. Are you able to update the rsyslog services?

All systems are RHEL 7.7 with the latest updates from RedHat. One of the NLS servers is also RHEL 7 and the other is CentOS 6 based off the VMware image Nagios provides. I'll look at the bug report you listed and see. Thanks

[GhostRider2110]

cdienger,

The latest from RHEL is:

root@igapubapi02 ~]# rsyslogd -v
rsyslogd 8.24.0-41.el7_7.2, compiled with:
PLATFORM: x86_64-redhat-linux-gnu
PLATFORM (lsb_release -d):
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support: Yes
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
memory allocator: system default
Runtime Instrumentation (slow code): No
uuid support: Yes
Number of Bits in RainerScript integers: 64

I looked at the link you sent, but that is not quite what is going on, I think. All the ones I reset yesterday as I described earlier in the thread, rotated last night and are still functioning. I guess I will need to find another on that is failing and run it with debug going, then one of the one that is working and see what the difference is. Thanks...