false log flood?

[lesnikov]

hello,

This night we got tons of security messages from nagios log for false positive logs? ( i hope so )
We tought it was security breach.. So after some coffe and blocking half of network we didnt find anything. in the end we blocked all traffic going to nagios log server (isoloate from network) and we see that logs are still showing. check image:
strange thing in log is that "acceptet password" for user that exsist from IP that shouldnt be login via ssh to that server.
any idea what could be? we did restart services/reboot logs for this are still updating.

nagios log server:
Linux nagioslog 2.6.32-279.11.1.el6.x86_64 #1 SMP Tue Oct 16 15:57:10 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
Linux version 2.6.32-279.11.1.el6.x86_64 ([email protected]) (gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #1 SMP Tue Oct 16 15:57:10 UTC 2012
Nagios Log Server 2.0.2

[scottwilkerson]

It's hard to say but you blocked out the part that actually says what server send the logs (the host column). If these are all the same, the investigations should start on that server.

[lesnikov]

Blocked host is remote rsyslog server (public IP). We checked logs from all servers on that site, also gw,fw etc... And didn't find anything. thats why we isolated nagios log server from receiving logs from outside and this "flood" was still showing for 8 hours. it started at 01:05:52 and stoped at 07:30. And started again yesterday and it is keep flooding. remote host is blocked in FW and we double checked in nagios ls and there is no connection. but logs keep showing. every new log with different port number "Accepted password for user from 10.0.0.8 port 54851 ssh2" 54852, 54853 etc...

we also blocked all incoming logs from local servers. and this flood is still in showing up.
so whit all traffic blocked from Nagios LS only logs that are showing now are flood from before and logs from 127.0.0.1 (nagios LS)


can you suggest what else can we check for on Nagios LS? is there any live log that we can tail and see. we are out of ideas what could it be.

regards,

[scottwilkerson]

One possible cause of this would be if the sending server had a different timestamp and the logs were actually dated in the future.

Which would mean they aren't coming in now, they came in earlier but had a timestamp for the time you saw them in the logs

A easy way to check for this would be to set the time period for the dashboard to Custom and choose an end date that is in the future

[lesnikov]

scottwilkerson:

good call. i added screenshot for logs in future.
But i still think that those logs are corupt in some way becouse we just cant find that log "Accepted log with different port every time". or they are really logs from future . ok joke a side.

so logs in future are from local and remote servers:
ubuntu/centos servers (standalone directly sending logs to nagios LS)
ubuntu/centos servers (sending to remote rsyslog colector than parsing to LS)

windows, freebsd are not affected in future logs (sending logs directly and remotly to nagios LS)
we used script from http://logserver/nagioslogserver/config ... log-source

without idea what to do next. or how to identify why timestamps are wrong?

[lesnikov]

ok i think i know what caused it. there was a spike 4 days ago from on of the remote rsyslog server. spike was 428,410 events per 30m. (attached screenshot). it was sending everything from 18.1.2018 --->
after some reading on google there are some bugs regarding this issue for rsyslog.

And for Nagios LS index status, i didnt figure out that logstash-2018.07.07 was name by date becase i saw names in future ))
removed spikes and logs in future. will recheck/reinstall syslong confs on all servers that there was log in future and try to figure it out which one caused it.

thank you for help you can mark this thread as resolved.

[scottwilkerson]

ok i think i know what caused it. there was a spike 4 days ago from on of the remote rsyslog server. spike was 428,410 events per 30m. (attached screenshot). it was sending everything from 18.1.2018 --->
after some reading on google there are some bugs regarding this issue for rsyslog.

And for Nagios LS index status, i didnt figure out that logstash-2018.07.07 was name by date becase i saw names in future ))
removed spikes and logs in future. will recheck/reinstall syslong confs on all servers that there was log in future and try to figure it out which one caused it.

thank you for help you can mark this thread as resolved.

Excellent, glad it is resolved.