We have noticed time delays between Windows clients and the Nagios log server itself.
We tested a Windows client (NXlog) in a different timezone. There seemed to be a long delay. I can't confirm it was in hour delay, but it seemed to be that long of delay. But, it raised a number of questions on how the NXclient works in relation to the log server.
What effect do timezone differences have on the log delays in the log reports?
Does it send logs in batches to the log server or is it a continuous stream of logs?
I remember during a demo that there was a way to watch life logs. How much of a delay is there between the live logs and the logs that are recorded in the DB?
Time Delays between log server and client
Re: Time Delays between log server and client
Elasticsearch uses UTC for every and there isn't currently a way to change that. I haven't tested the different TZs but can take a look tomorrow.
Do you have any skew between the Log Server and the sending device (the offset will be different but what about the exact time)?
There really shouldn't be any delay from when it's submitted to Log Server (outside of processing time) and when you see it unless you were really backlogged.
Please PM me a copy of your profile, you can download it from Admin > System Status by clicking the Download System Profile button.
Do you have any skew between the Log Server and the sending device (the offset will be different but what about the exact time)?
There really shouldn't be any delay from when it's submitted to Log Server (outside of processing time) and when you see it unless you were really backlogged.
Please PM me a copy of your profile, you can download it from Admin > System Status by clicking the Download System Profile button.
Re: Time Delays between log server and client
I see this in your profile:
Please attach this file:
Your logstash keeps dying as well:
Go to Configure > Global Config and remove (or change the port) on the 2nd syslog input listening on port 514, then Apply Configuration:
You cannot have two inputs listening on the same port.
See if that resolves it, it may have been failing to send because of logstash dying over and over.
Code: Select all
[Mon Nov 29 09:55:47.460505 2021] [:error] [pid 19345] [client XXXXXXXXXX] PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 80 bytes) in /var/www/html/nagioslogserver/application/libraries/Elasticsearch.php on line 0, referer: http://XXXXXXXX/nagioslogserver/
Code: Select all
/etc/php.ini
Code: Select all
{:timestamp=>"2021-11-29T11:22:51.555000-0600", :message=>"syslog listener died", :protocol=>:udp, :address=>"0.0.0.0:514", :exception=>#<SocketError: initialize: name or service not known>, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:106:in `initialize'", "org/jruby/ext/socket/RubyUDPSocket.java:115:in `initialize'", "org/jruby/RubyIO.java:871:in `new'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:134:in `udp_listener'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:117:in `server'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:97:in `run'"], :level=>:warn}
Code: Select all
syslog {
type => 'syslog'
port => 514
}
syslog {
type => 'syslog-esxi'
port => 514
}
See if that resolves it, it may have been failing to send because of logstash dying over and over.
Re: Time Delays between log server and client
The PM didn't contain the /etc/php.ini file, please re-send and make sure to click the Add the File button after selecting the file with the Browse button.
You can run this tail command to see if logstash is failing and just watch the logs:
You can run this tail command to see if logstash is failing and just watch the logs:
Code: Select all
tail -Fn50 /var/log/logstash/logstash.log