Time Delays between log server and client

[tim620]

We have noticed time delays between Windows clients and the Nagios log server itself.

We tested a Windows client (NXlog) in a different timezone. There seemed to be a long delay. I can't confirm it was in hour delay, but it seemed to be that long of delay. But, it raised a number of questions on how the NXclient works in relation to the log server.

What effect do timezone differences have on the log delays in the log reports?
Does it send logs in batches to the log server or is it a continuous stream of logs?
I remember during a demo that there was a way to watch life logs. How much of a delay is there between the live logs and the logs that are recorded in the DB?

[ssax]

Elasticsearch uses UTC for every and there isn't currently a way to change that. I haven't tested the different TZs but can take a look tomorrow.

Do you have any skew between the Log Server and the sending device (the offset will be different but what about the exact time)?

There really shouldn't be any delay from when it's submitted to Log Server (outside of processing time) and when you see it unless you were really backlogged.

Please PM me a copy of your profile, you can download it from Admin > System Status by clicking the Download System Profile button​.

[ssax]

I see this in your profile:

Code: Select all

[Mon Nov 29 09:55:47.460505 2021] [:error] [pid 19345] [client XXXXXXXXXX] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 80 bytes) in /var/www/html/nagioslogserver/application/libraries/Elasticsearch.php on line 0, referer: http://XXXXXXXX/nagioslogserver/

Please attach this file:

Code: Select all

/etc/php.ini

Your logstash keeps dying as well:

Code: Select all

{:timestamp=>"2021-11-29T11:22:51.555000-0600", :message=>"syslog listener died", :protocol=>:udp, :address=>"0.0.0.0:514", :exception=>#<SocketError: initialize: name or service not known>, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:106:in `initialize'", "org/jruby/ext/socket/RubyUDPSocket.java:115:in `initialize'", "org/jruby/RubyIO.java:871:in `new'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:134:in `udp_listener'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:117:in `server'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:97:in `run'"], :level=>:warn}

Go to Configure > Global Config and remove (or change the port) on the 2nd syslog input listening on port 514, then Apply Configuration:

Code: Select all

    syslog {
        type => 'syslog'
        port => 514
    }
    syslog {
    type => 'syslog-esxi'
    port => 514
    }

You cannot have two inputs listening on the same port.

See if that resolves it, it may have been failing to send because of logstash dying over and over.

[ssax]

The PM didn't contain the /etc/php.ini file, please re-send and make sure to click the Add the File button after selecting the file with the Browse button.

You can run this tail command to see if logstash is failing and just watch the logs:

Code: Select all

tail -Fn50 /var/log/logstash/logstash.log