Hourly Sending Check on Nagios Log Server 2.0

[cgutierr]

Yes, logstash and elasticsearch are both running and have been running. This seems to only happen during the early morning hours as well, say 02:00 to 05:00 PST where network devices do not check in.

[mcapra]

Do you have historical Logstash and ElasticSearch logs you could provide? Would it be possible for you to submit an attachment of everything in these 2 folders:

Code: Select all

/var/log/elasticsearch
/var/log/logstash

I'm interested in seeing what specifically is going on with these services between the hours of 02:00 and 05:00 PST.

[cgutierr]

Here are the logs you requested.

[mcapra]

This is your problem:

Code: Select all

{:timestamp=>"2018-02-13T03:39:34.175000-0800", :message=>"UDP listener died", :exception=>#<SocketError: bind: name or service not known>, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:160:in `bind'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-2.0.5/lib/logstash/inputs/udp.rb:67:in `udp_listener'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-2.0.5/lib/logstash/inputs/udp.rb:50:in `run'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:342:in `inputworker'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:336:in `start_input'"], :level=>:warn}

Which, after reviewing your Logstash configuration, might be the result of malformed configurations:

Code: Select all

    tcp {
        port => 6514    
        type => syslog
    }
    
    udp {
        port => 6514
        type => syslog
    }

For the type directive, the value needs to be in quotes. Try changing these configurations like so:

Code: Select all

    tcp {
        port => 6514    
        type => 'syslog'
    }
    
    udp {
        port => 6514
        type => 'syslog'
    }

And see if that helps solve the problem.

[cgutierr]

I fixed that configuration with single quotation marks and there is no change. I am still getting a message saying "UDP listener died".

[kyang]

With the profile you sent before, the logstash-confd file says Created Thu, 21 Dec 2017

Can you run this command and the output?

Code: Select all

cat /usr/local/nagioslogserver/logstash/etc/conf.d/*

[cgutierr]

Attached is the configurations for our NLS.

[scottwilkerson]

I fixed that configuration with single quotation marks and there is no change. I am still getting a message saying "UDP listener died".

Are these still every 5 seconds like they were before modifying the configuration?

Yes, logstash and elasticsearch are both running and have been running. This seems to only happen during the early morning hours as well, say 02:00 to 05:00 PST where network devices do not check in.

This also stood out to me, if the problem generally is only occurring in this timeframe, is there some maintenance that is happening? One of the problems using UDP packets is that if a packet gets dropped in transmission, it will be lost forever.

[cgutierr]

Ok, I have been monitoring for few days now and it seems entries in the "Not Sending List" are valid entries due to times of inactivity. Thank you for all your help. Please close the ticket.

[scottwilkerson]

Good to hear you could validate this!