Page 2 of 2
Re: Hourly Sending Check on Nagios Log Server 2.0
Posted: Tue Feb 13, 2018 10:52 am
by cgutierr
Yes, logstash and elasticsearch are both running and have been running. This seems to only happen during the early morning hours as well, say 02:00 to 05:00 PST where network devices do not check in.
Re: Hourly Sending Check on Nagios Log Server 2.0
Posted: Tue Feb 13, 2018 11:27 am
by mcapra
Do you have historical Logstash and ElasticSearch logs you could provide? Would it be possible for you to submit an attachment of everything in these 2 folders:
Code: Select all
/var/log/elasticsearch
/var/log/logstash
I'm interested in seeing what specifically is going on with these services between the hours of 02:00 and 05:00 PST.
Re: Hourly Sending Check on Nagios Log Server 2.0
Posted: Tue Feb 13, 2018 11:47 am
by cgutierr
Here are the logs you requested.
Re: Hourly Sending Check on Nagios Log Server 2.0
Posted: Tue Feb 13, 2018 12:20 pm
by mcapra
This is your problem:
Code: Select all
{:timestamp=>"2018-02-13T03:39:34.175000-0800", :message=>"UDP listener died", :exception=>#<SocketError: bind: name or service not known>, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:160:in `bind'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-2.0.5/lib/logstash/inputs/udp.rb:67:in `udp_listener'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-2.0.5/lib/logstash/inputs/udp.rb:50:in `run'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:342:in `inputworker'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:336:in `start_input'"], :level=>:warn}
Which, after reviewing your Logstash configuration,
might be the result of malformed configurations:
Code: Select all
tcp {
port => 6514
type => syslog
}
udp {
port => 6514
type => syslog
}
For the type directive, the value needs to be in quotes. Try changing these configurations like so:
Code: Select all
tcp {
port => 6514
type => 'syslog'
}
udp {
port => 6514
type => 'syslog'
}
And see if that helps solve the problem.
Re: Hourly Sending Check on Nagios Log Server 2.0
Posted: Tue Feb 13, 2018 1:06 pm
by cgutierr
I fixed that configuration with single quotation marks and there is no change. I am still getting a message saying "UDP listener died".
Re: Hourly Sending Check on Nagios Log Server 2.0
Posted: Tue Feb 13, 2018 4:17 pm
by kyang
With the profile you sent before, the logstash-confd file says
Created Thu, 21 Dec 2017
Can you run this command and the output?
Code: Select all
cat /usr/local/nagioslogserver/logstash/etc/conf.d/*
Re: Hourly Sending Check on Nagios Log Server 2.0
Posted: Wed Feb 21, 2018 3:20 pm
by cgutierr
Attached is the configurations for our NLS.
Re: Hourly Sending Check on Nagios Log Server 2.0
Posted: Thu Feb 22, 2018 11:16 am
by scottwilkerson
cgutierr wrote:I fixed that configuration with single quotation marks and there is no change. I am still getting a message saying "UDP listener died".
Are these still every 5 seconds like they were before modifying the configuration?
cgutierr wrote:Yes, logstash and elasticsearch are both running and have been running. This seems to only happen during the early morning hours as well, say 02:00 to 05:00 PST where network devices do not check in.
This also stood out to me, if the problem generally is only occurring in this timeframe, is there some maintenance that is happening? One of the problems using UDP packets is that if a packet gets dropped in transmission, it will be lost forever.
Re: Hourly Sending Check on Nagios Log Server 2.0
Posted: Mon Feb 26, 2018 2:15 pm
by cgutierr
Ok, I have been monitoring for few days now and it seems entries in the "Not Sending List" are valid entries due to times of inactivity. Thank you for all your help. Please close the ticket.
Re: Hourly Sending Check on Nagios Log Server 2.0
Posted: Mon Feb 26, 2018 2:18 pm
by scottwilkerson
Good to hear you could validate this!