No hosts logging anymore

[TheBassman]

I have been fighting with NLS throughout my trial period, and now I have 3 days left to make an informed decision to purchase.

The current problem is that no logs from any of the 37 hosts I have configured are showing up. This is a 2 node cluster. Cluster status shown 0 docs since sometime last week. I have stopped my maintenance jobs around that time. I needed to open logs going back in time for searching.

I have restarted the services/servers multiple times. Last week, and continually throughout my trial, both web interfaces become unresponsive, and CPU levels peg at near 100%

Both servers are receiving logs, visible using the tcpdump command. I know that this info will be requested, so I'm adding it below:

Logserver1:
tail -n20 /var/log/logstash/logstash.log
{:timestamp=>"2015-05-11T11:31:06.096000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"May 11 11:31:06", :exception=>java.lang.IllegalArgumentException: Invalid format: "May 11 11:31:06", :level=>:warn}
{:timestamp=>"2015-05-11T11:31:06.090000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"May 11 11:31:06", :exception=>java.lang.IllegalArgumentException: Invalid format: "May 11 11:31:06", :level=>:warn}
{:timestamp=>"2015-05-11T11:31:06.106000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"May 11 11:31:06", :exception=>java.lang.IllegalArgumentException: Invalid format: "May 11 11:31:06", :level=>:warn}
{:timestamp=>"2015-05-11T11:31:06.096000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"May 11 11:31:06", :exception=>java.lang.IllegalArgumentException: Invalid format: "May 11 11:31:06", :level=>:warn}
{:timestamp=>"2015-05-11T11:31:06.102000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"May 11 11:31:06", :exception=>java.lang.IllegalArgumentException: Invalid format: "May 11 11:31:06", :level=>:warn}
{:timestamp=>"2015-05-11T11:31:06.120000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"May 11 11:31:06", :exception=>java.lang.IllegalArgumentException: Invalid format: "May 11 11:31:06", :level=>:warn}
{:timestamp=>"2015-05-11T11:31:06.129000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"May 11 11:31:06", :exception=>java.lang.IllegalArgumentException: Invalid format: "May 11 11:31:06", :level=>:warn}
{:timestamp=>"2015-05-11T11:31:06.130000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"May 11 11:31:06", :exception=>java.lang.IllegalArgumentException: Invalid format: "May 11 11:31:06", :level=>:warn}
{:timestamp=>"2015-05-11T11:31:06.135000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"May 11 11:31:06", :exception=>java.lang.IllegalArgumentException: Invalid format: "May 11 11:31:06", :level=>:warn}
{:timestamp=>"2015-05-11T11:31:06.136000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"May 11 11:31:06", :exception=>java.lang.IllegalArgumentException: Invalid format: "May 11 11:31:06", :level=>:warn}
{:timestamp=>"2015-05-11T11:31:13.379000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"May 11 11:31:13", :exception=>java.lang.IllegalArgumentException: Invalid format: "May 11 11:31:13", :level=>:warn}
{:timestamp=>"2015-05-11T11:31:13.392000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"May 11 11:31:13", :exception=>java.lang.IllegalArgumentException: Invalid format: "May 11 11:31:13", :level=>:warn}
{:timestamp=>"2015-05-11T11:31:13.393000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"May 11 11:31:13", :exception=>java.lang.IllegalArgumentException: Invalid format: "May 11 11:31:13", :level=>:warn}
{:timestamp=>"2015-05-11T11:31:13.377000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"May 11 11:31:13", :exception=>java.lang.IllegalArgumentException: Invalid format: "May 11 11:31:13", :level=>:warn}
{:timestamp=>"2015-05-11T11:31:13.399000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"May 11 11:31:13", :exception=>java.lang.IllegalArgumentException: Invalid format: "May 11 11:31:13", :level=>:warn}
{:timestamp=>"2015-05-11T11:31:13.403000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"May 11 11:31:13", :exception=>java.lang.IllegalArgumentException: Invalid format: "May 11 11:31:13", :level=>:warn}
{:timestamp=>"2015-05-11T11:31:13.396000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"May 11 11:31:13", :exception=>java.lang.IllegalArgumentException: Invalid format: "May 11 11:31:13", :level=>:warn}
{:timestamp=>"2015-05-11T11:31:13.401000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"May 11 11:31:13", :exception=>java.lang.IllegalArgumentException: Invalid format: "May 11 11:31:13", :level=>:warn}
{:timestamp=>"2015-05-11T11:31:16.094000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"May 11 11:31:16", :exception=>java.lang.IllegalArgumentException: Invalid format: "May 11 11:31:16", :level=>:warn}
{:timestamp=>"2015-05-11T11:31:16.096000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"May 11 11:31:16", :exception=>java.lang.IllegalArgumentException: Invalid format: "May 11 11:31:16", :level=>:warn}


Logserver2:
tail -n20 /var/log/logstash/logstash.log
{:timestamp=>"2015-05-11T11:22:34.806000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"May 11 11:22:34", :exception=>java.lang.IllegalArgumentException: Invalid format: "May 11 11:22:34", :level=>:warn}
{:timestamp=>"2015-05-11T11:23:25.891000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"May 11 11:23:25", :exception=>java.lang.IllegalArgumentException: Invalid format: "May 11 11:23:25", :level=>:warn}
{:timestamp=>"2015-05-11T11:23:34.847000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"May 11 11:23:34", :exception=>java.lang.IllegalArgumentException: Invalid format: "May 11 11:23:34", :level=>:warn}
{:timestamp=>"2015-05-11T11:24:25.930000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"May 11 11:24:25", :exception=>java.lang.IllegalArgumentException: Invalid format: "May 11 11:24:25", :level=>:warn}
{:timestamp=>"2015-05-11T11:24:34.878000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"May 11 11:24:34", :exception=>java.lang.IllegalArgumentException: Invalid format: "May 11 11:24:34", :level=>:warn}
{:timestamp=>"2015-05-11T11:25:25.973000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"May 11 11:25:25", :exception=>java.lang.IllegalArgumentException: Invalid format: "May 11 11:25:25", :level=>:warn}
{:timestamp=>"2015-05-11T11:25:34.914000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"May 11 11:25:34", :exception=>java.lang.IllegalArgumentException: Invalid format: "May 11 11:25:34", :level=>:warn}
{:timestamp=>"2015-05-11T11:25:56.006000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"May 11 11:25:55", :exception=>java.lang.IllegalArgumentException: Invalid format: "May 11 11:25:55", :level=>:warn}
{:timestamp=>"2015-05-11T11:26:26.023000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"May 11 11:26:26", :exception=>java.lang.IllegalArgumentException: Invalid format: "May 11 11:26:26", :level=>:warn}
{:timestamp=>"2015-05-11T11:26:34.944000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"May 11 11:26:34", :exception=>java.lang.IllegalArgumentException: Invalid format: "May 11 11:26:34", :level=>:warn}
{:timestamp=>"2015-05-11T11:27:00.470000-0600", :message=>"syslog udp listener died", :address=>"0.0.0.0:5544", :exception=>#<SocketError: recvfrom: name or service not known>, :backtrace=>["/usr/local/nagioslogserver/logstash/lib/logstash/inputs/syslog.rb:119:in `udp_listener'", "org/jruby/RubyKernel.javain `loop'", "/usr/local/nagioslogserver/logstash/lib/logstash/inputs/syslog.rb:118:in `udp_listener'", "/usr/local/nagioslogserver/logstash/lib/logstash/inputs/syslog.rb:76:in `run'"], :level=>:warn}
:timestamp=>"2015-05-11T11:27:00.470000-0600", :message=>"syslog udp listener died", :address=>"0.0.0.0:1514", :exception=>#<SocketError: recvfrom: name or service not known>, :backtrace=>["/usr/local/nagioslogserver/logstash/lib/logstash/inputs/syslog.rb:119:in `udp_listener'", "org/jruby/RubyKernel.javain `loop'", "/usr/local/nagioslogserver/logstash/lib/logstash/inputs/syslog.rb:118:in `udp_listener'", "/usr/local/nagioslogserver/logstash/lib/logstash/inputs/syslog.rb:76:in `run'"], :level=>:warn}{:timestamp=>"2015-05-11T11:27:14.222000-0600", :message=>"Using milestone 1 input plugin 'syslog'. This plugin should work, but would benefit from use by folks like you. Please let us know if you find bugs or have suggestions on how to improve this plugin. For more information on plugin milestones, see http://logstash.net/docs/1.4.2/plugin-milestones", :level=>:warn}
{:timestamp=>"2015-05-11T11:27:14.276000-0600", :message=>"Using milestone 2 input plugin 'tcp'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.4.2/plugin-milestones", :level=>:warn}
{:timestamp=>"2015-05-11T11:27:26.385000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"May 11 11:27:26", :exception=>java.lang.IllegalArgumentException: Invalid format: "May 11 11:27:26", :level=>:warn}
{:timestamp=>"2015-05-11T11:27:34.985000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"May 11 11:27:34", :exception=>java.lang.IllegalArgumentException: Invalid format: "May 11 11:27:34", :level=>:warn}
{:timestamp=>"2015-05-11T11:28:26.643000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"May 11 11:28:26", :exception=>java.lang.IllegalArgumentException: Invalid format: "May 11 11:28:26", :level=>:warn}
{:timestamp=>"2015-05-11T11:28:35.030000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"May 11 11:28:35", :exception=>java.lang.IllegalArgumentException: Invalid format: "May 11 11:28:35", :level=>:warn}
{:timestamp=>"2015-05-11T11:29:06.185000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"May 11 11:29:06", :exception=>java.lang.IllegalArgumentException: Invalid format: "May 11 11:29:06", :level=>:warn}
{:timestamp=>"2015-05-11T11:29:26.211000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"May 11 11:29:26", :exception=>java.lang.IllegalArgumentException: Invalid format: "May 11 11:29:26", :level=>:warn}
{:timestamp=>"2015-05-11T11:29:35.028000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"May 11 11:29:35", :exception=>java.lang.IllegalArgumentException: Invalid format: "May 11 11:29:35", :level=>:warn}

ConfigFiles:
cat /usr/local/nagioslogserver/logstash/etc/conf.d/*
#
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Mon, 11 May 2015 11:27:08 -0600
#

#
# Global inputs
#

input {
syslog {
type => 'syslog'
port => 5544
}
tcp {
type => 'eventlog'
port => 3515
codec => json {
charset => 'CP1252'
}
}
tcp {
type => 'import_raw'
tags => 'import_raw'
port => 2056
}
tcp {
type => 'import_json'
tags => 'import_json'
port => 2057
codec => json
}
syslog {
type => 'syslog'
port => 1514
}
}

#
# Local inputs
#


#
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Mon, 11 May 2015 11:27:08 -0600
#

#
# Global filters
#

filter {
if [program] == 'apache_access' {
grok {
match => [ 'message', '%{COMBINEDAPACHELOG}']
}
date {
match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z' ]
}
mutate {
replace => [ 'type', 'apache_access' ]
convert => [ 'bytes', 'integer' ]
convert => [ 'response', 'integer' ]
}
}

if [program] == 'apache_error' {
grok {
match => [ 'message', '\[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\] \[%{WORD:class}\] \[%{WORD:originator} %{IP:clientip}\] %{GREEDYDATA:errmsg}']
}
mutate {
replace => [ 'type', 'apache_error' ]
}
}
if [program] == "mysqld_log" {
grok {
match => [ "message", "^%{NUMBER:date} *%{NOTSPACE:time}"]
}
mutate {
replace => [ "type", "mysqld_log" ]
}
}
}

#
# Local filters
#


#
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Mon, 11 May 2015 11:27:08 -0600
#

#
# Required output for Nagios Log Server
#

output {
elasticsearch {
cluster => '0d7ccb5f-4e48-4831-a718-2bc5b1764ad5'
host => 'localhost'
index_type => '%{type}'
node_name => '532ed9d8-59de-4ae8-9f44-66b2db06d1c4'
protocol => 'transport'
workers => 4
}
}

#
# Global outputs
#



#
# Local outputs
#

[TheBassman]

Also, I just ran the upgrade to NLS r1.4, and still nothing.

[jolson]

How much RAM is in this system, and is there any indication that the kernel is reaping the processes?

Code: Select all

free -m

Code: Select all

grep -i 'out of memory' /var/log/messages

Thanks for the inclusion of the information - it saved us a couple of back-and-forths.

What's interesting is that no logs are showing up at all. What happens if you restart logstash on both nodes? Do logs start coming in - even temporarily?

Code: Select all

service logstash restart

[TheBassman]

I have the logstash restarting every hour as it was constantly stopping.

Code: Select all

free -m
             total       used       free     shared    buffers     cached
Mem:          8001       7814        187          0        186       2494
-/+ buffers/cache:       5134       2867
Swap:          255          0        255

No "Out of memory" messages. (I even looked in the log rotated files.

[jolson]

CPU levels peg at near 100%
free: 187

I am thinking that these boxes could use more resources. Any chance you can up them to 16GB of RAM and add a few processor cores? It's interesting that your UDP listeners are dying, it's possible that they are getting backed up because data is not being indexed quickly enough.

When you restart the logstash services on your nodes, do you get a large spike in log activity until the services die? That is my guess at this point.

[TheBassman]

Yes they do spike.

I can add resources, but right now the nodes are not doing anything, and CPUs are low, because nothing is getting indexed. When I go to the "Hoe tab" they show 1 host being logged, but the graphs are blank.

[jolson]

Please try adding resources and restarting logstash on both nodes:

Code: Select all

service logstash restart

I expect you'll see a jump in log activity. Please tail the logstash logs while this is happening:

Code: Select all

tail -f /var/log/logstash/logstash.log

Both servers are receiving logs, visible using the tcpdump command.

As long as the logs are arriving in the proper format, this must mean that there's something off with logstash/elasticsearch, and the system resources are my guess.