Windows server logs not getting indexed

[TheBassman]

I have several domain controllers that were logging using the nxlog package, but one of them is not getting indexed now.

Nothing changed that I can think of.

Running TCPDUMP on the 3515 shows the traffic getting to one of the 2 instances. In fact I see both DCs traffic (dc01 and dc02), but only one is getting into the log server.

Here is a sample of the tcpdump output (sudo tcpdump port 3515 )
15:55:21.323074 IP dc01.truhearing.com.65188 > nagioslog1.truhearing.com.must-backplane: Flags [.], seq 32760:34220, ack 1, win 2053, length 1460
15:55:21.323090 IP dc01.truhearing.com.65188 > nagioslog1.truhearing.com.must-backplane: Flags [P.], seq 34220:34615, ack 1, win 2053, length 395
15:55:21.323106 IP nagioslog1.truhearing.com.must-backplane > dc01.truhearing.com.65188: Flags [.], ack 34615, win 1980, length 0
15:55:21.323697 IP dc01.truhearing.com.65188 > nagioslog1.truhearing.com.must-backplane: Flags [P.], seq 34615:35782, ack 1, win 2053, length 1167
15:55:21.324390 IP dc01.truhearing.com.65188 > nagioslog1.truhearing.com.must-backplane: Flags [P.], seq 35782:36949, ack 1, win 2053, length 1167
15:55:21.324412 IP nagioslog1.truhearing.com.must-backplane > dc01.truhearing.com.65188: Flags [.], ack 36949, win 1980, length 0
15:55:21.325830 IP dc01.truhearing.com.65188 > nagioslog1.truhearing.com.must-backplane: Flags [P.], seq 36949:38115, ack 1, win 2053, length 1166
15:55:21.325969 IP nagioslog1.truhearing.com.must-backplane > dc01.truhearing.com.65188: Flags [.], ack 38115, win 1980, length 0
15:55:22.156011 IP dc02.truhearing.com.58576 > nagioslog1.truhearing.com.must-backplane: Flags [P.], seq 43415:44827, ack 1, win 2053, length 1412
15:55:22.156052 IP nagioslog1.truhearing.com.must-backplane > dc02.truhearing.com.58576: Flags [.], ack 44827, win 1926, length 0
15:55:22.156461 IP dc02.truhearing.com.58576 > nagioslog1.truhearing.com.must-backplane: Flags [P.], seq 44827:46142, ack 1, win 2053, length 1315
15:55:22.156506 IP nagioslog1.truhearing.com.must-backplane > dc02.truhearing.com.58576: Flags [.], ack 46142, win 1926, length 0
15:55:22.156834 IP dc02.truhearing.com.58576 > nagioslog1.truhearing.com.must-backplane: Flags [P.], seq 46142:47556, ack 1, win 2053, length 1414
15:55:22.156855 IP nagioslog1.truhearing.com.must-backplane > dc02.truhearing.com.58576: Flags [.], ack 47556, win 1926, length 0
15:55:22.157344 IP dc02.truhearing.com.58576 > nagioslog1.truhearing.com.must-backplane: Flags [P.], seq 47556:48944, ack 1, win 2053, length 1388
15:55:22.157366 IP nagioslog1.truhearing.com.must-backplane > dc02.truhearing.com.58576: Flags [.], ack 48944, win 1926, length 0
15:55:22.158080 IP dc02.truhearing.com.58576 > nagioslog1.truhearing.com.must-backplane: Flags [P.], seq 48944:50358, ack 1, win 2053, length 1414
15:55:22.158106 IP nagioslog1.truhearing.com.must-backplane > dc02.truhearing.com.58576: Flags [.], ack 50358, win 1926, length 0

Anyone have any idea what might have happened?

[jolson]

If I'm reading this correctly, you have nxlog installed on both Windows Servers in the same way, and they are both sending to the same NLS port. Is that correct?

If so, I'd like you to collect the following for me from NLS:

Code: Select all

cat /usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf

Code: Select all

tail -n20 /var/log/logstash/logstash.log

Did this server randomly stop getting indexed? It's possible that some Windows Update broke nxlog - have you attempted stopping and restarting nxlog on the server in question? If that doesn't work, you could try reinstalling the nxlog package.

[TheBassman]

Yes, both servers are setup the same. One server just stopped getting indexed. I can see the traffic hitting the NLS using the TCP dump command from that server.

I also have another device that just stopped getting indexed. It is sending a standard syslog message. If I point it to a basic syslog program (ACSyslog) I can watch the output in real time, but when I send it back to NLS on the same port (1514) nothing.. It was working when I set it up a week ago or so.

# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Tue, 10 Mar 2015 16:15:40 -0600
#

#
# Global inputs
#

input {
syslog {
type => 'syslog'
port => 5544
}
tcp {
type => 'import_raw'
tags => 'import_raw'
port => 2056
}
tcp {
type => 'import_json'
tags => 'import_json'
port => 2057
codec => json
}
syslog {
type => 'syslog'
port => 1514
}
syslog {
type => 'syslog'
port => 514
}
tcp {
type => 'eventlog'
port => 3515
codec => json {
charset => 'CP1252'
}
}
}

#
# Local inputs
#


>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Here are the last 20 lines from the logstash.log:

{:timestamp=>"2015-03-17T11:02:23.666000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"Mar 17 11:03:41", :exception=>java.lang.IllegalArgumentException: Invalid format: "Mar 17 11:03:41", :level=>:warn}
{:timestamp=>"2015-03-17T11:02:23.669000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"Mar 17 11:03:41", :exception=>java.lang.IllegalArgumentException: Invalid format: "Mar 17 11:03:41", :level=>:warn}
{:timestamp=>"2015-03-17T11:02:23.676000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"Mar 17 11:03:41", :exception=>java.lang.IllegalArgumentException: Invalid format: "Mar 17 11:03:41", :level=>:warn}
{:timestamp=>"2015-03-17T11:02:23.685000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"Mar 17 11:03:41", :exception=>java.lang.IllegalArgumentException: Invalid format: "Mar 17 11:03:41", :level=>:warn}
{:timestamp=>"2015-03-17T11:02:23.687000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"Mar 17 11:03:41", :exception=>java.lang.IllegalArgumentException: Invalid format: "Mar 17 11:03:41", :level=>:warn}
{:timestamp=>"2015-03-17T11:02:23.694000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"Mar 17 11:03:41", :exception=>java.lang.IllegalArgumentException: Invalid format: "Mar 17 11:03:41", :level=>:warn}
{:timestamp=>"2015-03-17T11:02:23.697000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"Mar 17 11:03:41", :exception=>java.lang.IllegalArgumentException: Invalid format: "Mar 17 11:03:41", :level=>:warn}
{:timestamp=>"2015-03-17T11:02:23.819000-0600", :message=>"syslog udp listener died", :address=>"0.0.0.0:514", :exception=>#<SocketError: bind: name or service not known>, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:160:in `bind'", "/usr/local/nagioslogserver/logstash/lib/logstash/inputs/syslog.rb:116:in `udp_listener'", "/usr/local/nagioslogserver/logstash/lib/logstash/inputs/syslog.rb:76:in `run'"], :level=>:warn}
{:timestamp=>"2015-03-17T11:02:27.606000-0600", :message=>"syslog tcp listener died", :address=>"0.0.0.0:514", :exception=>#<Errno::EACCES: Permission denied - bind(2)>, :backtrace=>["org/jruby/ext/socket/RubyTCPServer.java:124:in `initialize'", "org/jruby/RubyIO.java:852:in `new'", "/usr/local/nagioslogserver/logstash/lib/logstash/inputs/syslog.rb:135:in `tcp_listener'", "/usr/local/nagioslogserver/logstash/lib/logstash/inputs/syslog.rb:90:in `run'"], :level=>:warn}
{:timestamp=>"2015-03-17T11:02:28.823000-0600", :message=>"syslog udp listener died", :address=>"0.0.0.0:514", :exception=>#<SocketError: bind: name or service not known>, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:160:in `bind'", "/usr/local/nagioslogserver/logstash/lib/logstash/inputs/syslog.rb:116:in `udp_listener'", "/usr/local/nagioslogserver/logstash/lib/logstash/inputs/syslog.rb:76:in `run'"], :level=>:warn}
{:timestamp=>"2015-03-17T11:02:29.200000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"Mar 17 11:02:27", :exception=>java.lang.IllegalArgumentException: Invalid format: "Mar 17 11:02:27", :level=>:warn}
{:timestamp=>"2015-03-17T11:02:29.202000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"Mar 17 11:02:27", :exception=>java.lang.IllegalArgumentException: Invalid format: "Mar 17 11:02:27", :level=>:warn}
{:timestamp=>"2015-03-17T11:02:29.203000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"Mar 17 11:02:27", :exception=>java.lang.IllegalArgumentException: Invalid format: "Mar 17 11:02:27", :level=>:warn}
{:timestamp=>"2015-03-17T11:02:29.228000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"Mar 17 11:02:27", :exception=>java.lang.IllegalArgumentException: Invalid format: "Mar 17 11:02:27", :level=>:warn}
{:timestamp=>"2015-03-17T11:02:29.207000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"Mar 17 11:02:27", :exception=>java.lang.IllegalArgumentException: Invalid format: "Mar 17 11:02:27", :level=>:warn}
{:timestamp=>"2015-03-17T11:02:29.230000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"Mar 17 11:02:27", :exception=>java.lang.IllegalArgumentException: Invalid format: "Mar 17 11:02:27", :level=>:warn}
{:timestamp=>"2015-03-17T11:02:29.231000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"Mar 17 11:02:27", :exception=>java.lang.IllegalArgumentException: Invalid format: "Mar 17 11:02:27", :level=>:warn}
{:timestamp=>"2015-03-17T11:02:29.235000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"Mar 17 11:02:27", :exception=>java.lang.IllegalArgumentException: Invalid format: "Mar 17 11:02:27", :level=>:warn}
{:timestamp=>"2015-03-17T11:02:29.237000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"Mar 17 11:02:27", :exception=>java.lang.IllegalArgumentException: Invalid format: "Mar 17 11:02:27", :level=>:warn}
{:timestamp=>"2015-03-17T11:02:29.241000-0600", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"Mar 17 11:02:27", :exception=>java.lang.IllegalArgumentException: Invalid format: "Mar 17 11:02:27", :level=>:warn}

[jolson]

First, I would like to review how this input is defined:

Code: Select all

type => 'eventlog'
port => 3515
codec => json {
charset => 'CP1252'

Is the 'charset' variable something that you need? It is a depreciated feature and may be removed in future versions of Logstash. In addition to this, it's only useful if your log-files are in Latin-1 format, as json is already UTF-8 and able to be parsed by default. If there is no reason to have that charset variable, please remove it via the web interface and Apply your configuration.

If you apply your configuration, please ensure that it took by running the following on each of your nodes:

Code: Select all

cat /usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf

There are some scenarios where the inputs won't apply cluster-wide - we just want to make sure that's not happening.

I see the following log in logstash as well:

Code: Select all

{:timestamp=>"2015-03-17T11:02:23.819000-0600", :message=>"syslog udp listener died", :address=>"0.0.0.0:514", :exception=>#<SocketError: bind: name or service not known>, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:160:in `bind'", "/usr/local/nagioslogserver/logstash/lib/logstash/inputs/syslog.rb:116:in `udp_listener'", "/usr/local/nagioslogserver/logstash/lib/logstash/inputs/syslog.rb:76:in `run'"], :level=>:warn}

It's possible that this is caused because port 514 is a privileged port, and logstash won't listen on privileged ports by default. To correct this, run through the following guide:
http://assets.nagios.com/downloads/nagi ... Server.pdf

Let me know if any of the above works for you. Thanks you very much!