No logs visible in NLS =>failed to parse [timestamp]

[WillemDH]

Hello,

I configured our Infoblox device to send syslog messages to our NLS. This immediately worked fine, but this morning I noticed NLS did not show any entries for our Infoblox device since 00:59:59. As I feared (after th problem with our esx servers) that the Infoblox had stopped sending, I started looking the Infoblox side, but after doing a tcpdump on the dedicated port I made for our Infoblox syslog messages, it seemed the syslog messages were still flowing in.

In the NLS dashboards, the messages from our Infoblox are not visible however.

EDIT: It seem the Infoblox is not the only device that stopped sending at 00:59:59. I added my Nagios production server too yesterday and it seems starting from 00:59:59 there is no trace of any log from my Nagios server anymore...

Why would the NLS suddenly stop processing syslog messages from several devices?? Tried re-applying configuration, but didn't help.

Checked the logstash log:

Code: Select all

{:timestamp=>"2015-01-27T21:49:33.617000+0100", :message=>"Using milestone 1 input plugin 'syslog'. This plugin should work, but would benefit from use by folks like you. Please let us kno$
{:timestamp=>"2015-01-27T21:49:33.699000+0100", :message=>"Using milestone 2 input plugin 'tcp'. This plugin should be stable, but if you see strange behavior, please let us know! For more$
{:timestamp=>"2015-01-28T13:00:48.245000+0100", :message=>"syslog udp listener died", :address=>"0.0.0.0:5544", :exception=>#<SocketError: recvfrom: name or service not known>, :backtrace=$
{:timestamp=>"2015-01-28T13:00:48.251000+0100", :message=>"syslog udp listener died", :address=>"0.0.0.0:514", :exception=>#<SocketError: recvfrom: name or service not known>, :backtrace=>$

The elasticsearch log shows more interesting info, check this piece out:

Code: Select all

org.elasticsearch.index.mapper.MapperParsingException: failed to parse [timestamp]
        at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:414)
        at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:648)
        at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:501)
        at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:534)
        at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:483)
        at org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java:376)
        at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:430)
        at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:158)
        at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:522)
        at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:421)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.mapper.MapperParsingException: failed to parse date field [Jan 28 13:30:54], tried both date format [dateOptionalTime], and timestamp number with locale []
        at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:610)
        at org.elasticsearch.index.mapper.core.DateFieldMapper.innerParseCreateField(DateFieldMapper.java:538)
        at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:223)
        at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:404)
        ... 12 more
Caused by: java.lang.IllegalArgumentException: Invalid format: "Jan 28 13:30:54"
        at org.elasticsearch.common.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:754)
        at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:604)
        ... 15 more

It seems he has troubles parsing the timestamp?? As far as I know nothing changed at 00:59:59 timewise...? Rechecked date, hwclock and phpdate and all seem correct.

Please advice.

Grtz

Willem

[tmcdonald]

What Logserver version are you on? There was an issue up to 2015R1.1 that dealt with logs being delayed severely.

Also, are the logs coming in in English or another language? Sometimes the date can cause issues if, for example, "Apr" for April is instead "Abr" which is "Abril", Spanish for April.

[WillemDH]

Trevor,

I upgraded yesterday morning to 2015R1.2b. The logs sent from my Nagios XI server and from Infoblox, both English devices / OS'es. The problems started today at 00:59:59. I can see the logs coming in with tcpdump....

Grtz

Willem

[WillemDH]

Do I have 10 separate email support tickets for Nagios Log Server? If so, could this be moved to ticket system, as it kind of cripples my NLS setup.

Grtz

[tmcdonald]

Go ahead and do that. As a rule we don't really discuss account specifics on the forum, but we can update you in the ticket.

[WillemDH]

Any news on this? I've been looking through some posts:
https://github.com/elasticsearch/elasti ... ssues/6156
https://github.com/elasticsearch/elasti ... /issues/22
http://stackoverflow.com/questions/2525 ... sticsearch
http://www.elasticsearch.org/guide/en/e ... ormat.html

My knowledge of NLS however is too limited to start making custom date formats. I'm still stuck on this.

[scottwilkerson]

I touched this in the ticket you have open, checking to see if possibly locale settings on this machine or sending machine is an issue, but I am going to lock this thread as to not have the same issue in multiple places.