No logs visible in NLS =>failed to parse [timestamp]
Posted: Wed Jan 28, 2015 4:53 am
Hello,
I configured our Infoblox device to send syslog messages to our NLS. This immediately worked fine, but this morning I noticed NLS did not show any entries for our Infoblox device since 00:59:59. As I feared (after th problem with our esx servers) that the Infoblox had stopped sending, I started looking the Infoblox side, but after doing a tcpdump on the dedicated port I made for our Infoblox syslog messages, it seemed the syslog messages were still flowing in.
In the NLS dashboards, the messages from our Infoblox are not visible however.
EDIT: It seem the Infoblox is not the only device that stopped sending at 00:59:59. I added my Nagios production server too yesterday and it seems starting from 00:59:59 there is no trace of any log from my Nagios server anymore...
Why would the NLS suddenly stop processing syslog messages from several devices?? Tried re-applying configuration, but didn't help.
Checked the logstash log:
The elasticsearch log shows more interesting info, check this piece out:
It seems he has troubles parsing the timestamp?? As far as I know nothing changed at 00:59:59 timewise...? Rechecked date, hwclock and phpdate and all seem correct.
Please advice.
Grtz
Willem
I configured our Infoblox device to send syslog messages to our NLS. This immediately worked fine, but this morning I noticed NLS did not show any entries for our Infoblox device since 00:59:59. As I feared (after th problem with our esx servers) that the Infoblox had stopped sending, I started looking the Infoblox side, but after doing a tcpdump on the dedicated port I made for our Infoblox syslog messages, it seemed the syslog messages were still flowing in.
In the NLS dashboards, the messages from our Infoblox are not visible however.
EDIT: It seem the Infoblox is not the only device that stopped sending at 00:59:59. I added my Nagios production server too yesterday and it seems starting from 00:59:59 there is no trace of any log from my Nagios server anymore...
Why would the NLS suddenly stop processing syslog messages from several devices?? Tried re-applying configuration, but didn't help.
Checked the logstash log:
Code: Select all
{:timestamp=>"2015-01-27T21:49:33.617000+0100", :message=>"Using milestone 1 input plugin 'syslog'. This plugin should work, but would benefit from use by folks like you. Please let us kno$
{:timestamp=>"2015-01-27T21:49:33.699000+0100", :message=>"Using milestone 2 input plugin 'tcp'. This plugin should be stable, but if you see strange behavior, please let us know! For more$
{:timestamp=>"2015-01-28T13:00:48.245000+0100", :message=>"syslog udp listener died", :address=>"0.0.0.0:5544", :exception=>#<SocketError: recvfrom: name or service not known>, :backtrace=$
{:timestamp=>"2015-01-28T13:00:48.251000+0100", :message=>"syslog udp listener died", :address=>"0.0.0.0:514", :exception=>#<SocketError: recvfrom: name or service not known>, :backtrace=>$
Code: Select all
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [timestamp]
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:414)
at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:648)
at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:501)
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:534)
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:483)
at org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java:376)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:430)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:158)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:522)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:421)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.mapper.MapperParsingException: failed to parse date field [Jan 28 13:30:54], tried both date format [dateOptionalTime], and timestamp number with locale []
at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:610)
at org.elasticsearch.index.mapper.core.DateFieldMapper.innerParseCreateField(DateFieldMapper.java:538)
at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:223)
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:404)
... 12 more
Caused by: java.lang.IllegalArgumentException: Invalid format: "Jan 28 13:30:54"
at org.elasticsearch.common.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:754)
at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:604)
... 15 more
Please advice.
Grtz
Willem
