Nagios Support Forum

Hello Folks,

I am unable to login to Nagios Log Server using nagiosadmin account credentials, and I am sure that the credentials are correct.

I get a error:
"Could not get user. Specified ID may be wrong."

But it hangs for a while before throwing the error.

Regards,
Swapnil

Log in to the NLS server via SSH and ensure that the processes are running:

[root@IWKSEUWPSSNLS01 ~]# service logstash status
Logstash Daemon (pid 10014) is running...
[root@IWKSEUWPSSNLS01 ~]# service elasticsearch status
elasticsearch (pid 10579) is running...
[root@IWKSEUWPSSNLS01 ~]#

Additionally, can you share the outputs of these commands executed from the CLI of your Nagios Log Server machine as well as which version of Nagios Log Server you are currently using:

[root@IWKSEUWPSSNLS01 ~]# curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
{
"cluster_name" : "0418383b-07ee-4161-ad5b-e7d7e3b1e9d8",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 1730,
"active_shards" : 1730,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 1732,
"number_of_pending_tasks" : 7,
"number_of_in_flight_fetch" : 0
}
[root@IWKSEUWPSSNLS01 ~]# curl -XGET 'http://localhost:9200/nagioslogserver/u ... h?size=100'
{"error":"SearchPhaseExecutionException[Failed to execute phase [query_fetch], all shards failed]","status":503}[root@IWKSEUWPSSNLS01 ~]# tail -n 100 /var/log/elasticsearch/*.log
==> /var/log/elasticsearch/0418383b-07ee-4161-ad5b-e7d7e3b1e9d8_index_indexing_slowlog.log <==

==> /var/log/elasticsearch/0418383b-07ee-4161-ad5b-e7d7e3b1e9d8_index_search_slowlog.log <==

==> /var/log/elasticsearch/0418383b-07ee-4161-ad5b-e7d7e3b1e9d8.log <==
[2017-07-11 17:34:36,781][DEBUG][action.search.type ] [2f526f36-8df4-4c76-b884-13a523776c3b] All shards failed for phase: [query_fetch]
org.elasticsearch.action.NoShardAvailableActionException: [nagioslogserver][0] null
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.java:160)
at org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchAction.java:57)
at org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchAction.java:47)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:104)
at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:43)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
at org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:98)
at org.elasticsearch.client.FilterClient.execute(FilterClient.java:66)
at org.elasticsearch.rest.BaseRestHandler$HeadersAndContextCopyClient.execute(BaseRestHandler.java:92)
at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:338)
at org.elasticsearch.rest.action.search.RestSearchAction.handleRequest(RestSearchAction.java:84)
at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:53)
at org.elasticsearch.rest.RestController.executeHandler(RestController.java:225)
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:170)
at org.elasticsearch.http.HttpServer.internalDispatchRequest(HttpServer.java:121)
at org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:83)
at org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:327)
at org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:63)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.messageReceived(HttpPipeliningHandler.java:60)
at org.elasticsearch.common.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.elasticsearch.common.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.elasticsearch.common.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2017-07-11 17:34:36,829][WARN ][index.engine ] [2f526f36-8df4-4c76-b884-13a523776c3b] [nagioslogserver][0] failed to sync translog
[2017-07-11 17:34:36,830][WARN ][indices.cluster ] [2f526f36-8df4-4c76-b884-13a523776c3b] [[nagioslogserver][0]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [nagioslogserver][0] failed to recover shard
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:297)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.translog.TranslogCorruptedException: translog corruption while reading from stream
at org.elasticsearch.index.translog.ChecksummedTranslogStream.read(ChecksummedTranslogStream.java:72)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:267)
... 4 more
Caused by: org.elasticsearch.ElasticsearchException: failed to read [alert][AVngWPIvPJ2p6-nmJ3af]
at org.elasticsearch.index.translog.Translog$Index.readFrom(Translog.java:522)
at org.elasticsearch.index.translog.ChecksummedTranslogStream.read(ChecksummedTranslogStream.java:68)
... 5 more
Caused by: org.elasticsearch.ElasticsearchIllegalArgumentException: No version type match [95]
at org.elasticsearch.index.VersionType.fromValue(VersionType.java:307)
at org.elasticsearch.index.translog.Translog$Index.readFrom(Translog.java:519)
... 6 more
[2017-07-11 17:34:36,830][WARN ][cluster.action.shard ] [2f526f36-8df4-4c76-b884-13a523776c3b] [nagioslogserver][0] received shard failed for [nagioslogserver][0], node[TR13rkKNSLSUzVR9olXgcg], [P], s[INITIALIZING], indexUUID [vWVQ1AuGSzWK6ba5Imvm-Q], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[nagioslogserver][0] failed to recover shard]; nested: TranslogCorruptedException[translog corruption while reading from stream]; nested: ElasticsearchException[failed to read [alert][AVngWPIvPJ2p6-nmJ3af]]; nested: ElasticsearchIllegalArgumentException[No version type match [95]]; ]]
[2017-07-11 17:34:36,912][WARN ][index.engine ] [2f526f36-8df4-4c76-b884-13a523776c3b] [nagioslogserver][0] failed to sync translog
[2017-07-11 17:34:36,913][WARN ][indices.cluster ] [2f526f36-8df4-4c76-b884-13a523776c3b] [[nagioslogserver][0]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [nagioslogserver][0] failed to recover shard
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:297)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.translog.TranslogCorruptedException: translog corruption while reading from stream
at org.elasticsearch.index.translog.ChecksummedTranslogStream.read(ChecksummedTranslogStream.java:72)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:267)
... 4 more
Caused by: org.elasticsearch.ElasticsearchException: failed to read [alert][AVngWPIvPJ2p6-nmJ3af]
at org.elasticsearch.index.translog.Translog$Index.readFrom(Translog.java:522)
at org.elasticsearch.index.translog.ChecksummedTranslogStream.read(ChecksummedTranslogStream.java:68)
... 5 more
Caused by: org.elasticsearch.ElasticsearchIllegalArgumentException: No version type match [95]
at org.elasticsearch.index.VersionType.fromValue(VersionType.java:307)
at org.elasticsearch.index.translog.Translog$Index.readFrom(Translog.java:519)
... 6 more
[2017-07-11 17:34:36,913][WARN ][cluster.action.shard ] [2f526f36-8df4-4c76-b884-13a523776c3b] [nagioslogserver][0] received shard failed for [nagioslogserver][0], node[TR13rkKNSLSUzVR9olXgcg], [P], s[INITIALIZING], indexUUID [vWVQ1AuGSzWK6ba5Imvm-Q], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[nagioslogserver][0] failed to recover shard]; nested: TranslogCorruptedException[translog corruption while reading from stream]; nested: ElasticsearchException[failed to read [alert][AVngWPIvPJ2p6-nmJ3af]]; nested: ElasticsearchIllegalArgumentException[No version type match [95]]; ]]
[root@IWKSEUWPSSNLS01 ~]#

Hmm, lots of things can corrupt the translog but that's typically related to disk space issues.

Round 2 of commands:
And please attach a full copy of /var/log/elasticsearch/0418383b-07ee-4161-ad5b-e7d7e3b1e9d8.log to your post.

[root@IWKSEUWPSSNLS01 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
rootfs 591G 411G 150G 74% /
udev 15G 112K 15G 1% /dev
tmpfs 15G 0 15G 0% /dev/shm
/dev/xvde1 591G 411G 150G 74% /
none 15G 0 15G 0% /dev/shm
[root@IWKSEUWPSSNLS01 ~]# free -m
total used free shared buffers cached
Mem: 30100 29460 640 0 62 11723
-/+ buffers/cache: 17674 12425
Swap: 255 0 255
[root@IWKSEUWPSSNLS01 ~]# service elasticsearch restart
Stopping elasticsearch: [ OK ]
Starting elasticsearch: [ OK ]

Edit: nevermind, I guess my reply didn't get posted in time.

My next guess is out of disk space or out of memory:

Well,there's also df -i, which gives different info than either of the previous df commands.

[root@IWKSEUWPSSNLS01 ~]# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
rootfs 39321600 204010 39117590 1% /
udev 3847735 549 3847186 1% /dev
tmpfs 3852881 1 3852880 1% /dev/shm
/dev/xvde1 39321600 204010 39117590 1% /
none 3852881 1 3852880 1% /dev/shm