Alerting

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
Locked
CFT6Server
Posts: 506
Joined: Wed Apr 15, 2015 4:21 pm

Alerting

Post by CFT6Server »

I've been noticing that the alerting doesn't seem to be working correctly. I would check the alerting tab and find the alert output at 0. When I click "to show the alert in dashboard" it shows events. The only way to fix this is to reset the jobs in the command subsystem. Then it starts to work again, but only for a short little while. I've done this reset a few times already. We are running version 1.4.

Thanks
Top
tmcdonald
Posts: 9117
Joined: Mon Sep 23, 2013 8:40 am

Re: Alerting

Post by tmcdonald »

Would updating be a possibility? The current version is 2.2 and was released yesterday. I can't see anything in the changelog that specifically mentions this, but many things have changed since 1.4 and this might have been resolved.
Former Nagios employee
Top
CFT6Server
Posts: 506
Joined: Wed Apr 15, 2015 4:21 pm

Re: Alerting

Post by CFT6Server »

Yes I am willing to update the log servers. They are under pretty heavy usage right now, so I will update them after-hours. Does the entire cluster have to be updated at once?
Top
User avatar
hsmith
Agent Smith
Posts: 3539
Joined: Thu Jul 30, 2015 11:09 am
Location: 127.0.0.1
Contact:
Contact hsmith

Re: Alerting

Post by hsmith »

CFT6Server wrote:Yes I am willing to update the log servers. They are under pretty heavy usage right now, so I will update them after-hours. Does the entire cluster have to be updated at once?
I highly recommend you do them all at the same time. Having multiple versions of the packages that it uses could cause a lot of unpredictable issues.
Former Nagios Employee.
me.
Top
CFT6Server
Posts: 506
Joined: Wed Apr 15, 2015 4:21 pm

Re: Alerting

Post by CFT6Server »

Ok, that means I would have to shutdown the cluster and all inputs and upgrade them at once?
Top
User avatar
hsmith
Agent Smith
Posts: 3539
Joined: Thu Jul 30, 2015 11:09 am
Location: 127.0.0.1
Contact:
Contact hsmith

Re: Alerting

Post by hsmith »

CFT6Server wrote:Ok, that means I would have to shutdown the cluster and all inputs and upgrade them at once?
Proper instructions can be found here. I don't believe you would have to shut down the cluster as long as you just run these scripts in a rolling out fashion like the article describes.
Former Nagios Employee.
me.
Top
CFT6Server
Posts: 506
Joined: Wed Apr 15, 2015 4:21 pm

Re: Alerting

Post by CFT6Server »

I tested this on our test instance that is also the same old version and got the following error in elasticsearch. Elasticsearch is no longer able to run...

Code: Select all

[2015-08-23 19:50:59,240][INFO ][node                     ] [16fcc224-849a-405f-bfaf-8321387b7294] version[1.6.0], pid[1966], build[cdd3ac4/2015-06-09T13:36:34Z]
[2015-08-23 19:50:59,240][INFO ][node                     ] [16fcc224-849a-405f-bfaf-8321387b7294] initializing ...
[2015-08-23 19:50:59,253][INFO ][plugins                  ] [16fcc224-849a-405f-bfaf-8321387b7294] loaded [knapsack-1.5.2.0-f340ad1], sites [head]
[2015-08-23 19:50:59,299][INFO ][env                      ] [16fcc224-849a-405f-bfaf-8321387b7294] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [140.7gb], net total_space [172.2gb], types [rootfs]
[2015-08-23 19:51:01,738][ERROR][gateway.local.state.shards] [16fcc224-849a-405f-bfaf-8321387b7294] failed to read local state (started shards), exiting...
org.elasticsearch.ElasticsearchIllegalStateException: Could not find a state file to recover from among [[id:2189, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2189], [id:661, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-661], [id:2180, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2180], [id:659, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-659], [id:2177, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2177], [id:660, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-660], [id:2186, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2186], [id:256, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-256], [id:2183, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2183]]
        at org.elasticsearch.gateway.local.state.meta.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:300)
        at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.loadShardStateInfo(LocalGatewayShardsState.java:188)
        at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.loadShardsStateInfo(LocalGatewayShardsState.java:173)
        at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.<init>(LocalGatewayShardsState.java:66)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:54)
        at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
        at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:837)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
        at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
        at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
        at org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42)
        at org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66)
        at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85)
        at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
        at org.elasticsearch.common.inject.FactoryProxy.get(FactoryProxy.java:52)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:837)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
        at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
        at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
        at org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42)
        at org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66)
        at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85)
        at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:837)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
        at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
        at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:200)
        at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:193)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:830)
        at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:193)
        at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:175)
        at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
        at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:93)
        at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70)
        at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:59)
        at org.elasticsearch.node.internal.InternalNode.<init>(InternalNode.java:207)
        at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:77)
        at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:245)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)
[2015-08-23 19:51:01,896][ERROR][gateway.local.state.shards] [16fcc224-849a-405f-bfaf-8321387b7294] failed to read local state (started shards), exiting...
org.elasticsearch.ElasticsearchIllegalStateException: Could not find a state file to recover from among [[id:2189, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2189], [id:661, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-661], [id:2180, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2180], [id:659, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-659], [id:2177, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2177], [id:660, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-660], [id:2186, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2186], [id:256, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-256], [id:2183, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2183]]
        at org.elasticsearch.gateway.local.state.meta.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:300)
        at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.loadShardStateInfo(LocalGatewayShardsState.java:188)
        at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.loadShardsStateInfo(LocalGatewayShardsState.java:173)
        at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.<init>(LocalGatewayShardsState.java:66)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:54)
        at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
        at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:837)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
        at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
        at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
        at org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42)
        at org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66)
        at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85)
        at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
        at org.elasticsearch.common.inject.FactoryProxy.get(FactoryProxy.java:52)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:837)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
        at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
        at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:200)
        at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:193)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:830)
        at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:193)
        at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:175)
        at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
        at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:93)
        at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70)
        at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:59)
        at org.elasticsearch.node.internal.InternalNode.<init>(InternalNode.java:207)
        at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:77)
        at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:245)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)
[2015-08-23 19:51:01,901][ERROR][gateway.local.state.shards] [16fcc224-849a-405f-bfaf-8321387b7294] failed to read local state (started shards), exiting...
org.elasticsearch.ElasticsearchIllegalStateException: Could not find a state file to recover from among [[id:2189, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2189], [id:661, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-661], [id:2180, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2180], [id:659, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-659], [id:2177, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2177], [id:660, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-660], [id:2186, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2186], [id:256, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-256], [id:2183, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2183]]
        at org.elasticsearch.gateway.local.state.meta.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:300)
        at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.loadShardStateInfo(LocalGatewayShardsState.java:188)
        at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.loadShardsStateInfo(LocalGatewayShardsState.java:173)
        at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.<init>(LocalGatewayShardsState.java:66)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:54)
        at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
        at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:837)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
        at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
        at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:200)
        at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:193)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:830)
        at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:193)
        at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:175)
        at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
        at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:93)
        at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70)
        at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:59)
        at org.elasticsearch.node.internal.InternalNode.<init>(InternalNode.java:207)
        at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:77)
        at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:245)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)
[2015-08-23 19:51:01,908][ERROR][bootstrap                ] Exception
org.elasticsearch.common.inject.CreationException: Guice creation errors:

1) Error injecting constructor, org.elasticsearch.ElasticsearchIllegalStateException: Could not find a state file to recover from among [[id:2189, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2189], [id:661, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-661], [id:2180, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2180], [id:659, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-659], [id:2177, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2177], [id:660, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-660], [id:2186, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2186], [id:256, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-256], [id:2183, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2183]]
  at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.<init>(Unknown Source)
  while locating org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState
    for parameter 3 at org.elasticsearch.gateway.local.LocalGateway.<init>(Unknown Source)
  while locating org.elasticsearch.gateway.local.LocalGateway
  while locating org.elasticsearch.gateway.Gateway
    for parameter 1 at org.elasticsearch.gateway.GatewayService.<init>(Unknown Source)
  while locating org.elasticsearch.gateway.GatewayService
Caused by: org.elasticsearch.ElasticsearchIllegalStateException: Could not find a state file to recover from among [[id:2189, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2189], [id:661, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-661], [id:2180, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2180], [id:659, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-659], [id:2177, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2177], [id:660, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-660], [id:2186, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2186], [id:256, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-256], [id:2183, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2183]]
        at org.elasticsearch.gateway.local.state.meta.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:300)
        at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.loadShardStateInfo(LocalGatewayShardsState.java:188)
        at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.loadShardsStateInfo(LocalGatewayShardsState.java:173)
        at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.<init>(LocalGatewayShardsState.java:66)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:54)
        at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
        at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:837)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
        at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
        at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
        at org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42)
        at org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66)
        at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85)
        at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
        at org.elasticsearch.common.inject.FactoryProxy.get(FactoryProxy.java:52)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:837)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
        at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
        at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
        at org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42)
        at org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66)
        at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85)
        at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:837)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
        at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
        at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:200)
        at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:193)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:830)
        at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:193)
        at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:175)
        at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
        at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:93)
        at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70)
        at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:59)
        at org.elasticsearch.node.internal.InternalNode.<init>(InternalNode.java:207)
        at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:77)
        at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:245)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)

2) Error injecting constructor, org.elasticsearch.ElasticsearchIllegalStateException: Could not find a state file to recover from among [[id:2189, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2189], [id:661, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-661], [id:2180, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2180], [id:659, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-659], [id:2177, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2177], [id:660, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-660], [id:2186, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2186], [id:256, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-256], [id:2183, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2183]]
  at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.<init>(Unknown Source)
  while locating org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState
    for parameter 3 at org.elasticsearch.gateway.local.LocalGateway.<init>(Unknown Source)
  while locating org.elasticsearch.gateway.local.LocalGateway
  while locating org.elasticsearch.gateway.Gateway
Caused by: org.elasticsearch.ElasticsearchIllegalStateException: Could not find a state file to recover from among [[id:2189, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2189], [id:661, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-661], [id:2180, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2180], [id:659, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-659], [id:2177, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2177], [id:660, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-660], [id:2186, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2186], [id:256, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-256], [id:2183, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2183]]
        at org.elasticsearch.gateway.local.state.meta.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:300)
        at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.loadShardStateInfo(LocalGatewayShardsState.java:188)
        at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.loadShardsStateInfo(LocalGatewayShardsState.java:173)
        at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.<init>(LocalGatewayShardsState.java:66)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:54)
        at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
        at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:837)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
        at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
        at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
        at org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42)
        at org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66)
        at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85)
        at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
        at org.elasticsearch.common.inject.FactoryProxy.get(FactoryProxy.java:52)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:837)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
        at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
        at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:200)
        at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:193)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:830)
        at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:193)
        at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:175)
        at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
        at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:93)
        at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70)
        at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:59)
        at org.elasticsearch.node.internal.InternalNode.<init>(InternalNode.java:207)
        at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:77)
        at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:245)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)

3) Error injecting constructor, org.elasticsearch.ElasticsearchIllegalStateException: Could not find a state file to recover from among [[id:2189, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2189], [id:661, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-661], [id:2180, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2180], [id:659, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-659], [id:2177, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2177], [id:660, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-660], [id:2186, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2186], [id:256, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-256], [id:2183, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2183]]
  at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.<init>(Unknown Source)
  while locating org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState
Caused by: org.elasticsearch.ElasticsearchIllegalStateException: Could not find a state file to recover from among [[id:2189, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2189], [id:661, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-661], [id:2180, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2180], [id:659, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-659], [id:2177, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2177], [id:660, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-660], [id:2186, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2186], [id:256, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-256], [id:2183, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2183]]
        at org.elasticsearch.gateway.local.state.meta.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:300)
        at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.loadShardStateInfo(LocalGatewayShardsState.java:188)
        at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.loadShardsStateInfo(LocalGatewayShardsState.java:173)
        at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.<init>(LocalGatewayShardsState.java:66)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:54)
        at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
        at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:837)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
        at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
        at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:200)
        at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:193)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:830)
        at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:193)
        at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:175)
        at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
        at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:93)
        at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70)
        at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:59)
        at org.elasticsearch.node.internal.InternalNode.<init>(InternalNode.java:207)
        at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:77)
        at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:245)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)

3 errors
        at org.elasticsearch.common.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:344)
        at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:178)
        at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
        at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:93)
        at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70)
        at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:59)
        at org.elasticsearch.node.internal.InternalNode.<init>(InternalNode.java:207)
        at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:77)
        at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:245)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)
Top
User avatar
Box293
Too Basu
Posts: 5126
Joined: Sun Feb 07, 2010 10:55 pm
Location: Deniliquin, Australia
Contact:
Contact Box293

Re: Alerting

Post by Box293 »

I've seen this before and I believe all you need to do is delete these state files and all will be OK.

Code: Select all

ls -R /usr/local/nagioslogserver/elasticsearch/data/*/nodes/0/indices/nagioslogserver_log/*/state*
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
Top
CFT6Server
Posts: 506
Joined: Wed Apr 15, 2015 4:21 pm

Re: Alerting

Post by CFT6Server »

Thanks. I'll give this a try. Do you know what these state files are for? Just trying to understand what we are removing.
Top
jolson
Attack Rabbit
Posts: 2560
Joined: Thu Feb 12, 2015 12:40 pm

Re: Alerting

Post by jolson »

You can find all of the necessary states as bandit described using 'find'.

Code: Select all

find /usr/local/nagioslogserver/elasticsearch/data/*/nodes/0/indices/nagioslogserver_log/ -iname *state*
Any state files that do not end in .st can be removed. I recommend simply moving them to your home directory just in case they are necessary later.

The state files are used to retain information about the indices that they reside in - this information is loaded into memory when elasticsearch starts up. Occasionally the upgrade process will cause the state files to not work - there are several github reports about this particular problem:
https://github.com/elastic/elasticsearch/issues/11590
https://github.com/elastic/elasticsearch/issues/10565
Twits Blog
Show me a man who lives alone and has a perpetually clean kitchen, and 8 times out of 9 I'll show you a man with detestable spiritual qualities.
Top
Locked

Return to “Nagios Log Server”