Nagios Support Forum

Support for Nagios products and services
https://support.nagios.com/forum/

Alerting

https://support.nagios.com/forum/viewtopic.php?t=34274

Page 1 of 2

Alerting

Posted: Wed Aug 19, 2015 7:39 pm
by CFT6Server
I've been noticing that the alerting doesn't seem to be working correctly. I would check the alerting tab and find the alert output at 0. When I click "to show the alert in dashboard" it shows events. The only way to fix this is to reset the jobs in the command subsystem. Then it starts to work again, but only for a short little while. I've done this reset a few times already. We are running version 1.4.

Thanks

Re: Alerting

Posted: Thu Aug 20, 2015 10:49 am
by tmcdonald
Would updating be a possibility? The current version is 2.2 and was released yesterday. I can't see anything in the changelog that specifically mentions this, but many things have changed since 1.4 and this might have been resolved.

Re: Alerting

Posted: Thu Aug 20, 2015 1:06 pm
by CFT6Server
Yes I am willing to update the log servers. They are under pretty heavy usage right now, so I will update them after-hours. Does the entire cluster have to be updated at once?

Re: Alerting

Posted: Thu Aug 20, 2015 1:10 pm
by hsmith
CFT6Server wrote:Yes I am willing to update the log servers. They are under pretty heavy usage right now, so I will update them after-hours. Does the entire cluster have to be updated at once?
I highly recommend you do them all at the same time. Having multiple versions of the packages that it uses could cause a lot of unpredictable issues.

Re: Alerting

Posted: Thu Aug 20, 2015 1:42 pm
by CFT6Server
Ok, that means I would have to shutdown the cluster and all inputs and upgrade them at once?

Re: Alerting

Posted: Thu Aug 20, 2015 1:58 pm
by hsmith
CFT6Server wrote:Ok, that means I would have to shutdown the cluster and all inputs and upgrade them at once?
Proper instructions can be found here. I don't believe you would have to shut down the cluster as long as you just run these scripts in a rolling out fashion like the article describes.

Re: Alerting

Posted: Sun Aug 23, 2015 9:57 pm
by CFT6Server
I tested this on our test instance that is also the same old version and got the following error in elasticsearch. Elasticsearch is no longer able to run...

Code: Select all

[2015-08-23 19:50:59,240][INFO ][node                     ] [16fcc224-849a-405f-bfaf-8321387b7294] version[1.6.0], pid[1966], build[cdd3ac4/2015-06-09T13:36:34Z]
[2015-08-23 19:50:59,240][INFO ][node                     ] [16fcc224-849a-405f-bfaf-8321387b7294] initializing ...
[2015-08-23 19:50:59,253][INFO ][plugins                  ] [16fcc224-849a-405f-bfaf-8321387b7294] loaded [knapsack-1.5.2.0-f340ad1], sites [head]
[2015-08-23 19:50:59,299][INFO ][env                      ] [16fcc224-849a-405f-bfaf-8321387b7294] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [140.7gb], net total_space [172.2gb], types [rootfs]
[2015-08-23 19:51:01,738][ERROR][gateway.local.state.shards] [16fcc224-849a-405f-bfaf-8321387b7294] failed to read local state (started shards), exiting...
org.elasticsearch.ElasticsearchIllegalStateException: Could not find a state file to recover from among [[id:2189, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2189], [id:661, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-661], [id:2180, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2180], [id:659, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-659], [id:2177, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2177], [id:660, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-660], [id:2186, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2186], [id:256, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-256], [id:2183, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2183]]
        at org.elasticsearch.gateway.local.state.meta.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:300)
        at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.loadShardStateInfo(LocalGatewayShardsState.java:188)
        at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.loadShardsStateInfo(LocalGatewayShardsState.java:173)
        at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.<init>(LocalGatewayShardsState.java:66)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:54)
        at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
        at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:837)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
        at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
        at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
        at org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42)
        at org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66)
        at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85)
        at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
        at org.elasticsearch.common.inject.FactoryProxy.get(FactoryProxy.java:52)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:837)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
        at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
        at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
        at org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42)
        at org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66)
        at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85)
        at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:837)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
        at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
        at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:200)
        at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:193)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:830)
        at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:193)
        at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:175)
        at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
        at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:93)
        at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70)
        at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:59)
        at org.elasticsearch.node.internal.InternalNode.<init>(InternalNode.java:207)
        at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:77)
        at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:245)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)
[2015-08-23 19:51:01,896][ERROR][gateway.local.state.shards] [16fcc224-849a-405f-bfaf-8321387b7294] failed to read local state (started shards), exiting...
org.elasticsearch.ElasticsearchIllegalStateException: Could not find a state file to recover from among [[id:2189, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2189], [id:661, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-661], [id:2180, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2180], [id:659, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-659], [id:2177, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2177], [id:660, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-660], [id:2186, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2186], [id:256, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-256], [id:2183, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2183]]
        at org.elasticsearch.gateway.local.state.meta.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:300)
        at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.loadShardStateInfo(LocalGatewayShardsState.java:188)
        at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.loadShardsStateInfo(LocalGatewayShardsState.java:173)
        at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.<init>(LocalGatewayShardsState.java:66)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:54)
        at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
        at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:837)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
        at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
        at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
        at org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42)
        at org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66)
        at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85)
        at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
        at org.elasticsearch.common.inject.FactoryProxy.get(FactoryProxy.java:52)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:837)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
        at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
        at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:200)
        at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:193)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:830)
        at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:193)
        at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:175)
        at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
        at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:93)
        at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70)
        at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:59)
        at org.elasticsearch.node.internal.InternalNode.<init>(InternalNode.java:207)
        at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:77)
        at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:245)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)
[2015-08-23 19:51:01,901][ERROR][gateway.local.state.shards] [16fcc224-849a-405f-bfaf-8321387b7294] failed to read local state (started shards), exiting...
org.elasticsearch.ElasticsearchIllegalStateException: Could not find a state file to recover from among [[id:2189, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2189], [id:661, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-661], [id:2180, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2180], [id:659, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-659], [id:2177, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2177], [id:660, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-660], [id:2186, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2186], [id:256, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-256], [id:2183, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2183]]
        at org.elasticsearch.gateway.local.state.meta.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:300)
        at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.loadShardStateInfo(LocalGatewayShardsState.java:188)
        at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.loadShardsStateInfo(LocalGatewayShardsState.java:173)
        at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.<init>(LocalGatewayShardsState.java:66)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:54)
        at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
        at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:837)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
        at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
        at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:200)
        at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:193)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:830)
        at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:193)
        at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:175)
        at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
        at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:93)
        at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70)
        at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:59)
        at org.elasticsearch.node.internal.InternalNode.<init>(InternalNode.java:207)
        at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:77)
        at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:245)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)
[2015-08-23 19:51:01,908][ERROR][bootstrap                ] Exception
org.elasticsearch.common.inject.CreationException: Guice creation errors:

1) Error injecting constructor, org.elasticsearch.ElasticsearchIllegalStateException: Could not find a state file to recover from among [[id:2189, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2189], [id:661, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-661], [id:2180, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2180], [id:659, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-659], [id:2177, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2177], [id:660, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-660], [id:2186, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2186], [id:256, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-256], [id:2183, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2183]]
  at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.<init>(Unknown Source)
  while locating org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState
    for parameter 3 at org.elasticsearch.gateway.local.LocalGateway.<init>(Unknown Source)
  while locating org.elasticsearch.gateway.local.LocalGateway
  while locating org.elasticsearch.gateway.Gateway
    for parameter 1 at org.elasticsearch.gateway.GatewayService.<init>(Unknown Source)
  while locating org.elasticsearch.gateway.GatewayService
Caused by: org.elasticsearch.ElasticsearchIllegalStateException: Could not find a state file to recover from among [[id:2189, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2189], [id:661, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-661], [id:2180, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2180], [id:659, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-659], [id:2177, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2177], [id:660, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-660], [id:2186, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2186], [id:256, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-256], [id:2183, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2183]]
        at org.elasticsearch.gateway.local.state.meta.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:300)
        at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.loadShardStateInfo(LocalGatewayShardsState.java:188)
        at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.loadShardsStateInfo(LocalGatewayShardsState.java:173)
        at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.<init>(LocalGatewayShardsState.java:66)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:54)
        at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
        at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:837)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
        at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
        at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
        at org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42)
        at org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66)
        at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85)
        at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
        at org.elasticsearch.common.inject.FactoryProxy.get(FactoryProxy.java:52)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:837)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
        at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
        at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
        at org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42)
        at org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66)
        at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85)
        at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:837)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
        at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
        at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:200)
        at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:193)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:830)
        at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:193)
        at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:175)
        at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
        at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:93)
        at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70)
        at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:59)
        at org.elasticsearch.node.internal.InternalNode.<init>(InternalNode.java:207)
        at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:77)
        at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:245)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)

2) Error injecting constructor, org.elasticsearch.ElasticsearchIllegalStateException: Could not find a state file to recover from among [[id:2189, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2189], [id:661, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-661], [id:2180, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2180], [id:659, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-659], [id:2177, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2177], [id:660, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-660], [id:2186, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2186], [id:256, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-256], [id:2183, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2183]]
  at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.<init>(Unknown Source)
  while locating org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState
    for parameter 3 at org.elasticsearch.gateway.local.LocalGateway.<init>(Unknown Source)
  while locating org.elasticsearch.gateway.local.LocalGateway
  while locating org.elasticsearch.gateway.Gateway
Caused by: org.elasticsearch.ElasticsearchIllegalStateException: Could not find a state file to recover from among [[id:2189, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2189], [id:661, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-661], [id:2180, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2180], [id:659, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-659], [id:2177, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2177], [id:660, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-660], [id:2186, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2186], [id:256, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-256], [id:2183, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2183]]
        at org.elasticsearch.gateway.local.state.meta.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:300)
        at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.loadShardStateInfo(LocalGatewayShardsState.java:188)
        at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.loadShardsStateInfo(LocalGatewayShardsState.java:173)
        at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.<init>(LocalGatewayShardsState.java:66)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:54)
        at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
        at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:837)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
        at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
        at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
        at org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42)
        at org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66)
        at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85)
        at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
        at org.elasticsearch.common.inject.FactoryProxy.get(FactoryProxy.java:52)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:837)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
        at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
        at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:200)
        at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:193)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:830)
        at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:193)
        at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:175)
        at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
        at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:93)
        at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70)
        at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:59)
        at org.elasticsearch.node.internal.InternalNode.<init>(InternalNode.java:207)
        at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:77)
        at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:245)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)

3) Error injecting constructor, org.elasticsearch.ElasticsearchIllegalStateException: Could not find a state file to recover from among [[id:2189, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2189], [id:661, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-661], [id:2180, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2180], [id:659, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-659], [id:2177, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2177], [id:660, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-660], [id:2186, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2186], [id:256, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-256], [id:2183, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2183]]
  at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.<init>(Unknown Source)
  while locating org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState
Caused by: org.elasticsearch.ElasticsearchIllegalStateException: Could not find a state file to recover from among [[id:2189, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2189], [id:661, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-661], [id:2180, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2180], [id:659, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-659], [id:2177, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2177], [id:660, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-660], [id:2186, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2186], [id:256, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-256], [id:2183, legacy:true, file:/usr/local/nagioslogserver/elasticsearch/data/87f95151-7003-42fc-a76a-bc101723dfc0/nodes/0/indices/nagioslogserver_log/3/_state/state-2183]]
        at org.elasticsearch.gateway.local.state.meta.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:300)
        at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.loadShardStateInfo(LocalGatewayShardsState.java:188)
        at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.loadShardsStateInfo(LocalGatewayShardsState.java:173)
        at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.<init>(LocalGatewayShardsState.java:66)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:54)
        at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
        at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:837)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
        at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
        at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:200)
        at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:193)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:830)
        at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:193)
        at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:175)
        at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
        at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:93)
        at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70)
        at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:59)
        at org.elasticsearch.node.internal.InternalNode.<init>(InternalNode.java:207)
        at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:77)
        at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:245)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)

3 errors
        at org.elasticsearch.common.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:344)
        at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:178)
        at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
        at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:93)
        at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70)
        at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:59)
        at org.elasticsearch.node.internal.InternalNode.<init>(InternalNode.java:207)
        at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:77)
        at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:245)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)

Re: Alerting

Posted: Sun Aug 23, 2015 10:04 pm
by Box293
I've seen this before and I believe all you need to do is delete these state files and all will be OK.

Code: Select all

ls -R /usr/local/nagioslogserver/elasticsearch/data/*/nodes/0/indices/nagioslogserver_log/*/state*

Re: Alerting

Posted: Mon Aug 24, 2015 10:48 am
by CFT6Server
Thanks. I'll give this a try. Do you know what these state files are for? Just trying to understand what we are removing.

Re: Alerting

Posted: Mon Aug 24, 2015 11:07 am
by jolson
You can find all of the necessary states as bandit described using 'find'.

Code: Select all

find /usr/local/nagioslogserver/elasticsearch/data/*/nodes/0/indices/nagioslogserver_log/ -iname *state*
Any state files that do not end in .st can be removed. I recommend simply moving them to your home directory just in case they are necessary later.

The state files are used to retain information about the indices that they reside in - this information is loaded into memory when elasticsearch starts up. Occasionally the upgrade process will cause the state files to not work - there are several github reports about this particular problem:
https://github.com/elastic/elasticsearch/issues/11590
https://github.com/elastic/elasticsearch/issues/10565
All times are UTC-05:00
Page 1 of 2

Powered by phpBB® Forum Software © phpBB Limited