After reboot receive Waiting for Database Startup screen

[scheived]

Rebooted nagios Log server, now I'm getting the Waiting for Database Start Screen.

systemctl status Elasticsearch
● elasticsearch.service - LSB: This service manages the elasticsearch daemon
Loaded: loaded (/etc/rc.d/init.d/elasticsearch; generated)
Active: active (exited) since Tue 2025-02-18 19:34:11 CST; 8s ago

Logs in /var/log/elasticsearch show,
264) Error injecting constructor, java.lang.IllegalStateException: This is a proxy used to support circular references involving constructors. The object we're proxying is not constructed yet. Please wait until after injection has completed to use this object.
at org.elasticsearch.river.routing.RiversRouter.<init>(Unknown Source)
while locating org.elasticsearch.river.routing.RiversRouter
for parameter 3 at org.elasticsearch.river.RiversManager.<init>(Unknown Source)
while locating org.elasticsearch.river.RiversManager
Caused by: java.lang.IllegalStateException: This is a proxy used to support circular references involving constructors. The object we're proxying is not constructed yet. Please wait until after injection has completed to use this object.
at org.elasticsearch.common.inject.internal.ConstructionContext$DelegatingInvocationHandler.invoke(ConstructionContext.java:102)
at com.sun.proxy.$Proxy12.add(Unknown Source)
at org.elasticsearch.river.routing.RiversRouter.<init>(RiversRouter.java:82)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:54)
at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:837)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
at org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42)
at org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66)
at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85)
at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:837)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:200)
at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:193)
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:830)
at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:193)
at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:175)
at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:96)
at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70)
at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:59)
at org.elasticsearch.node.internal.InternalNode.<init>(InternalNode.java:210)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:77)
at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:245)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)

264 errors
at org.elasticsearch.common.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:344)
at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:178)
at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:96)
at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70)
at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:59)
at org.elasticsearch.node.internal.InternalNode.<init>(InternalNode.java:210)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:77)
at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:245)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)

What does this mean?

[scheived]

Here is what the start of the log says,
[2025-02-18 19:45:17,994][INFO ][node ] [1ec25105-6692-4d16-b7a3-f79c06039ae0] version[1.7.6], pid[5251], build[c730b59/2022-04-28T19:47:24Z]
[2025-02-18 19:45:17,995][INFO ][node ] [1ec25105-6692-4d16-b7a3-f79c06039ae0] initializing ...
[2025-02-18 19:45:18,043][INFO ][plugins ] [1ec25105-6692-4d16-b7a3-f79c06039ae0] loaded [knapsack-1.7.3.0-d0ea246], sites []
[2025-02-18 19:45:18,067][INFO ][env ] [1ec25105-6692-4d16-b7a3-f79c06039ae0] using [1] data paths, mounts [[/ (/dev/mapper/cs-root)]], net usable_space [131.3gb], net total_space [196.9gb], types [xfs]
[2025-02-18 19:45:19,919][ERROR][bootstrap ] Exception
org.elasticsearch.common.inject.CreationException: Guice creation errors:

1) Error injecting constructor, org.elasticsearch.ElasticsearchIllegalArgumentException: Failed to resolve address for [fe80:0:0:0:250:56ff:fe87:bccc]
at org.elasticsearch.discovery.zen.ping.ZenPingService.<init>(Unknown Source)
while locating org.elasticsearch.discovery.zen.ping.ZenPingService
for parameter 7 at org.elasticsearch.discovery.zen.ZenDiscovery.<init>(Unknown Source)
while locating org.elasticsearch.discovery.zen.ZenDiscovery
while locating org.elasticsearch.discovery.Discovery
for parameter 2 at org.elasticsearch.discovery.DiscoveryService.<init>(Unknown Source)
while locating org.elasticsearch.discovery.DiscoveryService
for parameter 1 at org.elasticsearch.cluster.service.InternalClusterService.<init>(Unknown Source)
while locating org.elasticsearch.cluster.service.InternalClusterService
while locating org.elasticsearch.cluster.ClusterService
for parameter 1 at org.xbib.elasticsearch.knapsack.KnapsackService.<init>(Unknown Source)
while locating org.xbib.elasticsearch.knapsack.KnapsackService
Caused by: org.elasticsearch.ElasticsearchIllegalArgumentException: Failed to resolve address for [fe80:0:0:0:250:56ff:fe87:bccc]
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.<init>(UnicastZenPing.java:142)
at org.elasticsearch.discovery.zen.ping.ZenPingService.<init>(ZenPingService.java:68)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:54)

[scheived]

Also wanted to add,
java -version: openjdk version "1.8.0_362"
OpenJDK Runtime Environment (build 1.8.0_362-b08)
OpenJDK 64-Bit Server VM (build 25.362-b08, mixed mode)
whereis java: java: /usr/bin/java /usr/lib/java /etc/java /usr/share/java /usr/share/man/man1/java.1.gz
sestatus: SELinux status: disabled
ps -aux | grep -Ei 'java'
nagios 898 10.8 3.7 4149856 606312 ? SNsl 20:02 0:45 /bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -Xmx500m -Xss2048k -Djffi.boot.library.path=/usr/local/nagioslogserver/logstash/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -XX:HeapDumpPath=/usr/local/nagioslogserver/logstash/heapdump.hprof -Xbootclasspath/a:/usr/local/nagioslogserver/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/local/nagioslogserver/logstash/vendor/jruby -Djruby.lib=/usr/local/nagioslogserver/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /usr/local/nagioslogserver/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f /usr/local/nagioslogserver/logstash/etc/conf.d -l /var/log/logstash/logstash.log -w 4
root 3725 0.0 0.0 6400 2304 pts/0 S+ 20:09 0:00 grep --color=auto -Ei java

[scheived]

Tried a server restore, initially it was fine until I rebooted then the same issue. I'm going to try a restore again without rebooting and see if there's data.

[jmichaelson]

This is definitely unusual. Could you provide some system details: Amount of RAM, Linux Distribution and version.
After that, try restarting the elasticsearch service:

Code: Select all

systemctl restart elasticsearch.service

[kg2857]

Is this a single log server and not a cluster?
Have a look at cluster_hosts I believe.Make sure it has the correct address or fqdn.
Delete the ES log and try starting ES, and have a look at the log. Unfortunately, java logs are verbose but rarely informative.