Changed IP of Nagios Syslog from dhcp to static (same IP) and now it gives Waiting for Database Startup

[Brad_Shows]

Changed IP of Nagios Syslog from dhcp to static (same IP) and now it gives Waiting for Database Startup after shutting down and placing server in rack and powering on.

http://10.1.0.48/nagioslogserver/waiting
In the GUI we have:
"Waiting for Database Startup
It looks like your local elasticsearch service is starting."

elasticsearch is running (maybe)
sudo service elasticsearch status
● elasticsearch.service - LSB: Starts elasticsearch
Loaded: loaded (/etc/init.d/elasticsearch; generated)
Active: active (exited) since Thu 2025-03-06 10:25:38 JST; 3min 3s ago
Docs: man:systemd-sysv-generator(8)
Process: 774 ExecStart=/etc/init.d/elasticsearch start (code=exited, status=0/SUCCESS)
CPU: 17.183s

/var/log/logstash
tail logstash.log gives the following.
{:timestamp=>"2025-03-05T20:26:23.822000-0500", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://localhost:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}

any ideas on how to resolve this? Thanks!

[kg2857]

Active: active (exited) since Thu 2025-03-06 10:25:38 JST; 3min 3s ago
ES isn't running.
Just a guess but take a look at cluster_hosts and see what address/hostname is there.

[Brad_Shows]

I thought about that, and even commented one out and it behaved the same:

cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 debian

# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

I even commented out the IPv6 ones and still the same...

[kg2857]

cluster_hosts is not /etc/hosts.

[Brad_Shows]

Had no idea... This is new install it was running on the bench for a few weeks:
cat /usr/local/nagioslogserver/var/cluster_hosts
localhost
10.1.0.48
fe80:0:0:0:7a2b:cbff:fe12:f6aa

[kg2857]

I'd probably try removing localhost and the ipv6 address and restart logstash, assuming this is a single host system (which isn't recommended by nagios btw.) I'm assuming 10.1.0.48 and the ipv6 address are the same host.

[Brad_Shows]

Tried with this

# localhost
10.1.0.48
# fe80:0:0:0:7a2b:cbff:fe12:f6aa

Still no change. I even tried with just localhost.

{:timestamp=>"2025-03-06T00:03:21.862000-0500", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://localhost:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}

Could it be because port 9200 looks closed?

netstat -ntlp
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:587 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:8461 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
tcp6 0 0 :::5544 :::* LISTEN -
tcp6 0 0 :::5546 :::* LISTEN -
tcp6 0 0 :::5693 :::* LISTEN -
tcp6 0 0 :::2057 :::* LISTEN -
tcp6 0 0 :::2056 :::* LISTEN -
tcp6 0 0 :::3515 :::* LISTEN -
tcp6 0 0 :::22 :::* LISTEN -
tcp6 0 0 :::514 :::* LISTEN -

[kg2857]

If ES is down 9200 won't be open.

[Brad_Shows]

Did a reinstall by this
viewtopic.php?p=335692&hilit=reinstall+ ... er#p335692

Now I just think the ports are not right because I have no logs coming in. I need port 514 open, seems it is not.
I went through this to allow <1024 ports, https://assets.nagios.com/downloads/nag ... Server.pdf
but still no logs.

[jmichaelson]

A couple of questions:

What distro are you using?

Can you re-run that netstat command as root?

Finally systemctl start elasticsearch if its not running, and provide the output of journalctl -xeu elasticsearch if it fails