HTTP Error 500 when clicking Cluster Status and Index Status

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
Locked
tim620
Posts: 4
Joined: Tue May 07, 2019 1:37 pm

HTTP Error 500 when clicking Cluster Status and Index Status

Post by tim620 »

We are testing a 2 node log server cluster. Under "Admin", I noticed when I click the "Cluster Status" and the "Index Status", I get blank page with the HTTP error 500.
User avatar
pbroste
Posts: 1288
Joined: Tue Jun 01, 2021 1:27 pm

Re: HTTP Error 500 when clicking Cluster Status and Index St

Post by pbroste »

Hello @tim620

Thanks for reaching out, want to verify the status to see how things are looking:

Code: Select all

curl -XGET 'localhost:9200/_cluster/health?pretty'

Code: Select all

curl -XGET 'localhost:9200/_cluster/health?level=indices&pretty' | head -50

Code: Select all

curl -XGET 'localhost:9200/_cluster/health?level=shards&pretty' | head -50
Verify the service status on both:

Code: Select all

systemctl status logstash

Code: Select all

systemctl status elasticsearch

Code: Select all

systemctl status httpd   #or apache2 depending on distro
Restart services by:

Code: Select all

systemctl restart logstash

Code: Select all

systemctl restart elasticsearch

Code: Select all

systemctl restart httpd   #or apache2 depending on distro
Please send over the System Profile via Private Message if the issue persists.
/usr/local/nagioslogserver/scripts/profile.sh

This will create /tmp/system-profile.tar.gz.

Note that this file can be very large and may require the 'split' command
Thanks,
Perry
User avatar
pbroste
Posts: 1288
Joined: Tue Jun 01, 2021 1:27 pm

Re: HTTP Error 500 when clicking Cluster Status and Index St

Post by pbroste »

Hello @tim620

Thanks for sending over the System Profile along with details. In review we see
[cluster.routing.allocation.decider] [ae33e651-c4d1-4e51-8e0e-918994531f93] low disk watermark [85%] exceeded on [X4qm-ZCbQgi5wu2TI8o0JQ][77cbfa21-b292-4c2c-b954-468d171c1139] free: 60.9gb[12.4%], replicas will not be assigned to this node
The disk quota hit limit, node(s) is running out of disk space, which is halting the replica assignment process. The mountpoint is:
[INFO ][env] [ae33e651-c4d1-4e51-8e0e-918994531f93] using [1] data paths, mounts [[/ (/dev/mapper/centos-root)]], net usable_space [60.2gb], net total_space [488gb], types [ext4] mounted to: /dev/mapper/centos-root 489G 408G 61G 88% /
This mountpoint will either need to be expanded or mounted to a different mountpoint. When you have time you may want to look at the following syslog collection in your Configuration > Inputs; that is not working. We see that 'syslog listener' has died listening to '0.0.0.0' on port 514 <exception=>#<SocketError: bind: name or service not known>.

Please let us know how things are going and if you have further questions,
Perry
You do not have the required permissions to view the files attached to this post.
User avatar
pbroste
Posts: 1288
Joined: Tue Jun 01, 2021 1:27 pm

Re: HTTP Error 500 when clicking Cluster Status and Index St

Post by pbroste »

Hello @tim620

Thanks for following up, and you are correct that we should not see a 500 error from apache on render.

Want to provide a support document that details 'Status Yellow' and options to adjust the watermark threshold.
Also want to take a look at the Apache logs to see what is going on there:

Code: Select all

tar -czvf /tmp/apachelogs.tar.gz /var/log/httpd/ /var/log/apache2/     # one will error
Please [pm] the '/tmp/apachelog.tar.gz over to me when you get a chance.

Thanks,
Perry
Locked