HTTP Error 500 when clicking Cluster Status and Index Status

[tim620]

We are testing a 2 node log server cluster. Under "Admin", I noticed when I click the "Cluster Status" and the "Index Status", I get blank page with the HTTP error 500.

[pbroste]

Hello @tim620

Thanks for reaching out, want to verify the status to see how things are looking:

Code: Select all

curl -XGET 'localhost:9200/_cluster/health?pretty'

Code: Select all

curl -XGET 'localhost:9200/_cluster/health?level=indices&pretty' | head -50

Code: Select all

curl -XGET 'localhost:9200/_cluster/health?level=shards&pretty' | head -50

Verify the service status on both:

Code: Select all

systemctl status logstash

Code: Select all

systemctl status elasticsearch

Code: Select all

systemctl status httpd   #or apache2 depending on distro

Restart services by:

Code: Select all

systemctl restart logstash

Code: Select all

systemctl restart elasticsearch

Code: Select all

systemctl restart httpd   #or apache2 depending on distro

Please send over the System Profile via Private Message if the issue persists.

/usr/local/nagioslogserver/scripts/profile.sh

This will create /tmp/system-profile.tar.gz.

Note that this file can be very large and may require the 'split' command

Thanks,
Perry

[pbroste]

Hello @tim620

Thanks for sending over the System Profile along with details. In review we see

[cluster.routing.allocation.decider] [ae33e651-c4d1-4e51-8e0e-918994531f93] low disk watermark [85%] exceeded on [X4qm-ZCbQgi5wu2TI8o0JQ][77cbfa21-b292-4c2c-b954-468d171c1139] free: 60.9gb[12.4%], replicas will not be assigned to this node

The disk quota hit limit, node(s) is running out of disk space, which is halting the replica assignment process. The mountpoint is:

[INFO ][env] [ae33e651-c4d1-4e51-8e0e-918994531f93] using [1] data paths, mounts [[/ (/dev/mapper/centos-root)]], net usable_space [60.2gb], net total_space [488gb], types [ext4] mounted to: /dev/mapper/centos-root 489G 408G 61G 88% /

This mountpoint will either need to be expanded or mounted to a different mountpoint.

• https://assets.nagios.com/downloads/nag ... M-Disk.pdf
• Or see the attached document for steps to change the mountpoint

When you have time you may want to look at the following syslog collection in your Configuration > Inputs; that is not working. We see that 'syslog listener' has died listening to '0.0.0.0' on port 514 <exception=>#<SocketError: bind: name or service not known>.

Please let us know how things are going and if you have further questions,
Perry

[pbroste]

Hello @tim620

Thanks for following up, and you are correct that we should not see a 500 error from apache on render.

Want to provide a support document that details 'Status Yellow' and options to adjust the watermark threshold.

• https://support.nagios.com/kb/article/n ... h-469.html

Also want to take a look at the Apache logs to see what is going on there:

Code: Select all

tar -czvf /tmp/apachelogs.tar.gz /var/log/httpd/ /var/log/apache2/     # one will error

Please [pm] the '/tmp/apachelog.tar.gz over to me when you get a chance.

Thanks,
Perry