Page 1 of 1
HTTP Error 500 when clicking Cluster Status and Index Status
Posted: Mon Dec 06, 2021 4:02 pm
by tim620
We are testing a 2 node log server cluster. Under "Admin", I noticed when I click the "Cluster Status" and the "Index Status", I get blank page with the HTTP error 500.
Re: HTTP Error 500 when clicking Cluster Status and Index St
Posted: Mon Dec 06, 2021 5:54 pm
by pbroste
Hello
@tim620
Thanks for reaching out, want to verify the status to see how things are looking:
Code: Select all
curl -XGET 'localhost:9200/_cluster/health?pretty'
Code: Select all
curl -XGET 'localhost:9200/_cluster/health?level=indices&pretty' | head -50
Code: Select all
curl -XGET 'localhost:9200/_cluster/health?level=shards&pretty' | head -50
Verify the service status on both:
Code: Select all
systemctl status httpd #or apache2 depending on distro
Restart services by:
Code: Select all
systemctl restart httpd #or apache2 depending on distro
Please send over the System Profile via Private Message if the issue persists.
/usr/local/nagioslogserver/scripts/profile.sh
This will create /tmp/system-profile.tar.gz.
Note that this file can be very large and may require the 'split' command
Thanks,
Perry
Re: HTTP Error 500 when clicking Cluster Status and Index St
Posted: Tue Dec 07, 2021 3:16 pm
by pbroste
Hello
@tim620
Thanks for sending over the System Profile along with details. In review we see
[cluster.routing.allocation.decider] [ae33e651-c4d1-4e51-8e0e-918994531f93] low disk watermark [85%] exceeded on [X4qm-ZCbQgi5wu2TI8o0JQ][77cbfa21-b292-4c2c-b954-468d171c1139] free: 60.9gb[12.4%], replicas will not be assigned to this node
The disk quota hit limit, node(s) is running out of disk space, which is halting the replica assignment process. The mountpoint is:
[INFO ][env] [ae33e651-c4d1-4e51-8e0e-918994531f93] using [1] data paths, mounts [[/ (/dev/mapper/centos-root)]], net usable_space [60.2gb], net total_space [488gb], types [ext4] mounted to: /dev/mapper/centos-root 489G 408G 61G 88% /
This mountpoint will either need to be expanded or mounted to a different mountpoint.
When you have time you may want to look at the following syslog collection in your Configuration > Inputs; that is not working. We see that
'syslog listener' has died listening to '0.0.0.0' on port 514 <exception=>#<SocketError: bind: name or service not known>.
Please let us know how things are going and if you have further questions,
Perry
Re: HTTP Error 500 when clicking Cluster Status and Index St
Posted: Wed Dec 08, 2021 3:47 pm
by pbroste
Hello
@tim620
Thanks for following up, and you are correct that we should not see a 500 error from apache on render.
Want to provide a support document that details 'Status Yellow' and options to adjust the watermark threshold.
Also want to take a look at the Apache logs to see what is going on there:
Code: Select all
tar -czvf /tmp/apachelogs.tar.gz /var/log/httpd/ /var/log/apache2/ # one will error
Please [pm] the '/tmp/apachelog.tar.gz over to me when you get a chance.
Thanks,
Perry