This is a list of commands that can be used to help diagnose issues with Nagios Log Server.
Be careful, some commands can be destructive and can result in data loss!
Generic Information Gathering
Master:
curl 'localhost:9200/_cat/master?v'
Nodes:
curl 'localhost:9200/_cat/nodes?v'
Pending tasks:
curl 'localhost:9200/_cat/pending_tasks?v'
Recovery:
curl -XGET 'localhost:9200/_cat/recovery?v'
Running plugins:
curl 'localhost:9200/_cat/plugins?v'
Cluster basic health:
curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
Check jvm settings:
curl -XGET localhost:9200/_nodes/jvm?pretty
Shard status (use index status instead for clarify, in most cases):
curl -XGET http://localhost:9200/_cat/shards
Cluster state:
curl -XGET 'http://localhost:9200/_cluster/state?pretty'
Shard health (index health works better in most circumstances):
curl -XGET 'http://localhost:9200/_cluster/health/*?level=shards'
Red Cluster Recovery
Check index health:
curl 'localhost:9200/_cluster/health?level=indices&pretty'
Shard list:
curl -s localhost:9200/_cat/shards
Unassigned shards:
curl -s localhost:9200/_cat/shards | grep UNASS
Delete bad index:
curl -XDELETE 'http://localhost:9200/twitter/'
Active Troubleshooting
Tail maintenance logs:
tail -f /usr/local/nagioslogserver/var/jobs.log
Tail poller logs:
tail -f /usr/local/nagioslogserver/var/poller.log
Shut down all nodes:
curl -XPOST 'http://localhost:9200/_shutdown'
Check users:
curl -XGET 'http://localhost:9200/nagioslogserver/_search?type=user&pretty'
Backup Troubleshooting
Check running knapsack export:
curl -XPOST 'http://localhost:9200/_export/state'
Typically jobs.log should be tailed and the backup+maintenance command should be forced from the subsys - jobs.log should complain about a partial snapshot.
Delete a snapshot (versions <= 1.3.0):
curator delete --older-than 1 --prefix logstash-2015.05.19 (replace with snapshot name)
(versions >= 1.3.0):
curator delete snapshots --repository nlsbackup --prefix (replace with snapshot prefix)
You can also delete snapshots from the Web GUI under the backup page, or directly from the repo.
Purpose-Driven Commands
Check min master nodes:
curl localhost:9200/_cluster/settings?pretty
Set min master nodes:
curl http://localhost:9200/_cluster/settings -XPUT -d '{ "persistent": { "discovery.zen.minimum_master_nodes": 2} }'
Check users:
curl -XGET 'http://localhost:9200/nagioslogserver/_search?type=user&pretty'
Final Thoughts
For any support related questions please visit the Nagios Support Forums at:
http://support.nagios.com/forum/
Article ID: 837
Created On: Mon, Feb 11, 2019 at 7:01 PM
Last Updated On: Mon, Feb 11, 2019 at 7:01 PM
Authored by: tlea
Online URL: https://support.nagios.com/kb/article/nagios-log-server-troubleshooting-commands-837.html