Nagios Log Server - Troubleshooting Commands

Troubleshooting Commands

This is a list of commands that can be used to help diagnose issues with Nagios Log Server.

Be careful, some commands can be destructive and can result in data loss!

Generic Information Gathering

Master:

curl 'localhost:9200/_cat/master?v'

Nodes:

curl 'localhost:9200/_cat/nodes?v'

Pending tasks:

curl 'localhost:9200/_cat/pending_tasks?v'

Recovery:

curl -XGET 'localhost:9200/_cat/recovery?v'

Running plugins:

curl 'localhost:9200/_cat/plugins?v'

Cluster basic health:

curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'

Check jvm settings:

curl -XGET localhost:9200/_nodes/jvm?pretty

Shard status (use index status instead for clarify, in most cases):

curl -XGET http://localhost:9200/_cat/shards

Cluster state:

curl -XGET 'http://localhost:9200/_cluster/state?pretty'

Shard health (index health works better in most circumstances):

curl -XGET 'http://localhost:9200/_cluster/health/*?level=shards'

Red Cluster Recovery

Check index health:

curl 'localhost:9200/_cluster/health?level=indices&pretty'

Shard list:

curl -s localhost:9200/_cat/shards

Unassigned shards:

curl -s localhost:9200/_cat/shards | grep UNASS

Delete bad index:

curl -XDELETE 'http://localhost:9200/twitter/'

Active Troubleshooting

Tail maintenance logs:

tail -f /usr/local/nagioslogserver/var/jobs.log

Tail poller logs:

tail -f /usr/local/nagioslogserver/var/poller.log

Shut down all nodes:

curl -XPOST 'http://localhost:9200/_shutdown'

Check users:

curl -XGET 'http://localhost:9200/nagioslogserver/_search?type=user&pretty'

Backup Troubleshooting

Check running knapsack export:

curl -XPOST 'http://localhost:9200/_export/state'

Typically jobs.log should be tailed and the backup+maintenance command should be forced from the subsys - jobs.log should complain about a partial snapshot.

Delete a snapshot (versions <= 1.3.0):

curator delete --older-than 1 --prefix logstash-2015.05.19 (replace with snapshot name)

(versions >= 1.3.0):

curator delete snapshots --repository nlsbackup --prefix (replace with snapshot prefix)

 You can also delete snapshots from the Web GUI under the backup page, or directly from the repo.

Purpose-Driven Commands

Check min master nodes:

curl localhost:9200/_cluster/settings?pretty

Set min master nodes:

curl http://localhost:9200/_cluster/settings -XPUT -d '{ "persistent": { "discovery.zen.minimum_master_nodes": 2} }'

Check users:

curl -XGET 'http://localhost:9200/nagioslogserver/_search?type=user&pretty'

Final Thoughts

For any support related questions please visit the Nagios Support Forums at:

http://support.nagios.com/forum/