Page 2 of 2
Re: Nagios Logging - Snapshot scheduler
Posted: Thu Jul 16, 2020 12:02 pm
by cdienger
There does seem to be something weird here. Please PM me a profile so I can take a look at some logs and other system info. It can be gathered under Admin > System > System Status > Download System Profile or from the command line with:
Code: Select all
/usr/local/nagioslogserver/scripts/profile.sh
This will create /tmp/system-profile.tar.gz.
Note that this file can be very large and may not be able to be uploaded in a PM. This is usually due to the logs in the Logstash and/or Elasticsearch directories found in it. If it is too large, please open the profile, extract these directories/files and send them separately.
Re: Nagios Logging - Snapshot scheduler
Posted: Thu Jul 16, 2020 9:30 pm
by songviet
Hi,
Please find from attachments, there are from CLuster + 1 node.
Files have been downloaded for our review and removed from this post.
Re: Nagios Logging - Snapshot scheduler
Posted: Fri Jul 17, 2020 1:39 pm
by cdienger
The cluster is having issues with indices because of disk space issues:
[2020-07-17 02:27:21,635][WARN ][cluster.routing.allocation.decider] [6b1c0aaa-e3ca-420a-b57b-9ac39159df9f] high disk watermark [90%] exceeded on [uZmL-_E6T5-FnrKqtvQkrg][6b1c0aaa-e3ca-420a-b57b-9ac39159df9f] free: 95.9gb[9.5%], shards will be relocated away from this node
[2020-07-17 02:27:21,635][WARN ][cluster.routing.allocation.decider] [6b1c0aaa-e3ca-420a-b57b-9ac39159df9f] high disk watermark [90%] exceeded on [vuZTiFILQQ-KR1W0HlW1BQ][e966b108-935b-47ec-9f4b-427a675e95d9] free: 37.1gb[7.5%], shards will be relocated away from this node
You'll want to increase the size of the disk/partition that the elasticsearch database is on or delete old indices to free up space.
Deleting indices can be done with:
Code: Select all
/usr/local/nagioslogserver/scripts/curator.sh delete indices --older-than X --time-unit days --timestring %Y.%m.%d
(Replace X with the number of days)
You can also set this number of days that old data will be stored on the NLS file system so that it automatically deletes. Set this under Admin > System > Snapshots & Maintenance > Maintenance and Repository Settings > Maintenance Settings > Delete indexes older than.
Re: Nagios Logging - Snapshot scheduler
Posted: Fri Jul 17, 2020 1:51 pm
by cdienger
It appears that we have a ticket open for this issue and this is also a duplicate of
https://support.nagios.com/forum/viewto ... 38&t=59285.
Going forward please only open one ticket or one forum post per issue so that we can avoid duplicating our efforts.