Nagios log server Indeces are not deleting automatically
Posted: Thu Jun 18, 2020 9:55 am
by mejokj
Hello ,
The Nagios log server is not automatically deleting the indexes. I need to manually delete the indexes. for the current settings. Kindly go to the below links.
++++++++++++++++++++++++
https://ibb.co/LgG2yLM
++++++++++++++++++++++++
Regards,
Dhinil KV
Re: Nagios log server Indeces are not deleting automatically
Posted: Thu Jun 18, 2020 3:47 pm
by cdienger
How exactly are you manually deleting the indices? What command are you using?
Does this command work or provide any errors:
Code: Select all
/usr/local/nagioslogserver/scripts/curator.sh delete indices --older-than 90 --time-unit days --timestring %Y.%m.%d
?
Re: Nagios log server Indeces are not deleting automatically
Posted: Sat Jun 20, 2020 11:49 pm
by mejokj
Hello,
Below is the result of the command. Now i am trying to delete the log manually from web interface.
[root@nagios_log ~]# /usr/local/nagioslogserver/scripts/curator.sh delete indices --older-than 90 --time-unit days --timestring %Y.%m.%d
2020-06-21 08:41:33,140 INFO Job starting: delete indices
2020-06-21 08:41:33,284 INFO Pruning Kibana-related indices to prevent accidental deletion.
2020-06-21 08:41:33,284 INFO Action delete will be performed on the following indices: [u'logstash-2020.02.06', u'logstash-2020.02.07', u'logstash-2020.02.08', u'logstash-2020.02.09', u'logstash-2020.02.10', u'logstash-2020.02.11', u'logstash-2020.02.12', u'logstash-2020.02.13', u'logstash-2020.02.14', u'logstash-2020.02.15', u'logstash-2020.02.16', u'logstash-2020.02.17', u'logstash-2020.02.18', u'logstash-2020.02.19', u'logstash-2020.02.20', u'logstash-2020.02.21', u'logstash-2020.02.22', u'logstash-2020.02.23', u'logstash-2020.02.24', u'logstash-2020.02.25', u'logstash-2020.02.26', u'logstash-2020.02.27', u'logstash-2020.02.28', u'logstash-2020.02.29', u'logstash-2020.03.01', u'logstash-2020.03.02', u'logstash-2020.03.03', u'logstash-2020.03.04', u'logstash-2020.03.05', u'logstash-2020.03.06', u'logstash-2020.03.07', u'logstash-2020.03.08', u'logstash-2020.03.09', u'logstash-2020.03.10', u'logstash-2020.03.11', u'logstash-2020.03.12', u'logstash-2020.03.13', u'logstash-2020.03.14', u'logstash-2020.03.15', u'logstash-2020.03.16', u'logstash-2020.03.17', u'logstash-2020.03.18', u'logstash-2020.03.19', u'logstash-2020.03.20', u'logstash-2020.03.21', u'logstash-2020.03.22', u'logstash-2020.03.23']
2020-06-21 08:41:33,287 INFO Deleting indices as a batch operation:
2020-06-21 08:41:33,287 INFO ---deleting index logstash-2020.02.06
2020-06-21 08:41:33,287 INFO ---deleting index logstash-2020.02.07
2020-06-21 08:41:33,287 INFO ---deleting index logstash-2020.02.08
2020-06-21 08:41:33,287 INFO ---deleting index logstash-2020.02.09
2020-06-21 08:41:33,288 INFO ---deleting index logstash-2020.02.10
2020-06-21 08:41:33,288 INFO ---deleting index logstash-2020.02.11
2020-06-21 08:41:33,288 INFO ---deleting index logstash-2020.02.12
2020-06-21 08:41:33,288 INFO ---deleting index logstash-2020.02.13
2020-06-21 08:41:33,288 INFO ---deleting index logstash-2020.02.14
2020-06-21 08:41:33,288 INFO ---deleting index logstash-2020.02.15
2020-06-21 08:41:33,288 INFO ---deleting index logstash-2020.02.16
2020-06-21 08:41:33,288 INFO ---deleting index logstash-2020.02.17
2020-06-21 08:41:33,288 INFO ---deleting index logstash-2020.02.18
2020-06-21 08:41:33,288 INFO ---deleting index logstash-2020.02.19
2020-06-21 08:41:33,288 INFO ---deleting index logstash-2020.02.20
2020-06-21 08:41:33,288 INFO ---deleting index logstash-2020.02.21
2020-06-21 08:41:33,288 INFO ---deleting index logstash-2020.02.22
2020-06-21 08:41:33,288 INFO ---deleting index logstash-2020.02.23
2020-06-21 08:41:33,288 INFO ---deleting index logstash-2020.02.24
2020-06-21 08:41:33,289 INFO ---deleting index logstash-2020.02.25
2020-06-21 08:41:33,289 INFO ---deleting index logstash-2020.02.26
2020-06-21 08:41:33,289 INFO ---deleting index logstash-2020.02.27
2020-06-21 08:41:33,289 INFO ---deleting index logstash-2020.02.28
2020-06-21 08:41:33,289 INFO ---deleting index logstash-2020.02.29
2020-06-21 08:41:33,289 INFO ---deleting index logstash-2020.03.01
2020-06-21 08:41:33,289 INFO ---deleting index logstash-2020.03.02
2020-06-21 08:41:33,289 INFO ---deleting index logstash-2020.03.03
2020-06-21 08:41:33,289 INFO ---deleting index logstash-2020.03.04
2020-06-21 08:41:33,289 INFO ---deleting index logstash-2020.03.05
2020-06-21 08:41:33,289 INFO ---deleting index logstash-2020.03.06
2020-06-21 08:41:33,289 INFO ---deleting index logstash-2020.03.07
2020-06-21 08:41:33,289 INFO ---deleting index logstash-2020.03.08
2020-06-21 08:41:33,289 INFO ---deleting index logstash-2020.03.09
2020-06-21 08:41:33,289 INFO ---deleting index logstash-2020.03.10
2020-06-21 08:41:33,289 INFO ---deleting index logstash-2020.03.11
2020-06-21 08:41:33,290 INFO ---deleting index logstash-2020.03.12
2020-06-21 08:41:33,290 INFO ---deleting index logstash-2020.03.13
2020-06-21 08:41:33,290 INFO ---deleting index logstash-2020.03.14
2020-06-21 08:41:33,290 INFO ---deleting index logstash-2020.03.15
2020-06-21 08:41:33,290 INFO ---deleting index logstash-2020.03.16
2020-06-21 08:41:33,290 INFO ---deleting index logstash-2020.03.17
2020-06-21 08:41:33,290 INFO ---deleting index logstash-2020.03.18
2020-06-21 08:41:33,290 INFO ---deleting index logstash-2020.03.19
2020-06-21 08:41:33,290 INFO ---deleting index logstash-2020.03.20
2020-06-21 08:41:33,290 INFO ---deleting index logstash-2020.03.21
2020-06-21 08:41:33,290 INFO ---deleting index logstash-2020.03.22
2020-06-21 08:41:33,290 INFO ---deleting index logstash-2020.03.23
2020-06-21 08:41:48,361 INFO Job completed successfully.
Re: Nagios log server Indeces are not deleting automatically
Posted: Mon Jun 22, 2020 1:12 pm
by cdienger
Interesting. I'm not sure what happened but keep an eye on it and if it starts to fail again we can poke around some more. First thing I would want to see is what is captured in jobs.log:
Code: Select all
tail -f /usr/local/nagioslogserver/var/jobs.log
while the snapshots_maintenance job under Admin > System > Command Subsystem is forced to run.
Re: Nagios log server Indeces are not deleting automatically
Posted: Tue Jun 23, 2020 2:35 am
by mejokj
Below is the output of the log file.
[root@nagios_log ~]# tail -f /usr/local/nagioslogserver/var/jobs.log
Processed 0 node jobs.
Processed 0 global jobs.
Running command do_maintenance with args ' ' for job id: snapshots_maintenance
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
Also
Under the system >> command sub system, all date last runtime and next run time are showing the 2019 year.
Please see the link to see the screenhsot.
https://ibb.co/n7gwNpZ
Seems the jobs are not running.
Re: Nagios log server Indeces are not deleting automatically
Posted: Tue Jun 23, 2020 11:50 am
by cdienger
Check the /var/log/cron log for errors. It looks like the jobs are not running. Also run:
to see if the nagios account is expired. If it is then you can run:
Code: Select all
chage -I -1 -m 0 -M 99999 -E -1 nagios
Re: Nagios log server Indeces are not deleting automatically
Posted: Wed Jun 24, 2020 6:01 am
by mejokj
Hi Team,
We did see that cron job are running but didn't find any error in /var/log/cron and nagios user password is never expired.
Cron logs
========
server/var/poller.log 2>&1)
Jun 24 14:44:01 nagios_log CROND[7598]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1)
Jun 24 14:45:01 nagios_log CROND[7747]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1)
Jun 24 14:45:01 nagios_log CROND[7749]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1)
Jun 24 14:46:01 nagios_log CROND[7887]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1)
Jun 24 14:46:01 nagios_log CROND[7888]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1)
[root@nagios_log ~]# chage -l nagios
Last password change : يول 27, 2017
Password expires : never
Password inactive : never
Account expires : never
Minimum number of days between password change : 0
Maximum number of days between password change : 99999
Number of days of warning before password expires : 7
[root@nagios_log ~]#
Thanks,
Re: Nagios log server Indeces are not deleting automatically
Posted: Wed Jun 24, 2020 1:48 pm
by cdienger
Hmmm. Use the "Reset All Jobs" button on the subsystem page to clear things out and see if that does the trick. If it is still having issues then PM me a copy of the nagioslogserver index:
Code: Select all
curl -XPOST http://localhost:9200/nagioslogserver/_export?path=/tmp/nagioslogserver.tar.gz
The above downloads a copy of the index and saves it to /tmp/nagioslogserver.tar.gz. Please PM me this file.
Re: Nagios log server Indeces are not deleting automatically
Posted: Sat Jun 27, 2020 2:15 am
by mejokj
Hello,
On resetting the job it seems not success please see the below screenshot.
https://ibb.co/Wnjy4Jy
On exporting the curl command it shows the file is not found.
[root@nagios_log syslog_data]# curl -XPOST
http://localhost:9200/nagioslogserver/_ ... ver.tar.gz
{"error":"FileNotFoundException[can't open for output, check existence or access rights: /tmp/nagioslogserver.tar.gz]","status":500}[root@nagios_log syslog_data]#
Re: Nagios log server Indeces are not deleting automatically
Posted: Mon Jun 29, 2020 9:59 am
by cdienger
500 internal errors can be a result of a low memory setting. Follow
https://support.nagios.com/kb/article.php?id=132 to increase it.
Also try running it and saving to the /root directory instead:
Code: Select all
curl -XPOST http://localhost:9200/nagioslogserver/_export?path=/root/nagioslogserver.tar.gz