Nagios log server Indeces are not deleting automatically

[mejokj]

Hello ,

The Nagios log server is not automatically deleting the indexes. I need to manually delete the indexes. for the current settings. Kindly go to the below links.
++++++++++++++++++++++++
https://ibb.co/LgG2yLM
++++++++++++++++++++++++

Regards,
Dhinil KV

[cdienger]

How exactly are you manually deleting the indices? What command are you using?

Does this command work or provide any errors:

Code: Select all

/usr/local/nagioslogserver/scripts/curator.sh delete indices --older-than 90 --time-unit days --timestring %Y.%m.%d

?

[mejokj]

Hello,

Below is the result of the command. Now i am trying to delete the log manually from web interface.

[root@nagios_log ~]# /usr/local/nagioslogserver/scripts/curator.sh delete indices --older-than 90 --time-unit days --timestring %Y.%m.%d
2020-06-21 08:41:33,140 INFO Job starting: delete indices
2020-06-21 08:41:33,284 INFO Pruning Kibana-related indices to prevent accidental deletion.
2020-06-21 08:41:33,284 INFO Action delete will be performed on the following indices: [u'logstash-2020.02.06', u'logstash-2020.02.07', u'logstash-2020.02.08', u'logstash-2020.02.09', u'logstash-2020.02.10', u'logstash-2020.02.11', u'logstash-2020.02.12', u'logstash-2020.02.13', u'logstash-2020.02.14', u'logstash-2020.02.15', u'logstash-2020.02.16', u'logstash-2020.02.17', u'logstash-2020.02.18', u'logstash-2020.02.19', u'logstash-2020.02.20', u'logstash-2020.02.21', u'logstash-2020.02.22', u'logstash-2020.02.23', u'logstash-2020.02.24', u'logstash-2020.02.25', u'logstash-2020.02.26', u'logstash-2020.02.27', u'logstash-2020.02.28', u'logstash-2020.02.29', u'logstash-2020.03.01', u'logstash-2020.03.02', u'logstash-2020.03.03', u'logstash-2020.03.04', u'logstash-2020.03.05', u'logstash-2020.03.06', u'logstash-2020.03.07', u'logstash-2020.03.08', u'logstash-2020.03.09', u'logstash-2020.03.10', u'logstash-2020.03.11', u'logstash-2020.03.12', u'logstash-2020.03.13', u'logstash-2020.03.14', u'logstash-2020.03.15', u'logstash-2020.03.16', u'logstash-2020.03.17', u'logstash-2020.03.18', u'logstash-2020.03.19', u'logstash-2020.03.20', u'logstash-2020.03.21', u'logstash-2020.03.22', u'logstash-2020.03.23']
2020-06-21 08:41:33,287 INFO Deleting indices as a batch operation:
2020-06-21 08:41:33,287 INFO ---deleting index logstash-2020.02.06
2020-06-21 08:41:33,287 INFO ---deleting index logstash-2020.02.07
2020-06-21 08:41:33,287 INFO ---deleting index logstash-2020.02.08
2020-06-21 08:41:33,287 INFO ---deleting index logstash-2020.02.09
2020-06-21 08:41:33,288 INFO ---deleting index logstash-2020.02.10
2020-06-21 08:41:33,288 INFO ---deleting index logstash-2020.02.11
2020-06-21 08:41:33,288 INFO ---deleting index logstash-2020.02.12
2020-06-21 08:41:33,288 INFO ---deleting index logstash-2020.02.13
2020-06-21 08:41:33,288 INFO ---deleting index logstash-2020.02.14
2020-06-21 08:41:33,288 INFO ---deleting index logstash-2020.02.15
2020-06-21 08:41:33,288 INFO ---deleting index logstash-2020.02.16
2020-06-21 08:41:33,288 INFO ---deleting index logstash-2020.02.17
2020-06-21 08:41:33,288 INFO ---deleting index logstash-2020.02.18
2020-06-21 08:41:33,288 INFO ---deleting index logstash-2020.02.19
2020-06-21 08:41:33,288 INFO ---deleting index logstash-2020.02.20
2020-06-21 08:41:33,288 INFO ---deleting index logstash-2020.02.21
2020-06-21 08:41:33,288 INFO ---deleting index logstash-2020.02.22
2020-06-21 08:41:33,288 INFO ---deleting index logstash-2020.02.23
2020-06-21 08:41:33,288 INFO ---deleting index logstash-2020.02.24
2020-06-21 08:41:33,289 INFO ---deleting index logstash-2020.02.25
2020-06-21 08:41:33,289 INFO ---deleting index logstash-2020.02.26
2020-06-21 08:41:33,289 INFO ---deleting index logstash-2020.02.27
2020-06-21 08:41:33,289 INFO ---deleting index logstash-2020.02.28
2020-06-21 08:41:33,289 INFO ---deleting index logstash-2020.02.29
2020-06-21 08:41:33,289 INFO ---deleting index logstash-2020.03.01
2020-06-21 08:41:33,289 INFO ---deleting index logstash-2020.03.02
2020-06-21 08:41:33,289 INFO ---deleting index logstash-2020.03.03
2020-06-21 08:41:33,289 INFO ---deleting index logstash-2020.03.04
2020-06-21 08:41:33,289 INFO ---deleting index logstash-2020.03.05
2020-06-21 08:41:33,289 INFO ---deleting index logstash-2020.03.06
2020-06-21 08:41:33,289 INFO ---deleting index logstash-2020.03.07
2020-06-21 08:41:33,289 INFO ---deleting index logstash-2020.03.08
2020-06-21 08:41:33,289 INFO ---deleting index logstash-2020.03.09
2020-06-21 08:41:33,289 INFO ---deleting index logstash-2020.03.10
2020-06-21 08:41:33,289 INFO ---deleting index logstash-2020.03.11
2020-06-21 08:41:33,290 INFO ---deleting index logstash-2020.03.12
2020-06-21 08:41:33,290 INFO ---deleting index logstash-2020.03.13
2020-06-21 08:41:33,290 INFO ---deleting index logstash-2020.03.14
2020-06-21 08:41:33,290 INFO ---deleting index logstash-2020.03.15
2020-06-21 08:41:33,290 INFO ---deleting index logstash-2020.03.16
2020-06-21 08:41:33,290 INFO ---deleting index logstash-2020.03.17
2020-06-21 08:41:33,290 INFO ---deleting index logstash-2020.03.18
2020-06-21 08:41:33,290 INFO ---deleting index logstash-2020.03.19
2020-06-21 08:41:33,290 INFO ---deleting index logstash-2020.03.20
2020-06-21 08:41:33,290 INFO ---deleting index logstash-2020.03.21
2020-06-21 08:41:33,290 INFO ---deleting index logstash-2020.03.22
2020-06-21 08:41:33,290 INFO ---deleting index logstash-2020.03.23
2020-06-21 08:41:48,361 INFO Job completed successfully.

[cdienger]

Interesting. I'm not sure what happened but keep an eye on it and if it starts to fail again we can poke around some more. First thing I would want to see is what is captured in jobs.log:

Code: Select all

tail -f /usr/local/nagioslogserver/var/jobs.log

while the snapshots_maintenance job under Admin > System > Command Subsystem is forced to run.

[mejokj]

Below is the output of the log file.

[root@nagios_log ~]# tail -f /usr/local/nagioslogserver/var/jobs.log
Processed 0 node jobs.
Processed 0 global jobs.
Running command do_maintenance with args ' ' for job id: snapshots_maintenance
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated

Also
Under the system >> command sub system, all date last runtime and next run time are showing the 2019 year.
Please see the link to see the screenhsot.

https://ibb.co/n7gwNpZ

Seems the jobs are not running.

[cdienger]

Check the /var/log/cron log for errors. It looks like the jobs are not running. Also run:

Code: Select all

chage -l nagios

to see if the nagios account is expired. If it is then you can run:

Code: Select all

chage -I -1 -m 0 -M 99999 -E -1 nagios

[mejokj]

Hi Team,

We did see that cron job are running but didn't find any error in /var/log/cron and nagios user password is never expired.

Cron logs
========

server/var/poller.log 2>&1)
Jun 24 14:44:01 nagios_log CROND[7598]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1)
Jun 24 14:45:01 nagios_log CROND[7747]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1)
Jun 24 14:45:01 nagios_log CROND[7749]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1)
Jun 24 14:46:01 nagios_log CROND[7887]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1)
Jun 24 14:46:01 nagios_log CROND[7888]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1)

[root@nagios_log ~]# chage -l nagios
Last password change : يول 27, 2017
Password expires : never
Password inactive : never
Account expires : never
Minimum number of days between password change : 0
Maximum number of days between password change : 99999
Number of days of warning before password expires : 7
[root@nagios_log ~]#

Thanks,

[cdienger]

Hmmm. Use the "Reset All Jobs" button on the subsystem page to clear things out and see if that does the trick. If it is still having issues then PM me a copy of the nagioslogserver index:

Code: Select all

curl -XPOST http://localhost:9200/nagioslogserver/_export?path=/tmp/nagioslogserver.tar.gz

The above downloads a copy of the index and saves it to /tmp/nagioslogserver.tar.gz. Please PM me this file.

[mejokj]

Hello,

On resetting the job it seems not success please see the below screenshot.

https://ibb.co/Wnjy4Jy

On exporting the curl command it shows the file is not found.
[root@nagios_log syslog_data]# curl -XPOST http://localhost:9200/nagioslogserver/_ ... ver.tar.gz
{"error":"FileNotFoundException[can't open for output, check existence or access rights: /tmp/nagioslogserver.tar.gz]","status":500}[root@nagios_log syslog_data]#

[cdienger]

500 internal errors can be a result of a low memory setting. Follow https://support.nagios.com/kb/article.php?id=132 to increase it.

Also try running it and saving to the /root directory instead:

Code: Select all

curl -XPOST http://localhost:9200/nagioslogserver/_export?path=/root/nagioslogserver.tar.gz