Indexes do not close automatically

[joltapolme]

Hello.

I have configured that the indexes are automatically closed with more than 15 days but it is not working.





I can close them manually.

Any idea of whats happening?

Thanks!

[joltapolme]

Can I check some logs to see what happens?
What job runs the closing of the indexes?

Thanks in advance.

[cdienger]

NLS will run this command to close indices:

Code: Select all

/usr/local/nagioslogserver/scripts/curator.sh close indices --older-than 15 --time-unit days --timestring %Y.%m.%d

Try running this on the command line and let us know what output it returns.

[joltapolme]

Hi cdienger.
Thanks for the reply.

The output was this:

Code: Select all

2021-01-15 08:24:32,984 INFO      Job starting: close indices
2021-01-15 08:24:33,004 INFO      Action close will be performed on the following indices: [u'logstash-2020.07.11', u'logstash-2020.07.12', u'logstash-2020.07.13', u'logstash-2020.07.14', u'logstash-2020.07.15', u'logstash-2020.07.16', u'logstash-2020.07.17', u'logstash-2020.07.18', u'logstash-2020.07.19', u'logstash-2020.07.20', u'logstash-2020.07.21', u'logstash-2020.07.22', u'logstash-2020.07.23', u'logstash-2020.07.24', u'logstash-2020.07.25', u'logstash-2020.07.26', u'logstash-2020.07.27', u'logstash-2020.07.28', u'logstash-2020.07.29', u'logstash-2020.07.30', u'logstash-2020.07.31', u'logstash-2020.08.01', u'logstash-2020.08.02', u'logstash-2020.08.03', u'logstash-2020.08.04', u'logstash-2020.08.05', u'logstash-2020.08.06', u'logstash-2020.08.07', u'logstash-2020.08.08', u'logstash-2020.08.09', u'logstash-2020.08.10', u'logstash-2020.08.11', u'logstash-2020.08.12', u'logstash-2020.08.13', u'logstash-2020.08.14', u'logstash-2020.08.15', u'logstash-2020.08.16', u'logstash-2020.08.17', u'logstash-2020.08.18', u'logstash-2020.08.19', u'logstash-2020.08.20', u'logstash-2020.08.21', u'logstash-2020.08.22', u'logstash-2020.08.23', u'logstash-2020.08.24', u'logstash-2020.08.25', u'logstash-2020.08.26', u'logstash-2020.08.27', u'logstash-2020.08.28', u'logstash-2020.08.29', u'logstash-2020.08.30', u'logstash-2020.08.31', u'logstash-2020.09.01', u'logstash-2020.09.02', u'logstash-2020.09.03', u'logstash-2020.09.04', u'logstash-2020.09.05', u'logstash-2020.09.06', u'logstash-2020.09.07', u'logstash-2020.09.08', u'logstash-2020.09.09', u'logstash-2020.09.10', u'logstash-2020.09.11', u'logstash-2020.09.12', u'logstash-2020.09.13', u'logstash-2020.09.14', u'logstash-2020.09.15', u'logstash-2020.09.16', u'logstash-2020.09.17', u'logstash-2020.09.18', u'logstash-2020.09.19', u'logstash-2020.09.20', u'logstash-2020.09.21', u'logstash-2020.09.22', u'logstash-2020.09.23', u'logstash-2020.09.24', u'logstash-2020.09.25', u'logstash-2020.09.26', u'logstash-2020.09.27', u'logstash-2020.09.28', u'logstash-2020.09.29', u'logstash-2020.09.30', u'logstash-2020.10.01', u'logstash-2020.10.02', u'logstash-2020.10.03', u'logstash-2020.10.04', u'logstash-2020.10.05', u'logstash-2020.10.06', u'logstash-2020.10.07', u'logstash-2020.10.08', u'logstash-2020.10.09', u'logstash-2020.10.10', u'logstash-2020.10.11', u'logstash-2020.10.12', u'logstash-2020.10.13', u'logstash-2020.10.14', u'logstash-2020.10.15', u'logstash-2020.10.16', u'logstash-2020.10.17', u'logstash-2020.10.18', u'logstash-2020.10.19', u'logstash-2020.10.20', u'logstash-2020.10.21', u'logstash-2020.10.22', u'logstash-2020.10.23', u'logstash-2020.10.24', u'logstash-2020.10.25', u'logstash-2020.10.26', u'logstash-2020.10.27', u'logstash-2020.10.28', u'logstash-2020.10.29', u'logstash-2020.10.30', u'logstash-2020.10.31', u'logstash-2020.11.01', u'logstash-2020.11.02', u'logstash-2020.11.03', u'logstash-2020.11.04', u'logstash-2020.11.05', u'logstash-2020.11.06', u'logstash-2020.11.07', u'logstash-2020.11.08', u'logstash-2020.11.09', u'logstash-2020.11.10', u'logstash-2020.11.11', u'logstash-2020.11.12', u'logstash-2020.11.13', u'logstash-2020.11.14', u'logstash-2020.11.15', u'logstash-2020.11.16', u'logstash-2020.11.17', u'logstash-2020.11.18', u'logstash-2020.11.19', u'logstash-2020.11.20', u'logstash-2020.11.21', u'logstash-2020.11.22', u'logstash-2020.11.23', u'logstash-2020.11.24', u'logstash-2020.11.25', u'logstash-2020.11.26', u'logstash-2020.11.27', u'logstash-2020.11.28', u'logstash-2020.11.29', u'logstash-2020.11.30', u'logstash-2020.12.01', u'logstash-2020.12.02', u'logstash-2020.12.03', u'logstash-2020.12.04', u'logstash-2020.12.05', u'logstash-2020.12.06', u'logstash-2020.12.07', u'logstash-2020.12.08', u'logstash-2020.12.09', u'logstash-2020.12.10', u'logstash-2020.12.11', u'logstash-2020.12.12', u'logstash-2020.12.13', u'logstash-2020.12.14', u'logstash-2020.12.15', u'logstash-2020.12.16', u'logstash-2020.12.17', u'logstash-2020.12.18', u'logstash-2020.12.19', u'logstash-2020.12.20', u'logstash-2020.12.21', u'logstash-2020.12.22', u'logstash-2020.12.23', u'logstash-2020.12.24', u'logstash-2020.12.25', u'logstash-2020.12.26', u'logstash-2020.12.27', u'logstash-2020.12.28', u'logstash-2020.12.29', u'logstash-2020.12.30', u'logstash-2020.12.31']
2021-01-15 08:24:33,004 WARNING   Very large list of indices.  Breaking it up into smaller chunks.
2021-01-15 08:24:33,537 WARNING   No indices to seal.
2021-01-15 08:24:35,668 INFO      Provided indices successfully sealed. (Shown with --debug flag enabled.)
2021-01-15 08:24:37,732 INFO      Job completed successfully.

After run the command, the indices was closed.

Regards.

[cdienger]

Keep an eye on this for a few more days as it runs the close script once a day(by default) I've seen cases were it times out and is only able to close a few indices at a time if there is a large list. Now that it appears to be caught up it should work better going forward.

[joltapolme]

Hi cdienger

Sorry but It is not closing the indices automatically.
I have launched the command again and this is the output:

Code: Select all

2021-01-18 11:54:44,800 INFO      Job starting: close indices
2021-01-18 11:54:44,820 INFO      Action close will be performed on the following indices: [u'logstash-2020.07.11', u'logstash-2020.07.12', u'logstash-2020.07.13', u'logstash-2020.07.14', u'logstash-2020.07.15', u'logstash-2020.07.16', u'logstash-2020.07.17', u'logstash-2020.07.18', u'logstash-2020.07.19', u'logstash-2020.07.20', u'logstash-2020.07.21', u'logstash-2020.07.22', u'logstash-2020.07.23', u'logstash-2020.07.24', u'logstash-2020.07.25', u'logstash-2020.07.26', u'logstash-2020.07.27', u'logstash-2020.07.28', u'logstash-2020.07.29', u'logstash-2020.07.30', u'logstash-2020.07.31', u'logstash-2020.08.01', u'logstash-2020.08.02', u'logstash-2020.08.03', u'logstash-2020.08.04', u'logstash-2020.08.05', u'logstash-2020.08.06', u'logstash-2020.08.07', u'logstash-2020.08.08', u'logstash-2020.08.09', u'logstash-2020.08.10', u'logstash-2020.08.11', u'logstash-2020.08.12', u'logstash-2020.08.13', u'logstash-2020.08.14', u'logstash-2020.08.15', u'logstash-2020.08.16', u'logstash-2020.08.17', u'logstash-2020.08.18', u'logstash-2020.08.19', u'logstash-2020.08.20', u'logstash-2020.08.21', u'logstash-2020.08.22', u'logstash-2020.08.23', u'logstash-2020.08.24', u'logstash-2020.08.25', u'logstash-2020.08.26', u'logstash-2020.08.27', u'logstash-2020.08.28', u'logstash-2020.08.29', u'logstash-2020.08.30', u'logstash-2020.08.31', u'logstash-2020.09.01', u'logstash-2020.09.02', u'logstash-2020.09.03', u'logstash-2020.09.04', u'logstash-2020.09.05', u'logstash-2020.09.06', u'logstash-2020.09.07', u'logstash-2020.09.08', u'logstash-2020.09.09', u'logstash-2020.09.10', u'logstash-2020.09.11', u'logstash-2020.09.12', u'logstash-2020.09.13', u'logstash-2020.09.14', u'logstash-2020.09.15', u'logstash-2020.09.16', u'logstash-2020.09.17', u'logstash-2020.09.18', u'logstash-2020.09.19', u'logstash-2020.09.20', u'logstash-2020.09.21', u'logstash-2020.09.22', u'logstash-2020.09.23', u'logstash-2020.09.24', u'logstash-2020.09.25', u'logstash-2020.09.26', u'logstash-2020.09.27', u'logstash-2020.09.28', u'logstash-2020.09.29', u'logstash-2020.09.30', u'logstash-2020.10.01', u'logstash-2020.10.02', u'logstash-2020.10.03', u'logstash-2020.10.04', u'logstash-2020.10.05', u'logstash-2020.10.06', u'logstash-2020.10.07', u'logstash-2020.10.08', u'logstash-2020.10.09', u'logstash-2020.10.10', u'logstash-2020.10.11', u'logstash-2020.10.12', u'logstash-2020.10.13', u'logstash-2020.10.14', u'logstash-2020.10.15', u'logstash-2020.10.16', u'logstash-2020.10.17', u'logstash-2020.10.18', u'logstash-2020.10.19', u'logstash-2020.10.20', u'logstash-2020.10.21', u'logstash-2020.10.22', u'logstash-2020.10.23', u'logstash-2020.10.24', u'logstash-2020.10.25', u'logstash-2020.10.26', u'logstash-2020.10.27', u'logstash-2020.10.28', u'logstash-2020.10.29', u'logstash-2020.10.30', u'logstash-2020.10.31', u'logstash-2020.11.01', u'logstash-2020.11.02', u'logstash-2020.11.03', u'logstash-2020.11.04', u'logstash-2020.11.05', u'logstash-2020.11.06', u'logstash-2020.11.07', u'logstash-2020.11.08', u'logstash-2020.11.09', u'logstash-2020.11.10', u'logstash-2020.11.11', u'logstash-2020.11.12', u'logstash-2020.11.13', u'logstash-2020.11.14', u'logstash-2020.11.15', u'logstash-2020.11.16', u'logstash-2020.11.17', u'logstash-2020.11.18', u'logstash-2020.11.19', u'logstash-2020.11.20', u'logstash-2020.11.21', u'logstash-2020.11.22', u'logstash-2020.11.23', u'logstash-2020.11.24', u'logstash-2020.11.25', u'logstash-2020.11.26', u'logstash-2020.11.27', u'logstash-2020.11.28', u'logstash-2020.11.29', u'logstash-2020.11.30', u'logstash-2020.12.01', u'logstash-2020.12.02', u'logstash-2020.12.03', u'logstash-2020.12.04', u'logstash-2020.12.05', u'logstash-2020.12.06', u'logstash-2020.12.07', u'logstash-2020.12.08', u'logstash-2020.12.09', u'logstash-2020.12.10', u'logstash-2020.12.11', u'logstash-2020.12.12', u'logstash-2020.12.13', u'logstash-2020.12.14', u'logstash-2020.12.15', u'logstash-2020.12.16', u'logstash-2020.12.17', u'logstash-2020.12.18', u'logstash-2020.12.19', u'logstash-2020.12.20', u'logstash-2020.12.21', u'logstash-2020.12.22', u'logstash-2020.12.23', u'logstash-2020.12.24', u'logstash-2020.12.25', u'logstash-2020.12.26', u'logstash-2020.12.27', u'logstash-2020.12.28', u'logstash-2020.12.29', u'logstash-2020.12.30', u'logstash-2020.12.31', u'logstash-2021.01.01', u'logstash-2021.01.02', u'logstash-2021.01.03']
2021-01-18 11:54:44,820 WARNING   Very large list of indices.  Breaking it up into smaller chunks.
2021-01-18 11:54:45,185 WARNING   No indices to seal.
2021-01-18 11:54:47,278 INFO      Provided indices successfully sealed. (Shown with --debug flag enabled.)
2021-01-18 11:54:49,302 INFO      Job completed successfully.

Regards

[cdienger]

There may be issues with the cron job or nagios account. Check the nagios account:

Code: Select all

chage -l nagios

and set expiration settings with this if necessary:

Code: Select all

chage -I -1 -m 0 -M 99999 -E -1 nagios

Also check in /var/log/cron for any errors related to the nagios account.

Finally, tail the jobs.log account:

Code: Select all

tail -f /usr/local/nagioslogserver/var/jobs.log

Run the above while you force the snapshots_maintenance job under Admin > System > Command Subsystem.

[joltapolme]

Hi cdienger.

Code: Select all

tail -f /usr/local/nagioslogserver/var/jobs.log

Running command do_maintenance with args ' ' for job id: snapshots_maintenance
/usr/local/nagioslogserver/scripts/curator.sh: line 5: /usr/bin/curator: Permission denied
-----
Running cmd: /usr/local/nagioslogserver/scripts/curator.sh optimize indices --older-than 2 --time-unit days --timestring %Y.%m.%d
Return: 126

I don't know if it's a problem with the permissions of the curator.sh file or others.
I'm checking but can't find the cause of the problem.
Any ideas?

Thank you very much.

[ssax]

What is the output of this command?

Code: Select all

ls -l /usr/bin/curator

Should be this:

Code: Select all

chmod 0775 /usr/bin/curator

[joltapolme]

Hi again.

Debugging yesterday I modified the permissions on the file you indicate, but the error is now different:

Code: Select all

Running command do_maintenance with args ' ' for job id: snapshots_maintenance
Traceback (most recent call last):
  File "/usr/bin/curator", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 3007, in <module>
    working_set.require(__requires__)
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 728, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 626, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: elasticsearch-curator==3.4.0
-----
Running cmd: /usr/local/nagioslogserver/scripts/curator.sh optimize indices --older-than 2 --time-unit days --timestring %Y.%m.%d
Return: 1

-----
Traceback (most recent call last):
  File "/usr/bin/curator", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 3007, in <module>
    working_set.require(__requires__)
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 728, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 626, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: elasticsearch-curator==3.4.0
-----
Running cmd: /usr/local/nagioslogserver/scripts/curator.sh close indices --older-than 15 --time-unit days --timestring %Y.%m.%d
Return: 1

-----
SUCCESS

Thanks!