logstash.log filling up disk space

[jmsmith]

nls_dashboard.JPG

Hello,
I am running 2 2-node clusters and am having some issues with both nodes in 1 cluster only.
logstash.log is consuming all disk space, and there are no .gz files being created.
I currently have my maintenance settings configured as such as seen in my attachment.
When I try to run logrotate manually, I get the following:
sudo logrotate -d /etc/logrotate.d/nagioslogserver
reading config file /etc/logrotate.d/nagioslogserver
Allocating hash table for state file, size 15360 B

Handling 2 logs

rotating pattern: /var/log/logstash/*.log after 1 days (7 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/logstash/logstash.log
log does not need rotating (log has been already rotated)
rotating pattern: /var/log/elasticsearch/*.log after 1 days (7 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/elasticsearch/13a885a0-fce8-431e-9741-453d7c76249b_index_indexing_slowlog.log
log does not need rotating (log has been already rotated)considering log /var/log/elasticsearch/13a885a0-fce8-431e-9741-453d7c76249b_index_search_slowlog.log
log does not need rotating (log has been already rotated)considering log /var/log/elasticsearch/13a885a0-fce8-431e-9741-453d7c76249b.log
log does not need rotating (log has been already rotated)considering log /var/log/elasticsearch/cb1cf9de-5fdc-4118-a851-20f15ae554ec_index_indexing_slowlog.log
log does not need rotating (log has been already rotated)considering log /var/log/elasticsearch/cb1cf9de-5fdc-4118-a851-20f15ae554ec_index_search_slowlog.log
log does not need rotating (log has been already rotated)considering log /var/log/elasticsearch/cb1cf9de-5fdc-4118-a851-20f15ae554ec.log

[cdienger]

What OS is NLS installed on? Please provide a copy of /etc/logrotate.d/nagioslogserver.

Go ahead and remove the log and allow it to be recreated:

Code: Select all

systemctl stop logstash
rm /var/log/logstash/logstash.log

[jmsmith]

We have been stopping logstash and removing the log when it fills up.
We have done some modifications to the logrotate.d/nagioslogserver to troubleshoot the issue. These changes were made a few minutes ago, so not sure if it will work yet.
/etc/logrotate.d/nagioslogserver
/var/log/logstash/*.log {
hourly
maxsize 1G
rotate 1
copytruncate
compress
missingok
notifempty
}

/var/log/elasticsearch/*.log {
daily
rotate 7
copytruncate
compress
missingok
notifempty
}

[cdienger]

Let us know if that has improved things. If not, please provide a directory listing:

Code: Select all

ls -alh /var/log/logstash

as well as copy of /var/lib/logrotate/logrotate.status

Also, logstash.log should be relatively small unless there is a reoccuring error. What do you see if you run:

Code: Select all

tail -f /var/log/logstash/logstash.log

?