Log Server Logs

[OptimusB]

Noticed that the log files in /var/log/elasticsearch is accumulating and doesn't purge since we had it running. I had over 20GB of logs from about 5 months of usage. Is there a job that purges these in the current version?

[jolson]

I'm taking a look at our logrotate configuration files, and they appear to be alright. Double-check to ensure that you have the following configs on your system.

Code: Select all

[root@nls1 ~]# cat /etc/logrotate.d/nagioslogserver
/var/log/logstash/*.log {
        daily
        rotate 7
        copytruncate
        compress
        missingok
        notifempty
}

/var/log/elasticsearch/*.log {
        daily
        rotate 7
        copytruncate
        compress
        missingok
        notifempty
}

Code: Select all

[root@nls1 ~]# cat /etc/cron.daily/logrotate
#!/bin/sh

/usr/sbin/logrotate /etc/logrotate.conf >/dev/null 2>&1
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
    /usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
exit 0

Code: Select all

[root@nls1 ~]# cat /etc/logrotate.conf
# see "man logrotate" for details
# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# use date as a suffix of the rotated file
dateext

# uncomment this if you want your log files compressed
#compress

# RPM packages drop log rotation information into this directory
include /etc/logrotate.d

# no packages own wtmp and btmp -- we'll rotate them here
/var/log/wtmp {
    monthly
    create 0664 root utmp
        minsize 1M
    rotate 1
}

/var/log/btmp {
    missingok
    monthly
    create 0600 root utmp
    rotate 1
}

# system-specific logs may be also be configured here.

Is your cron daemon running okay? I'm looking further into this, but please check the above while I do.

[OptimusB]

Code: Select all

# cat nagioslogserver
/var/log/logstash/*.log {
        daily
        rotate 7
        copytruncate
        compress
        missingok
        notifempty
}

/var/log/elasticsearch/*.log {
        daily
        rotate 7
        copytruncate
        compress
        missingok
        notifempty
}

Code: Select all

# cat /etc/cron.daily/logrotate
#!/bin/sh

/usr/sbin/logrotate /etc/logrotate.conf >/dev/null 2>&1
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
    /usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
exit 0

Code: Select all

# cat /etc/logrotate.conf
# see "man logrotate" for details
# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# use date as a suffix of the rotated file
dateext

# uncomment this if you want your log files compressed
#compress

# RPM packages drop log rotation information into this directory
include /etc/logrotate.d

# no packages own wtmp and btmp -- we'll rotate them here
/var/log/wtmp {
    monthly
    create 0664 root utmp
        minsize 1M
    rotate 1
}

/var/log/btmp {
    missingok
    monthly
    create 0600 root utmp
    rotate 1
}

cron logs looks like it starts...

Code: Select all

# tail -2500 cron | grep cron
May 13 20:01:01 kdcbchngols01 CROND[27537]: (root) CMD (run-parts /etc/cron.hourly)
May 13 20:01:01 kdcbchngols01 run-parts(/etc/cron.hourly)[27537]: starting 0anacron
May 13 20:01:01 kdcbchngols01 run-parts(/etc/cron.hourly)[27550]: finished 0anacron
May 13 21:01:01 kdcbchngols01 CROND[31815]: (root) CMD (run-parts /etc/cron.hourly)
May 13 21:01:01 kdcbchngols01 run-parts(/etc/cron.hourly)[31815]: starting 0anacron
May 13 21:01:01 kdcbchngols01 run-parts(/etc/cron.hourly)[31828]: finished 0anacron
May 13 22:01:01 kdcbchngols01 CROND[3887]: (root) CMD (run-parts /etc/cron.hourly)
May 13 22:01:01 kdcbchngols01 run-parts(/etc/cron.hourly)[3887]: starting 0anacron
May 13 22:01:01 kdcbchngols01 run-parts(/etc/cron.hourly)[3901]: finished 0anacron
May 13 23:01:01 kdcbchngols01 CROND[8217]: (root) CMD (run-parts /etc/cron.hourly)
May 13 23:01:01 kdcbchngols01 run-parts(/etc/cron.hourly)[8217]: starting 0anacron
May 13 23:01:01 kdcbchngols01 run-parts(/etc/cron.hourly)[8229]: finished 0anacron
May 14 00:01:01 kdcbchngols01 CROND[12472]: (root) CMD (run-parts /etc/cron.hourly)
May 14 00:01:01 kdcbchngols01 run-parts(/etc/cron.hourly)[12472]: starting 0anacron
May 14 00:01:01 kdcbchngols01 anacron[12485]: Anacron started on 2015-05-14
May 14 00:01:01 kdcbchngols01 run-parts(/etc/cron.hourly)[12487]: finished 0anacron
May 14 00:01:01 kdcbchngols01 anacron[12485]: Jobs will be executed sequentially
May 14 00:01:01 kdcbchngols01 anacron[12485]: Normal exit (0 jobs run)
May 14 01:01:01 kdcbchngols01 CROND[16740]: (root) CMD (run-parts /etc/cron.hourly)
May 14 01:01:01 kdcbchngols01 run-parts(/etc/cron.hourly)[16740]: starting 0anacron
May 14 01:01:01 kdcbchngols01 anacron[16753]: Anacron started on 2015-05-14
May 14 01:01:01 kdcbchngols01 run-parts(/etc/cron.hourly)[16755]: finished 0anacron
May 14 01:01:01 kdcbchngols01 anacron[16753]: Jobs will be executed sequentially
May 14 01:01:01 kdcbchngols01 anacron[16753]: Normal exit (0 jobs run)
May 14 02:01:01 kdcbchngols01 CROND[20923]: (root) CMD (run-parts /etc/cron.hourly)
May 14 02:01:01 kdcbchngols01 run-parts(/etc/cron.hourly)[20923]: starting 0anacron
May 14 02:01:01 kdcbchngols01 anacron[20935]: Anacron started on 2015-05-14
May 14 02:01:01 kdcbchngols01 run-parts(/etc/cron.hourly)[20937]: finished 0anacron
May 14 02:01:01 kdcbchngols01 anacron[20935]: Jobs will be executed sequentially
May 14 02:01:01 kdcbchngols01 anacron[20935]: Normal exit (0 jobs run)
May 14 03:01:01 kdcbchngols01 CROND[25211]: (root) CMD (run-parts /etc/cron.hourly)
May 14 03:01:01 kdcbchngols01 run-parts(/etc/cron.hourly)[25211]: starting 0anacron
May 14 03:01:01 kdcbchngols01 anacron[25220]: Anacron started on 2015-05-14
May 14 03:01:01 kdcbchngols01 anacron[25220]: Will run job `cron.daily' in 8 min.
May 14 03:01:01 kdcbchngols01 anacron[25220]: Jobs will be executed sequentially
May 14 03:01:01 kdcbchngols01 run-parts(/etc/cron.hourly)[25222]: finished 0anacron
May 14 03:09:01 kdcbchngols01 anacron[25220]: Job `cron.daily' started
May 14 03:09:01 kdcbchngols01 run-parts(/etc/cron.daily)[25807]: starting logrotate
May 14 03:09:01 kdcbchngols01 run-parts(/etc/cron.daily)[25816]: finished logrotate
May 14 03:09:01 kdcbchngols01 anacron[25220]: Job `cron.daily' terminated
May 14 03:09:01 kdcbchngols01 anacron[25220]: Normal exit (1 job run)
May 14 04:01:01 kdcbchngols01 CROND[29481]: (root) CMD (run-parts /etc/cron.hourly)
May 14 04:01:01 kdcbchngols01 run-parts(/etc/cron.hourly)[29481]: starting 0anacron
May 14 04:01:01 kdcbchngols01 run-parts(/etc/cron.hourly)[29494]: finished 0anacron

[OptimusB]

FYI I have another cluster in the environment I am configuring and looks like the logs are in place since April 29th and did not purge.
Cron daily is running on this cluster as well. Cheers.

[jolson]

I think that this is a bug in the way logrotate is handling the elasticsearch logs. If I could get an ls of your logs, that's the last piece that I'd need to file a report:

Code: Select all

ls -lh /var/log/elasticsearch
ls -lh /var/log/logstash

Jesse

[OptimusB]

Code: Select all

# ls -lh /var/log/elasticsearch
total 36M
-rw-r--r-- 1 nagios users 74K May 14 13:02 87f95151-7003-42fc-a76a-bc101723dfc0.log
-rw-r--r-- 1 nagios users 36M May 10 03:33 87f95151-7003-42fc-a76a-bc101723dfc0.log-20150510.gz
-rw-r--r-- 1 nagios users 12K May 11 22:43 87f95151-7003-42fc-a76a-bc101723dfc0.log.2015-05-11
-rw-r--r-- 1 nagios users 178 May 11 03:42 87f95151-7003-42fc-a76a-bc101723dfc0.log-20150511.gz
-rw-r--r-- 1 nagios users 12K May 12 22:43 87f95151-7003-42fc-a76a-bc101723dfc0.log.2015-05-12
-rw-r--r-- 1 nagios users 177 May 12 03:13 87f95151-7003-42fc-a76a-bc101723dfc0.log-20150512.gz
-rw-r--r-- 1 nagios users 11K May 13 22:43 87f95151-7003-42fc-a76a-bc101723dfc0.log.2015-05-13
-rw-r--r-- 1 nagios users 188 May 13 03:20 87f95151-7003-42fc-a76a-bc101723dfc0.log-20150513.gz
-rw-r--r-- 1 nagios users 182 May 14 03:09 87f95151-7003-42fc-a76a-bc101723dfc0.log-20150514.gz

Code: Select all

# ls -lh /var/log/logstash
total 460K
-rw-r--r-- 1 nagios users 180K May 14 12:19 logstash.log
-rw-r--r-- 1 nagios users 234K Mar  3 04:37 logstash.log-20150303
-rw-r--r-- 1 nagios users  19K Apr 26 03:06 logstash.log-20150426.gz
-rw-r--r-- 1 nagios users 2.8K Apr 27 03:49 logstash.log-20150427.gz
-rw-r--r-- 1 nagios users  583 Apr 28 03:31 logstash.log-20150428.gz
-rw-r--r-- 1 nagios users  821 Apr 29 04:14 logstash.log-20150429.gz
-rw-r--r-- 1 nagios users  529 Apr 30 04:31 logstash.log-20150430.gz
-rw-r--r-- 1 nagios users  514 May  5 03:43 logstash.log-20150505.gz
-rw-r--r-- 1 nagios users  661 May  7 03:07 logstash.log-20150507.gz

FYI I cleared out the logs as it was taking up too much space on our test environment. Here's also a sample from the prod size that isn't in production yet.

Code: Select all

# ls -lh /var/log/elasticsearch
total 44K
-rw-r--r-- 1 nagios users    0 Apr 29 16:40 80e9022e-f73f-429e-8927-f23d0d88dfd2_index_indexing_slowlog.log
-rw-r--r-- 1 nagios users    0 Apr 29 16:40 80e9022e-f73f-429e-8927-f23d0d88dfd2_index_search_slowlog.log
-rw-r--r-- 1 nagios users    0 May 14 03:41 80e9022e-f73f-429e-8927-f23d0d88dfd2.log
-rw-r--r-- 1 nagios users    0 Apr 30 04:17 80e9022e-f73f-429e-8927-f23d0d88dfd2.log.2015-04-29
-rw-r--r-- 1 nagios users    0 May  1 03:27 80e9022e-f73f-429e-8927-f23d0d88dfd2.log.2015-04-30
-rw-r--r-- 1 nagios users    0 May  2 03:48 80e9022e-f73f-429e-8927-f23d0d88dfd2.log.2015-05-01
-rw-r--r-- 1 nagios users    0 May  3 03:34 80e9022e-f73f-429e-8927-f23d0d88dfd2.log.2015-05-02
-rw-r--r-- 1 nagios users    0 May  4 03:14 80e9022e-f73f-429e-8927-f23d0d88dfd2.log.2015-05-03
-rw-r--r-- 1 nagios users    0 May  5 03:30 80e9022e-f73f-429e-8927-f23d0d88dfd2.log.2015-05-04
-rw-r--r-- 1 nagios users    0 May  6 03:45 80e9022e-f73f-429e-8927-f23d0d88dfd2.log.2015-05-05
-rw-r--r-- 1 nagios users  12K May  6 11:07 80e9022e-f73f-429e-8927-f23d0d88dfd2.log.2015-05-06
-rw-r--r-- 1 nagios users    0 May  8 03:22 80e9022e-f73f-429e-8927-f23d0d88dfd2.log.2015-05-07
-rw-r--r-- 1 nagios users 6.4K May  8 03:22 80e9022e-f73f-429e-8927-f23d0d88dfd2.log-20150508.gz
-rw-r--r-- 1 nagios users    0 May  9 03:27 80e9022e-f73f-429e-8927-f23d0d88dfd2.log.2015-05-09
-rw-r--r-- 1 nagios users 2.1K May  9 03:27 80e9022e-f73f-429e-8927-f23d0d88dfd2.log-20150509.gz
-rw-r--r-- 1 nagios users    0 May 10 03:42 80e9022e-f73f-429e-8927-f23d0d88dfd2.log.2015-05-10
-rw-r--r-- 1 nagios users 2.3K May 10 03:42 80e9022e-f73f-429e-8927-f23d0d88dfd2.log-20150510.gz
-rw-r--r-- 1 nagios users    0 May 11 03:22 80e9022e-f73f-429e-8927-f23d0d88dfd2.log.2015-05-11
-rw-r--r-- 1 nagios users 2.3K May 11 03:22 80e9022e-f73f-429e-8927-f23d0d88dfd2.log-20150511.gz
-rw-r--r-- 1 nagios users    0 May 12 03:11 80e9022e-f73f-429e-8927-f23d0d88dfd2.log.2015-05-12
-rw-r--r-- 1 nagios users 2.4K May 12 03:11 80e9022e-f73f-429e-8927-f23d0d88dfd2.log-20150512.gz
-rw-r--r-- 1 nagios users    0 May 13 03:45 80e9022e-f73f-429e-8927-f23d0d88dfd2.log.2015-05-13
-rw-r--r-- 1 nagios users 2.4K May 13 03:45 80e9022e-f73f-429e-8927-f23d0d88dfd2.log-20150513.gz
-rw-r--r-- 1 nagios users 2.5K May 14 03:41 80e9022e-f73f-429e-8927-f23d0d88dfd2.log-20150514.gz

# ls -lh /var/log/logstash
total 76K
-rw-r--r-- 1 nagios users    0 May  7 03:41 logstash.log
-rw-r--r-- 1 nagios users 1.5K Apr 30 04:17 logstash.log-20150430.gz
-rw-r--r-- 1 nagios users  69K May  7 03:41 logstash.log-20150507.gz

[jolson]

Thanks Optimus, I'll get with the developers and see if we can pinpoint what might be wrong. At the moment, I think it has something to do with the naming scheme combined with the compression.