Page 1 of 2

Logs not rotating

Posted: Wed Aug 28, 2019 3:31 pm
by BanditBBS
On my XI 5.6.3 host my drive just fille dup because the log files located in /usr/local/nagiosxi/var are not automatically rotating and sysstat.log was 15GB! It doesn't look like any of the log file sin that folder are rotating, any idea why and how to resolve!

Re: Logs not rotating

Posted: Wed Aug 28, 2019 3:41 pm
by scottwilkerson
Can you show the output of this

Code: Select all

cat /etc/logrotate.d/nagiosxi
I would expect something like

Code: Select all

/usr/local/nagiosxi/var/*log {
    missingok
    notifempty
    size 5M
    rotate 1
    compress
}

/usr/local/nagiosxi/var/xidebug.log {
    missingok
    notifempty
    size 100M
    create 0660 apache nagios
    rotate 1
    compress
}

/usr/local/nagiosxi/var/xidebug.log.backtrace {
    missingok
    notifempty
    size 100M
    create 0660 apache nagios
    rotate 1
    compress
}
also, do you have logrotate here?

Code: Select all

ll /etc/cron.daily/logrotate
Finally, if so does it include the /etc/logrotate.d directory?

Code: Select all

grep include /etc/logrotate.conf

Re: Logs not rotating

Posted: Wed Aug 28, 2019 3:48 pm
by benjaminsmith
Hello @BanditBBS,

Nagios XI uses the logrotate program to hanndle compresssion and archiving of the log files in /usr/local/nagiosxi/var. This is setup as a daily cron ( see /etc/cron.daily ).

It's likely the cron job is not running or you have some kind of permissions issues. Please search or post the cron log in /var/log/cron to determine if this program is running or other related errors.

The configuration file can be found at /etc/logrotate.d/nagiosxi.

Let us know if you get it resolved.

Re: Logs not rotating

Posted: Wed Aug 28, 2019 3:58 pm
by BanditBBS
All the configuration exists like both of you to asked. here is the tail end of the cron log, see anything missing?

Code: Select all

Aug 28 15:55:01 arco-chi-pap01 CROND[31506]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php >> /usr/local/nagiosxi/var/cleaner.log 2>&1)
Aug 28 15:55:01 arco-chi-pap01 CROND[31509]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/snmptt_service_results.php >> /usr/local/nagiosxi/var/snmptt_service_results.log 2>&1)
Aug 28 15:55:01 arco-chi-pap01 CROND[31510]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lib/mrtg/mrtg.lock --confcache-file /var/lib/mrtg/mrtg.ok --user=nagios --group=nagios --user=nagios --group=nagios)
Aug 28 15:55:01 arco-chi-pap01 CROND[31511]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php >> /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Aug 28 15:55:01 arco-chi-pap01 CROND[31513]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php >> /usr/local/nagiosxi/var/nom.log 2>&1)
Aug 28 15:55:01 arco-chi-pap01 CROND[31515]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php >> /usr/local/nagiosxi/var/dbmaint.log 2>&1)
Aug 28 15:55:01 arco-chi-pap01 CROND[31518]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php >> /usr/local/nagiosxi/var/eventman.log 2>&1)
Aug 28 15:55:01 arco-chi-pap01 CROND[31520]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php >> /usr/local/nagiosxi/var/reportengine.log 2>&1)
Aug 28 15:55:01 arco-chi-pap01 CROND[31522]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php >> /usr/local/nagiosxi/var/feedproc.log 2>&1)
Aug 28 15:55:01 arco-chi-pap01 CROND[31523]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php >> /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Aug 28 15:55:01 arco-chi-pap01 CROND[31529]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/deadpool.php >> /usr/local/nagiosxi/var/deadpool.log 2>&1)
Aug 28 15:55:01 arco-chi-pap01 CROND[31521]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php >> /usr/local/nagiosxi/var/event_handler.log 2>&1)
Aug 28 15:55:01 arco-chi-pap01 CROND[31519]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php >> /usr/local/nagiosxi/var/sysstat.log 2>&1)
Aug 28 15:56:01 arco-chi-pap01 CROND[2113]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php >> /usr/local/nagiosxi/var/cleaner.log 2>&1)
Aug 28 15:56:01 arco-chi-pap01 CROND[2114]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php >> /usr/local/nagiosxi/var/reportengine.log 2>&1)
Aug 28 15:56:01 arco-chi-pap01 CROND[2115]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/deadpool.php >> /usr/local/nagiosxi/var/deadpool.log 2>&1)
Aug 28 15:56:01 arco-chi-pap01 CROND[2116]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/snmptt_service_results.php >> /usr/local/nagiosxi/var/snmptt_service_results.log 2>&1)
Aug 28 15:56:01 arco-chi-pap01 CROND[2118]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php >> /usr/local/nagiosxi/var/sysstat.log 2>&1)
Aug 28 15:56:01 arco-chi-pap01 CROND[2123]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php >> /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Aug 28 15:56:01 arco-chi-pap01 CROND[2127]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php >> /usr/local/nagiosxi/var/event_handler.log 2>&1)
Aug 28 15:56:01 arco-chi-pap01 CROND[2129]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php >> /usr/local/nagiosxi/var/eventman.log 2>&1)
Aug 28 15:56:01 arco-chi-pap01 CROND[2130]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php >> /usr/local/nagiosxi/var/feedproc.log 2>&1)
Aug 28 15:56:01 arco-chi-pap01 CROND[2122]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php >> /usr/local/nagiosxi/var/nom.log 2>&1)
Aug 28 15:56:01 arco-chi-pap01 CROND[2134]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php >> /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)

Re: Logs not rotating

Posted: Thu Aug 29, 2019 7:15 am
by scottwilkerson
Can you show the output of the following:

Code: Select all

grep logrotate /var/log/cron

Re: Logs not rotating

Posted: Thu Aug 29, 2019 9:03 am
by BanditBBS
I think I may be good...leave this open until Tuesday(I'm off until then :) )

I manually ran logrotate and got a bunch of these errors:

Code: Select all

error: skipping "/usr/local/nagiosxi/var/sysstat.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
I removed the group write permission and I think it ran last night. I'll keep an eye on it the next couple days to make sure they are still rotating.

Re: Logs not rotating

Posted: Thu Aug 29, 2019 9:49 am
by scottwilkerson
BanditBBS wrote:I think I may be good...leave this open until Tuesday(I'm off until then :) )

I manually ran logrotate and got a bunch of these errors:

Code: Select all

error: skipping "/usr/local/nagiosxi/var/sysstat.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
I removed the group write permission and I think it ran last night. I'll keep an eye on it the next couple days to make sure they are still rotating.
Sounds good

Re: Logs not rotating

Posted: Fri Sep 13, 2019 4:45 am
Gents,

At one of your customer I also found an issue related to permissions of cron file /etc/logrotate.d/nagiosxi that prevented logrotate from starting aforementioned file.

Nagios XI log files were not rotated at all (sysstat.log was 6 GB) so I ran a logrotate in debug mode
/usr/sbin/logrotate -d -s /var/lib/logrotate/logrotate.status /etc/logrotate.conf

and found the reason the cron file was completely ignoted in the error output:
Ignoring nagiosxi because the file owner is wrong (should be root).

After changing ownership to root it was able to rotate all files underneath /usr/local/nagiosxi/var/ ending on 'log'

@scott

Although "su nagios nagios" was introduced in XI version 5.6.6 for the first cron paragraph there's still some work to do on other files and folders:
error: skipping "/usr/local/nagiosxi/var/xidebug.log" because parent directory has insecure permissions
error: skipping "/usr/local/nagiosxi/var/xidebug.log.backtrace" because parent directory has insecure permissions
error: skipping "/var/log/snmptt/snmptt.log" because parent directory has insecure permissions
error: skipping "/var/log/snmptt/snmpttsystem.log" because parent directory has insecure permissions
error: skipping "/var/log/snmptt/snmpttunknown.log" because parent directory has insecure permissions
error: skipping "/var/log/snmptt/snmptthandler.debug" because parent directory has insecure permissions
error: skipping "/var/log/snmptt/snmptt.debug" because parent directory has insecure permissions

Regards, Jørgen van der Meulen

Re: Logs not rotating

Posted: Fri Sep 13, 2019 7:33 am
Update:
[email protected] wrote:(..)
Nagios XI log files were not rotated at all (sysstat.log was 6 GB) so I ran a logrotate in debug mode
/usr/sbin/logrotate -d -s /var/lib/logrotate/logrotate.status /etc/logrotate.conf

and found the reason the cron file was completely ignoted in the error output:
Ignoring nagiosxi because the file owner is wrong (should be root).
(..)
On another customer's system /etc/cron.d/nagiosxi was owned by root but file permissions were invalid. Running logrotate in debug mode gave me this:
error: Ignoring nagiosxi because of bad file mode - must be 0644 or 0444.

In my opinion this is definitely that should be checked during a minor release (XI 5.6.7?) by the installer.

Re: Logs not rotating

Posted: Fri Sep 13, 2019 11:54 am
by scottwilkerson
[email protected] wrote:On another customer's system /etc/cron.d/nagiosxi was owned by root but file permissions were invalid. Running logrotate in debug mode gave me this:
error: Ignoring nagiosxi because of bad file mode - must be 0644 or 0444.

In my opinion this is definitely that should be checked during a minor release (XI 5.6.7?) by the installer.
The installer and upgrade scripts literally run this

Code: Select all

install -m 644 nagiosxi/nagiosxi.cron.d /etc/cron.d/nagiosxi 
I'm not sure how your user had permissions other than 644