Logs not rotating

[BanditBBS]

On my XI 5.6.3 host my drive just fille dup because the log files located in /usr/local/nagiosxi/var are not automatically rotating and sysstat.log was 15GB! It doesn't look like any of the log file sin that folder are rotating, any idea why and how to resolve!

[scottwilkerson]

Can you show the output of this

Code: Select all

cat /etc/logrotate.d/nagiosxi

I would expect something like

Code: Select all

/usr/local/nagiosxi/var/*log {
    missingok
    notifempty
    size 5M
    rotate 1
    compress
}

/usr/local/nagiosxi/var/xidebug.log {
    missingok
    notifempty
    size 100M
    create 0660 apache nagios
    rotate 1
    compress
}

/usr/local/nagiosxi/var/xidebug.log.backtrace {
    missingok
    notifempty
    size 100M
    create 0660 apache nagios
    rotate 1
    compress
}

also, do you have logrotate here?

Code: Select all

ll /etc/cron.daily/logrotate

Finally, if so does it include the /etc/logrotate.d directory?

Code: Select all

grep include /etc/logrotate.conf

[benjaminsmith]

Hello @BanditBBS,

Nagios XI uses the logrotate program to hanndle compresssion and archiving of the log files in /usr/local/nagiosxi/var. This is setup as a daily cron ( see /etc/cron.daily ).

It's likely the cron job is not running or you have some kind of permissions issues. Please search or post the cron log in /var/log/cron to determine if this program is running or other related errors.

The configuration file can be found at /etc/logrotate.d/nagiosxi.

Let us know if you get it resolved.

[BanditBBS]

All the configuration exists like both of you to asked. here is the tail end of the cron log, see anything missing?

Code: Select all

Aug 28 15:55:01 arco-chi-pap01 CROND[31506]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php >> /usr/local/nagiosxi/var/cleaner.log 2>&1)
Aug 28 15:55:01 arco-chi-pap01 CROND[31509]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/snmptt_service_results.php >> /usr/local/nagiosxi/var/snmptt_service_results.log 2>&1)
Aug 28 15:55:01 arco-chi-pap01 CROND[31510]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lib/mrtg/mrtg.lock --confcache-file /var/lib/mrtg/mrtg.ok --user=nagios --group=nagios --user=nagios --group=nagios)
Aug 28 15:55:01 arco-chi-pap01 CROND[31511]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php >> /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Aug 28 15:55:01 arco-chi-pap01 CROND[31513]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php >> /usr/local/nagiosxi/var/nom.log 2>&1)
Aug 28 15:55:01 arco-chi-pap01 CROND[31515]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php >> /usr/local/nagiosxi/var/dbmaint.log 2>&1)
Aug 28 15:55:01 arco-chi-pap01 CROND[31518]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php >> /usr/local/nagiosxi/var/eventman.log 2>&1)
Aug 28 15:55:01 arco-chi-pap01 CROND[31520]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php >> /usr/local/nagiosxi/var/reportengine.log 2>&1)
Aug 28 15:55:01 arco-chi-pap01 CROND[31522]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php >> /usr/local/nagiosxi/var/feedproc.log 2>&1)
Aug 28 15:55:01 arco-chi-pap01 CROND[31523]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php >> /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Aug 28 15:55:01 arco-chi-pap01 CROND[31529]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/deadpool.php >> /usr/local/nagiosxi/var/deadpool.log 2>&1)
Aug 28 15:55:01 arco-chi-pap01 CROND[31521]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php >> /usr/local/nagiosxi/var/event_handler.log 2>&1)
Aug 28 15:55:01 arco-chi-pap01 CROND[31519]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php >> /usr/local/nagiosxi/var/sysstat.log 2>&1)
Aug 28 15:56:01 arco-chi-pap01 CROND[2113]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php >> /usr/local/nagiosxi/var/cleaner.log 2>&1)
Aug 28 15:56:01 arco-chi-pap01 CROND[2114]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php >> /usr/local/nagiosxi/var/reportengine.log 2>&1)
Aug 28 15:56:01 arco-chi-pap01 CROND[2115]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/deadpool.php >> /usr/local/nagiosxi/var/deadpool.log 2>&1)
Aug 28 15:56:01 arco-chi-pap01 CROND[2116]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/snmptt_service_results.php >> /usr/local/nagiosxi/var/snmptt_service_results.log 2>&1)
Aug 28 15:56:01 arco-chi-pap01 CROND[2118]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php >> /usr/local/nagiosxi/var/sysstat.log 2>&1)
Aug 28 15:56:01 arco-chi-pap01 CROND[2123]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php >> /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Aug 28 15:56:01 arco-chi-pap01 CROND[2127]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php >> /usr/local/nagiosxi/var/event_handler.log 2>&1)
Aug 28 15:56:01 arco-chi-pap01 CROND[2129]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php >> /usr/local/nagiosxi/var/eventman.log 2>&1)
Aug 28 15:56:01 arco-chi-pap01 CROND[2130]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php >> /usr/local/nagiosxi/var/feedproc.log 2>&1)
Aug 28 15:56:01 arco-chi-pap01 CROND[2122]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php >> /usr/local/nagiosxi/var/nom.log 2>&1)
Aug 28 15:56:01 arco-chi-pap01 CROND[2134]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php >> /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)

[scottwilkerson]

Can you show the output of the following:

Code: Select all

grep logrotate /var/log/cron

[BanditBBS]

I think I may be good...leave this open until Tuesday(I'm off until then )

I manually ran logrotate and got a bunch of these errors:

Code: Select all

error: skipping "/usr/local/nagiosxi/var/sysstat.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.

I removed the group write permission and I think it ran last night. I'll keep an eye on it the next couple days to make sure they are still rotating.

[scottwilkerson]

I think I may be good...leave this open until Tuesday(I'm off until then )

I manually ran logrotate and got a bunch of these errors:

Code: Select all

error: skipping "/usr/local/nagiosxi/var/sysstat.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.

I removed the group write permission and I think it ran last night. I'll keep an eye on it the next couple days to make sure they are still rotating.

Sounds good

[[email protected]]

Gents,

At one of your customer I also found an issue related to permissions of cron file /etc/logrotate.d/nagiosxi that prevented logrotate from starting aforementioned file.

Nagios XI log files were not rotated at all (sysstat.log was 6 GB) so I ran a logrotate in debug mode
/usr/sbin/logrotate -d -s /var/lib/logrotate/logrotate.status /etc/logrotate.conf

and found the reason the cron file was completely ignoted in the error output:
Ignoring nagiosxi because the file owner is wrong (should be root).

After changing ownership to root it was able to rotate all files underneath /usr/local/nagiosxi/var/ ending on 'log'

@scott

Although "su nagios nagios" was introduced in XI version 5.6.6 for the first cron paragraph there's still some work to do on other files and folders:
error: skipping "/usr/local/nagiosxi/var/xidebug.log" because parent directory has insecure permissions
error: skipping "/usr/local/nagiosxi/var/xidebug.log.backtrace" because parent directory has insecure permissions
error: skipping "/var/log/snmptt/snmptt.log" because parent directory has insecure permissions
error: skipping "/var/log/snmptt/snmpttsystem.log" because parent directory has insecure permissions
error: skipping "/var/log/snmptt/snmpttunknown.log" because parent directory has insecure permissions
error: skipping "/var/log/snmptt/snmptthandler.debug" because parent directory has insecure permissions
error: skipping "/var/log/snmptt/snmptt.debug" because parent directory has insecure permissions

Regards, Jørgen van der Meulen

[[email protected]]

Update:

(..)
Nagios XI log files were not rotated at all (sysstat.log was 6 GB) so I ran a logrotate in debug mode
/usr/sbin/logrotate -d -s /var/lib/logrotate/logrotate.status /etc/logrotate.conf

and found the reason the cron file was completely ignoted in the error output:
Ignoring nagiosxi because the file owner is wrong (should be root).
(..)

On another customer's system /etc/cron.d/nagiosxi was owned by root but file permissions were invalid. Running logrotate in debug mode gave me this:
error: Ignoring nagiosxi because of bad file mode - must be 0644 or 0444.

In my opinion this is definitely that should be checked during a minor release (XI 5.6.7?) by the installer.

[scottwilkerson]

On another customer's system /etc/cron.d/nagiosxi was owned by root but file permissions were invalid. Running logrotate in debug mode gave me this:
error: Ignoring nagiosxi because of bad file mode - must be 0644 or 0444.

In my opinion this is definitely that should be checked during a minor release (XI 5.6.7?) by the installer.

The installer and upgrade scripts literally run this

Code: Select all

install -m 644 nagiosxi/nagiosxi.cron.d /etc/cron.d/nagiosxi 

I'm not sure how your user had permissions other than 644