Nagios Support Forum

We received a low disk space alert for our XI server and upon investigation it appears the log files have grown to 250+GB. Any advice on how to troubleshoot the cause would be greatly appreciated. I attached a screen cap showing the logs files in question.

Hi @dshearon,

Nagios XI uses the logrotate program to handle compression and archiving of the log files in /usr/local/nagiosxi/var. This is set up as a daily cron ( see /etc/cron.daily ).

Check to see if cron is running on the server: It's likely the cron job is not running or you have some kind of permissions issues. Please search or post the cron log in /var/log/cron to determine if this program is running or other related errors.

Let us know what you find out.

I attached a screen cap of the service running and the cron log file. I didn't notice anything that stood out but I may be missing something.

Hi @dshearon,

Thank you for the log. Logrotate is running.

Jul 27 03:36:01 cltisvcorpmon01 run-parts(/etc/cron.daily)[227736]: starting logrotate
Jul 27 03:36:01 cltisvcorpmon01 run-parts(/etc/cron.daily)[227775]: finished logrotate

Let's manually run it in debug mode, and post the output to the thread, it may be skipping over those directories.

Output is below, under that is the current owner/group of the file in question

[root@cltisvcorpmon01 ~]# logrotate -d /etc/logrotate.d/nagiosxi
Ignoring /etc/logrotate.d/nagiosxi because the file owner is wrong (should be root).
Allocating hash table for state file, size 15360 B


[root@cltisvcorpmon01 logrotate.d]# ls -l nagiosxi
-rw-r--r--. 1 nagios users 389 Dec 5 2017 nagiosxi

HI,

Ignoring /etc/logrotate.d/nagiosxi because the file owner is wrong (should be root).

Interesting. What version of Nagios XI are you using, this is what I have on my 5.7.2 test server: Try changing it back to root and 644, and then run it manually in debug mode again.

We're on version 5.6.1 right now with plans to upgrade to 5.7 once we get the disk space back.

I reset permissions and manually ran the job, output is below.

Handling 3 logs

rotating pattern: /usr/local/nagiosxi/var/*log 5242880 bytes (1 rotations)
empty log files are not rotated, old logs are removed
considering log /usr/local/nagiosxi/var/cleaner.log
error: skipping "/usr/local/nagiosxi/var/cleaner.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /usr/local/nagiosxi/var/cmdsubsys.log
error: skipping "/usr/local/nagiosxi/var/cmdsubsys.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /usr/local/nagiosxi/var/dbmaint.log
error: skipping "/usr/local/nagiosxi/var/dbmaint.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /usr/local/nagiosxi/var/deadpool.log
error: skipping "/usr/local/nagiosxi/var/deadpool.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /usr/local/nagiosxi/var/event_handler.log
error: skipping "/usr/local/nagiosxi/var/event_handler.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /usr/local/nagiosxi/var/eventman.log
error: skipping "/usr/local/nagiosxi/var/eventman.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /usr/local/nagiosxi/var/feedproc.log
error: skipping "/usr/local/nagiosxi/var/feedproc.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /usr/local/nagiosxi/var/load_url.log
error: skipping "/usr/local/nagiosxi/var/load_url.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /usr/local/nagiosxi/var/nom.log
error: skipping "/usr/local/nagiosxi/var/nom.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /usr/local/nagiosxi/var/perfdataproc.log
error: skipping "/usr/local/nagiosxi/var/perfdataproc.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /usr/local/nagiosxi/var/recurringdowntime.log
error: skipping "/usr/local/nagiosxi/var/recurringdowntime.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /usr/local/nagiosxi/var/reportengine.log
error: skipping "/usr/local/nagiosxi/var/reportengine.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /usr/local/nagiosxi/var/scheduledreporting.log
error: skipping "/usr/local/nagiosxi/var/scheduledreporting.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /usr/local/nagiosxi/var/sysstat.log
error: skipping "/usr/local/nagiosxi/var/sysstat.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.

rotating pattern: /usr/local/nagiosxi/var/xidebug.log 104857600 bytes (1 rotations)
empty log files are not rotated, old logs are removed
considering log /usr/local/nagiosxi/var/xidebug.log
error: skipping "/usr/local/nagiosxi/var/xidebug.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.

rotating pattern: /usr/local/nagiosxi/var/xidebug.log.backtrace 104857600 bytes (1 rotations)
empty log files are not rotated, old logs are removed
considering log /usr/local/nagiosxi/var/xidebug.log.backtrace
error: skipping "/usr/local/nagiosxi/var/xidebug.log.backtrace" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.

Here are the set permissions for that directory.

[root@cltisvcorpmon01 ~]# ls -l /usr/local/nagiosxi/
total 64
drwxr-xr-x 2 root nagios 308 Jul 24 2018 cron
drwxr-xr-x 4 root nagios 67 May 3 2019 etc
drwxr-xr-x 19 root nagios 4096 May 1 2018 html
-rw-r--r-- 1 root nagios 0 Apr 23 2015 is.installed
drwxr-xr-x 3 root nagios 25 Apr 23 2015 nom
drwxr-xr-x 4 root nagios 4096 May 3 2019 scripts
drwsrwsr-x 4 root nagios 32768 Jul 10 11:45 tmp
drwxr-xr-x 2 root nagios 4096 Dec 5 2017 tools
drwxrwxr-x 7 root nagios 4096 Jul 30 15:47 var

Please attach this file:
We'll need to add "su root nagios" OR "su nagios nagios" to the /etc/logrotate.d/nagiosxi definitions for each.

Like this:

https://support.microfocus.com/kb/doc.php?id=7005219

The requested file is attached. I needed to add the .txt extension to allow the attachment.