We received a low disk space alert for our XI server and upon investigation it appears the log files have grown to 250+GB. Any advice on how to troubleshoot the cause would be greatly appreciated. I attached a screen cap showing the logs files in question.
Issue with growing log files
Issue with growing log files
You do not have the required permissions to view the files attached to this post.
-
benjaminsmith
- Posts: 5324
- Joined: Wed Aug 22, 2018 4:39 pm
- Location: saint paul
Re: Issue with growing log files
Hi @dshearon,
Nagios XI uses the logrotate program to handle compression and archiving of the log files in /usr/local/nagiosxi/var. This is set up as a daily cron ( see /etc/cron.daily ).
Check to see if cron is running on the server:
It's likely the cron job is not running or you have some kind of permissions issues. Please search or post the cron log in /var/log/cron to determine if this program is running or other related errors.
Let us know what you find out.
Nagios XI uses the logrotate program to handle compression and archiving of the log files in /usr/local/nagiosxi/var. This is set up as a daily cron ( see /etc/cron.daily ).
Check to see if cron is running on the server:
Code: Select all
systemctl status crond
Let us know what you find out.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
Be sure to check out our Knowledgebase for helpful articles and solutions!
Be sure to check out our Knowledgebase for helpful articles and solutions!
Re: Issue with growing log files
I attached a screen cap of the service running and the cron log file. I didn't notice anything that stood out but I may be missing something.
You do not have the required permissions to view the files attached to this post.
-
benjaminsmith
- Posts: 5324
- Joined: Wed Aug 22, 2018 4:39 pm
- Location: saint paul
Re: Issue with growing log files
Hi @dshearon,
Thank you for the log. Logrotate is running.
Thank you for the log. Logrotate is running.
Let's manually run it in debug mode, and post the output to the thread, it may be skipping over those directories.Jul 27 03:36:01 cltisvcorpmon01 run-parts(/etc/cron.daily)[227736]: starting logrotate
Jul 27 03:36:01 cltisvcorpmon01 run-parts(/etc/cron.daily)[227775]: finished logrotate
Code: Select all
logrotate -d /etc/logrotate.d/nagiosxi
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
Be sure to check out our Knowledgebase for helpful articles and solutions!
Be sure to check out our Knowledgebase for helpful articles and solutions!
Re: Issue with growing log files
Output is below, under that is the current owner/group of the file in question
[root@cltisvcorpmon01 ~]# logrotate -d /etc/logrotate.d/nagiosxi
Ignoring /etc/logrotate.d/nagiosxi because the file owner is wrong (should be root).
Allocating hash table for state file, size 15360 B
[root@cltisvcorpmon01 logrotate.d]# ls -l nagiosxi
-rw-r--r--. 1 nagios users 389 Dec 5 2017 nagiosxi
[root@cltisvcorpmon01 ~]# logrotate -d /etc/logrotate.d/nagiosxi
Ignoring /etc/logrotate.d/nagiosxi because the file owner is wrong (should be root).
Allocating hash table for state file, size 15360 B
[root@cltisvcorpmon01 logrotate.d]# ls -l nagiosxi
-rw-r--r--. 1 nagios users 389 Dec 5 2017 nagiosxi
-
benjaminsmith
- Posts: 5324
- Joined: Wed Aug 22, 2018 4:39 pm
- Location: saint paul
Re: Issue with growing log files
HI,
Try changing it back to root and 644, and then run it manually in debug mode again.
Interesting. What version of Nagios XI are you using, this is what I have on my 5.7.2 test server:Ignoring /etc/logrotate.d/nagiosxi because the file owner is wrong (should be root).
Code: Select all
[root@localhost cron.d]# ls -l /etc/logrotate.d/nagiosxi
-rw-r--r--. 1 root root 605 Jul 24 16:30 /etc/logrotate.d/nagiosxi
Code: Select all
chmod 644 /etc/logrotate.d/nagiosxi
chown root:root /etc/logrotate.d/nagiosxi
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
Be sure to check out our Knowledgebase for helpful articles and solutions!
Be sure to check out our Knowledgebase for helpful articles and solutions!
Re: Issue with growing log files
We're on version 5.6.1 right now with plans to upgrade to 5.7 once we get the disk space back.
I reset permissions and manually ran the job, output is below.
Handling 3 logs
rotating pattern: /usr/local/nagiosxi/var/*log 5242880 bytes (1 rotations)
empty log files are not rotated, old logs are removed
considering log /usr/local/nagiosxi/var/cleaner.log
error: skipping "/usr/local/nagiosxi/var/cleaner.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /usr/local/nagiosxi/var/cmdsubsys.log
error: skipping "/usr/local/nagiosxi/var/cmdsubsys.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /usr/local/nagiosxi/var/dbmaint.log
error: skipping "/usr/local/nagiosxi/var/dbmaint.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /usr/local/nagiosxi/var/deadpool.log
error: skipping "/usr/local/nagiosxi/var/deadpool.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /usr/local/nagiosxi/var/event_handler.log
error: skipping "/usr/local/nagiosxi/var/event_handler.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /usr/local/nagiosxi/var/eventman.log
error: skipping "/usr/local/nagiosxi/var/eventman.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /usr/local/nagiosxi/var/feedproc.log
error: skipping "/usr/local/nagiosxi/var/feedproc.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /usr/local/nagiosxi/var/load_url.log
error: skipping "/usr/local/nagiosxi/var/load_url.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /usr/local/nagiosxi/var/nom.log
error: skipping "/usr/local/nagiosxi/var/nom.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /usr/local/nagiosxi/var/perfdataproc.log
error: skipping "/usr/local/nagiosxi/var/perfdataproc.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /usr/local/nagiosxi/var/recurringdowntime.log
error: skipping "/usr/local/nagiosxi/var/recurringdowntime.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /usr/local/nagiosxi/var/reportengine.log
error: skipping "/usr/local/nagiosxi/var/reportengine.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /usr/local/nagiosxi/var/scheduledreporting.log
error: skipping "/usr/local/nagiosxi/var/scheduledreporting.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /usr/local/nagiosxi/var/sysstat.log
error: skipping "/usr/local/nagiosxi/var/sysstat.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
rotating pattern: /usr/local/nagiosxi/var/xidebug.log 104857600 bytes (1 rotations)
empty log files are not rotated, old logs are removed
considering log /usr/local/nagiosxi/var/xidebug.log
error: skipping "/usr/local/nagiosxi/var/xidebug.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
rotating pattern: /usr/local/nagiosxi/var/xidebug.log.backtrace 104857600 bytes (1 rotations)
empty log files are not rotated, old logs are removed
considering log /usr/local/nagiosxi/var/xidebug.log.backtrace
error: skipping "/usr/local/nagiosxi/var/xidebug.log.backtrace" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
I reset permissions and manually ran the job, output is below.
Handling 3 logs
rotating pattern: /usr/local/nagiosxi/var/*log 5242880 bytes (1 rotations)
empty log files are not rotated, old logs are removed
considering log /usr/local/nagiosxi/var/cleaner.log
error: skipping "/usr/local/nagiosxi/var/cleaner.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /usr/local/nagiosxi/var/cmdsubsys.log
error: skipping "/usr/local/nagiosxi/var/cmdsubsys.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /usr/local/nagiosxi/var/dbmaint.log
error: skipping "/usr/local/nagiosxi/var/dbmaint.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /usr/local/nagiosxi/var/deadpool.log
error: skipping "/usr/local/nagiosxi/var/deadpool.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /usr/local/nagiosxi/var/event_handler.log
error: skipping "/usr/local/nagiosxi/var/event_handler.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /usr/local/nagiosxi/var/eventman.log
error: skipping "/usr/local/nagiosxi/var/eventman.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /usr/local/nagiosxi/var/feedproc.log
error: skipping "/usr/local/nagiosxi/var/feedproc.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /usr/local/nagiosxi/var/load_url.log
error: skipping "/usr/local/nagiosxi/var/load_url.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /usr/local/nagiosxi/var/nom.log
error: skipping "/usr/local/nagiosxi/var/nom.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /usr/local/nagiosxi/var/perfdataproc.log
error: skipping "/usr/local/nagiosxi/var/perfdataproc.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /usr/local/nagiosxi/var/recurringdowntime.log
error: skipping "/usr/local/nagiosxi/var/recurringdowntime.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /usr/local/nagiosxi/var/reportengine.log
error: skipping "/usr/local/nagiosxi/var/reportengine.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /usr/local/nagiosxi/var/scheduledreporting.log
error: skipping "/usr/local/nagiosxi/var/scheduledreporting.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /usr/local/nagiosxi/var/sysstat.log
error: skipping "/usr/local/nagiosxi/var/sysstat.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
rotating pattern: /usr/local/nagiosxi/var/xidebug.log 104857600 bytes (1 rotations)
empty log files are not rotated, old logs are removed
considering log /usr/local/nagiosxi/var/xidebug.log
error: skipping "/usr/local/nagiosxi/var/xidebug.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
rotating pattern: /usr/local/nagiosxi/var/xidebug.log.backtrace 104857600 bytes (1 rotations)
empty log files are not rotated, old logs are removed
considering log /usr/local/nagiosxi/var/xidebug.log.backtrace
error: skipping "/usr/local/nagiosxi/var/xidebug.log.backtrace" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
Re: Issue with growing log files
Here are the set permissions for that directory.
[root@cltisvcorpmon01 ~]# ls -l /usr/local/nagiosxi/
total 64
drwxr-xr-x 2 root nagios 308 Jul 24 2018 cron
drwxr-xr-x 4 root nagios 67 May 3 2019 etc
drwxr-xr-x 19 root nagios 4096 May 1 2018 html
-rw-r--r-- 1 root nagios 0 Apr 23 2015 is.installed
drwxr-xr-x 3 root nagios 25 Apr 23 2015 nom
drwxr-xr-x 4 root nagios 4096 May 3 2019 scripts
drwsrwsr-x 4 root nagios 32768 Jul 10 11:45 tmp
drwxr-xr-x 2 root nagios 4096 Dec 5 2017 tools
drwxrwxr-x 7 root nagios 4096 Jul 30 15:47 var
[root@cltisvcorpmon01 ~]# ls -l /usr/local/nagiosxi/
total 64
drwxr-xr-x 2 root nagios 308 Jul 24 2018 cron
drwxr-xr-x 4 root nagios 67 May 3 2019 etc
drwxr-xr-x 19 root nagios 4096 May 1 2018 html
-rw-r--r-- 1 root nagios 0 Apr 23 2015 is.installed
drwxr-xr-x 3 root nagios 25 Apr 23 2015 nom
drwxr-xr-x 4 root nagios 4096 May 3 2019 scripts
drwsrwsr-x 4 root nagios 32768 Jul 10 11:45 tmp
drwxr-xr-x 2 root nagios 4096 Dec 5 2017 tools
drwxrwxr-x 7 root nagios 4096 Jul 30 15:47 var
Re: Issue with growing log files
Please attach this file:
We'll need to add "su root nagios" OR "su nagios nagios" to the /etc/logrotate.d/nagiosxi definitions for each.
Like this:
https://support.microfocus.com/kb/doc.php?id=7005219
Code: Select all
/etc/logrotate.d/nagiosxiLike this:
https://support.microfocus.com/kb/doc.php?id=7005219
Re: Issue with growing log files
The requested file is attached. I needed to add the .txt extension to allow the attachment.
You do not have the required permissions to view the files attached to this post.