XI logs taking up too much space

[vazudevan]

The nom.log and sysstat.log logs are taking up too huge a space and there appears to be no log rotation or purging.

Code: Select all

[root@nagios var]# ls -lh nom.log sysstat.log
-rw-r--r-- 1 nagios nagios 22G Aug 16 01:47 nom.log
-rw-r--r-- 1 nagios nagios 26G Aug 16 01:47 sysstat.log

Also, we are seeing a lot of these errors in nom.log

Code: Select all

chown: changing ownership of ‘/var/nagiosramdisk/spool/checkresults/XXXXXX’: Operation not permitted

Followed by

Code: Select all

Password: su: Authentication failure
ERROR: Could not create or update '/usr/local/nagios/var/nagios.configtest'
Config test failed.  Checkpoint aborted.

occurring every minute or so.
1. Why are these errors?
2. How to correct them?
3. How to tame these logs ?

[cdienger]

It looks like permission or write problems. First, feel free to remove the logs with "rm /usr/local/nagiosxi/var/sysstat.log" and "rm /usr/local/nagiosxi/var/nom.log".

Then check that /var/nagiosramdisk has rw permissions with "mount". It should look something like:

tmpfs on /var/nagiosramdisk type tmpfs (rw,size=100m)

/etc/sudoers should also contain entries like the following:

User_Alias NAGIOSXI=nagios
User_Alias NAGIOSXIWEB=apache
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios start
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios stop
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios restart
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios reload
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios status
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios checkconfig
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db start
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db stop
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db restart
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db reload
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db status
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd start
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd stop
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd restart
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd reload
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd status
NAGIOSXI ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/html/includes/components/autodiscovery/scripts/autodiscover_new.php *
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/html/includes/components/profile/getprofile.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/upgrade_to_latest.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/change_timezone.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/manage_services.sh *
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/reset_config_perms.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/manage_ssl_config.sh *
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/backup_xi.sh *
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/messages
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/httpd/error_log
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/mysqld.log
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/html/includes/components/autodiscovery/scripts/autodiscover_new.php *
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/html/includes/components/profile/getprofile.sh
NAGIOSXIWEB ALL = NOPASSWD:/etc/init.d/snmptt restart
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/scripts/repair_databases.sh
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/scripts/manage_services.sh *

Finally, check if the naigos account has expired with "chage -l nagios" and update it with "chage -I -1 -m 0 -M 99999 -E -1 nagios" if needed.

[vazudevan]

Thank you for the details, Checked on these and verified the permissions. Sudoers entry are exactly the same. They all appear to be fine. With the Ramdisk we had the nom and sys stat logs getting in GBs very quickly.

Reverted back from Ramdisk to local disks and things seems to have settled down now. Those errors are not to be seen in nom.log anymore. There is only one error that is repeating on every cycle of nom cron.

Code: Select all

Password: su: Authentication failure
ERROR: Could not create or update '/usr/local/nagios/var/nagios.configtest'
Config test failed.  Checkpoint aborted.

[cdienger]

The block of code that is causing that error is in /etc/init.d/nagios:

Code: Select all

        if ! su $NagiosUser -c "touch $NagiosCfgtestFile"; then
                echo "ERROR: Could not create or update '$NagiosCfgtestFile'"
                exit 8
        fi

Do you get anything interesting if you run:

su nagios -c "touch /usr/local/nagiosxi/var/nom.log"

?

[vazudevan]

it did not make a difference.

As stated this error is seen in nom.log and is related exactly every minute. its not coming from the init script, I believe its instead from the cron

Code: Select all

* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/nom.php >> /usr/local/nagiosxi/var/nom.log 2>&1

[cdienger]

Sorry - typo on my end. The command should have been:

su nagios -c "touch /usr/local/nagios/var/nagios.configtest"

This is the command that is automatically running and producing the error. By running it manually at the command line I'm not expecting it to "do" anything really besides hopefully giving us a clue as to why it isn't working.

[lmiltchev]

You should see the following error message:

Config test failed. Checkpoint aborted.

when configuration verification fails. What is the output that you see after running the following command from the CLI?

Code: Select all

/usr/local/nagiosxi/scripts/nom_create_nagioscore_checkpoint_cond.sh

[vazudevan]

Code: Select all

/usr/local/nagiosxi/scripts/nom_create_nagioscore_checkpoint_cond.sh
OK.
RESETTING PERMS
/usr/local/nagiosxi/nom/checkpoints/nagioscore ~
tar: Removing leading `/' from member names
~
Config test passed.  Checkpoint created.

[scottwilkerson]

Please show the output of the following

Code: Select all

ls -al /usr/local/nagios/var/nagios.configtest

If it is owned by something other than nagios run

Code: Select all

chown nagios /usr/local/nagios/var/nagios.configtest

[vazudevan]

Code: Select all

[root@phlprcnagnxi001 etc]# ls -l /usr/local/nagios | grep var
drwxr-xr-x  6 nagios nagios 4096 Aug 24 15:04 var
[root@phlprcnagnxi001 etc]# ls -al /usr/local/nagios/var/nagios.configtest
ls: cannot access /usr/local/nagios/var/nagios.configtest: No such file or directory
[root@phlprcnagnxi001 etc]#