Page 1 of 2
Filesystem filled up, now Nagios XI is not fully functional
Posted: Thu Mar 12, 2015 12:02 pm
by sgd
Hello, we're running Nagios XI release 2014R2.6 on a 64-bit Centos 6.0 machine (not a VM), and we ran out of disk in /usr/local about a week ago.
I've increased the partition size with lvm and there is now plenty of disk, but Nagios is unhappy. It's still monitoring and sending notifications, but I cannot do any provisioning changes at all. The XI system component status shows that Database Maintenance, Command Subsystem, Event manager, Feed Processor, Report Engine, Cleaner, Nonstop Operations Manager, and System Statistics are all problematic. If I attempt any provisioning changes the database update times out.
I've confirmed that there is plenty of free disk in /usr/local now:
/dev/mapper/vg_nagios-lv_usr_local
119G 71G 42G 64% /usr/local
I've run the repair_databases.sh script multiple times and every time it finds a number of indices to fix.
There was a problem running that script - when it called repairmysql.sh $BASEDIR was not set for some reason, so I had to modify repairmysql.sh to get it run:# diff repairmysql.sh repairmysql.sh.dist
25c25
< /usr/local/nagiosxi/scripts/manage_services.sh status mysqld
---
> $BASEDIR/manage_services.sh status mysqld
52c52
< /usr/local/nagiosxi/scripts/manage_services.sh stop mysqld
---
> $BASEDIR/manage_services.sh stop mysqld
54c54
< /usr/local/nagiosxi/scripts/manage_services.sh start mysqld
---
> $BASEDIR/manage_services.sh start mysqld
66c66
< exit 0
---
> exit 0
\ No newline at end of file
I've run the fixperms.sh script.
/usr/local/nagios/var/nagios.log logs these two entries every time nagios runs:
[1426179854] Warning: File '/usr/local/nagios/var/service-perfdata' could not be opened - service performance data will not be written to file!
[1426179854] Warning: File '/usr/local/nagios/var/host-perfdata' could not be opened - host performance data will not be written to file!
Those two files have not been updated since the disk filled up:
[root@nagios var]# ls -l host-perfdata service-perfdata
-rw-r--r-- 1 apache nagios 3219 Mar 5 10:41 host-perfdata
-rw-r--r-- 1 apache nagios 7106 Mar 5 10:41 service-perfdata
I'm not sure what to do next, but this is becoming critical - we have some new monitoring objects that we have to add ASAP to address a network problem.
Please advise.
Thanks!
Re: Filesystem filled up, now Nagios XI is not fully functio
Posted: Thu Mar 12, 2015 1:10 pm
by lmiltchev
Let's try to stop/start services and check some of the logs. Run the following commands and show us the output in code wraps:
Code: Select all
df -h
df -i
service nagios stop
killall nagios
service ndo2db stop
killall ndo2db
service ndo2db start
service nagios start
service mysqld restart
service crond restart
service nagios status
service ndo2db status
service mysqld status
service crond status
tail -20 /var/log/mysqld.log
tail -20 /usr/local/nagios/var/nagios.log
ps -ef | grep cron
Re: Filesystem filled up, now Nagios XI is not fully functio
Posted: Thu Mar 12, 2015 1:37 pm
by sgd
Hello, I've run the commands you requested. It does look like there's some database corruption, but I don't think that should be fatal, and as soon as I"m able to make provisioning changes again I can fix those minor problems. I show the component status unchanged from my previous post after restarting everything.
I'm attaching a transcript of my command session:
NagiosXI.debug.session.txt
Thanks!
Re: Filesystem filled up, now Nagios XI is not fully functio
Posted: Thu Mar 12, 2015 1:58 pm
by jdalrymple
You may need to perform a more aggressive repair:
http://assets.nagios.com/downloads/nagi ... tabase.pdf
Ignore the bits at the bottom regarding truncating tables. If you have already completed all that is mentioned here besides the truncating tables bit let us know.
Re: Filesystem filled up, now Nagios XI is not fully functio
Posted: Thu Mar 12, 2015 3:04 pm
by sgd
Hi, thanks for you reply.
As noted in my original post, I've run the repairmysql.sh script several times.
I stopped Nagios, ran it again, and restarted Nagios just now though, and that does not have any affect on the problem.
It seems to have start up just fine:
Code: Select all
150312 12:57:53 [Note] /usr/libexec/mysqld: Normal shutdown
150312 12:57:53 [Note] Event Scheduler: Purging the queue. 0 events
150312 12:57:55 InnoDB: Starting shutdown...
150312 12:57:58 InnoDB: Shutdown completed; log sequence number 0 44233
150312 12:57:58 [Note] /usr/libexec/mysqld: Shutdown complete
150312 12:57:58 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150312 12:58:32 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150312 12:58:32 InnoDB: Started; log sequence number 0 44233
150312 12:58:32 [Note] Event Scheduler: Loaded 0 events
150312 12:58:32 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.52' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
What's the next step?
Thanks.
Re: Filesystem filled up, now Nagios XI is not fully functio
Posted: Thu Mar 12, 2015 3:08 pm
by jdalrymple
Did you run myisamchk on any inconsistent tables?
If you receive an error, similar to this one:
Code: Select all
SQL: DELETE FROM nagios_logentries WHERE logentry_time < FROM_UNIXTIME(1293570334)
SQL: SQL Error [ndoutils] :</b> Table './nagios/nagios_logentries' is marked as crashed
and last (automatic?) repair failedCLEANING ndoutils TABLE 'notifications'...
you may need to run a force repair on the tables:
Code: Select all
service mysqld stop
cd /var/lib/mysql/nagios
myisamchk -r -f nagios_<corrupted_table>
service mysqld start
Re: Filesystem filled up, now Nagios XI is not fully functio
Posted: Thu Mar 12, 2015 3:21 pm
by sgd
I've done the force repair on all tables in the database, with no change in Nagios behavior - the component status has not changed and I still cannot make any provisioning changes.
Re: Filesystem filled up, now Nagios XI is not fully functio
Posted: Thu Mar 12, 2015 3:37 pm
by lmiltchev
Your cron is NOT running... When you run:
you should see something like this:
Code: Select all
nagios 13170 13167 0 15:35 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1
nagios 13172 13163 0 15:35 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1
nagios 13176 13166 0 15:35 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1
nagios 13179 13164 0 15:35 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1
nagios 13182 13165 0 15:35 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1
nagios 13184 13170 0 15:35 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php
nagios 13187 13172 19 15:35 ? 00:00:04 /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php
nagios 13188 13176 0 15:35 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php
nagios 13191 13182 0 15:35 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php
nagios 13192 13179 0 15:35 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php
Run the following commands and show us the output in code wraps:
Code: Select all
tail -100 /var/log/cron
chage -l nagios
chage -l apache
grep nag /etc/group
Re: Filesystem filled up, now Nagios XI is not fully functio
Posted: Thu Mar 12, 2015 3:59 pm
by sgd
Crond is running, for sure.
Here's the result of the commands you requested:
Code: Select all
Script started on Thu 12 Mar 2015 02:00:02 PM PDT
[root@nagios sgd]# tail -100 /var/log/cron
Mar 12 13:50:01 nagios CROND[24358]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1)
Mar 12 13:50:01 nagios CROND[24362]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1)
Mar 12 13:50:01 nagios CROND[24366]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1)
Mar 12 13:50:01 nagios CROND[24368]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1)
Mar 12 13:50:01 nagios CROND[24363]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok)
Mar 12 13:50:01 nagios CROND[24370]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1)
Mar 12 13:50:01 nagios CROND[24369]: (cacti) CMD (/usr/bin/php /usr/share/cacti/poller.php > /dev/null 2>&1)
Mar 12 13:50:01 nagios CROND[24361]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/deadpool.php > /usr/local/nagiosxi/var/deadpool.log 2>&1)
Mar 12 13:50:01 nagios CROND[24372]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Mar 12 13:51:01 nagios CROND[27418]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Mar 12 13:51:01 nagios CROND[27419]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1)
Mar 12 13:51:01 nagios CROND[27420]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1)
Mar 12 13:51:01 nagios CROND[27421]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1)
Mar 12 13:51:01 nagios CROND[27424]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1)
Mar 12 13:51:01 nagios CROND[27422]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1)
Mar 12 13:51:01 nagios CROND[27423]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1)
Mar 12 13:51:01 nagios CROND[27425]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Mar 12 13:51:37 nagios crontab[29196]: (root) LIST (nagios)
Mar 12 13:52:01 nagios CROND[30464]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1)
Mar 12 13:52:01 nagios CROND[30468]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1)
Mar 12 13:52:01 nagios CROND[30467]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1)
Mar 12 13:52:01 nagios CROND[30466]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1)
Mar 12 13:52:01 nagios CROND[30469]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Mar 12 13:52:01 nagios CROND[30470]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1)
Mar 12 13:52:01 nagios CROND[30471]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1)
Mar 12 13:52:01 nagios CROND[30473]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Mar 12 13:53:01 nagios CROND[1107]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1)
Mar 12 13:53:01 nagios CROND[1108]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1)
Mar 12 13:53:01 nagios CROND[1114]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1)
Mar 12 13:53:01 nagios CROND[1113]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1)
Mar 12 13:53:01 nagios CROND[1112]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1)
Mar 12 13:53:01 nagios CROND[1116]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Mar 12 13:53:01 nagios CROND[1115]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1)
Mar 12 13:53:01 nagios CROND[1118]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Mar 12 13:54:01 nagios CROND[4207]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1)
Mar 12 13:54:01 nagios CROND[4208]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1)
Mar 12 13:54:01 nagios CROND[4209]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1)
Mar 12 13:54:01 nagios CROND[4213]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1)
Mar 12 13:54:02 nagios CROND[4216]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Mar 12 13:54:02 nagios CROND[4212]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1)
Mar 12 13:54:02 nagios CROND[4217]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1)
Mar 12 13:54:02 nagios CROND[4218]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Mar 12 13:55:01 nagios CROND[7222]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1)
Mar 12 13:55:01 nagios CROND[7223]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/deadpool.php > /usr/local/nagiosxi/var/deadpool.log 2>&1)
Mar 12 13:55:01 nagios CROND[7224]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1)
Mar 12 13:55:01 nagios CROND[7230]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1)
Mar 12 13:55:01 nagios CROND[7226]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1)
Mar 12 13:55:01 nagios CROND[7228]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok)
Mar 12 13:55:01 nagios CROND[7229]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Mar 12 13:55:01 nagios CROND[7233]: (cacti) CMD (/usr/bin/php /usr/share/cacti/poller.php > /dev/null 2>&1)
Mar 12 13:55:01 nagios CROND[7231]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1)
Mar 12 13:55:01 nagios CROND[7234]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Mar 12 13:55:01 nagios CROND[7227]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1)
Mar 12 13:55:01 nagios CROND[7232]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1)
Mar 12 13:56:01 nagios CROND[10256]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1)
Mar 12 13:56:01 nagios CROND[10255]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1)
Mar 12 13:56:01 nagios CROND[10257]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1)
Mar 12 13:56:01 nagios CROND[10258]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1)
Mar 12 13:56:01 nagios CROND[10259]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1)
Mar 12 13:56:01 nagios CROND[10261]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1)
Mar 12 13:56:01 nagios CROND[10254]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Mar 12 13:56:01 nagios CROND[10262]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Mar 12 13:57:01 nagios CROND[13271]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Mar 12 13:57:01 nagios CROND[13270]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1)
Mar 12 13:57:01 nagios CROND[13272]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1)
Mar 12 13:57:01 nagios CROND[13273]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1)
Mar 12 13:57:01 nagios CROND[13274]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1)
Mar 12 13:57:01 nagios CROND[13275]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Mar 12 13:57:01 nagios CROND[13278]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1)
Mar 12 13:57:01 nagios CROND[13277]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1)
Mar 12 13:58:01 nagios CROND[16225]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1)
Mar 12 13:58:01 nagios CROND[16224]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1)
Mar 12 13:58:01 nagios CROND[16226]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1)
Mar 12 13:58:01 nagios CROND[16230]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Mar 12 13:58:01 nagios CROND[16228]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Mar 12 13:58:01 nagios CROND[16227]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1)
Mar 12 13:58:01 nagios CROND[16231]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1)
Mar 12 13:58:01 nagios CROND[16229]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1)
Mar 12 13:58:42 nagios crontab[18366]: (root) LIST (nagios)
Mar 12 13:59:01 nagios CROND[19263]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1)
Mar 12 13:59:01 nagios CROND[19264]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1)
Mar 12 13:59:01 nagios CROND[19265]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1)
Mar 12 13:59:01 nagios CROND[19266]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1)
Mar 12 13:59:01 nagios CROND[19267]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1)
Mar 12 13:59:01 nagios CROND[19268]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Mar 12 13:59:01 nagios CROND[19270]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1)
Mar 12 13:59:01 nagios CROND[19269]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Mar 12 14:00:01 nagios CROND[22316]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Mar 12 14:00:01 nagios CROND[22317]: (root) CMD (/usr/lib/sa/sa1 -S DISK 1 1)
Mar 12 14:00:01 nagios CROND[22318]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1)
Mar 12 14:00:01 nagios CROND[22319]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1)
Mar 12 14:00:01 nagios CROND[22320]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1)
Mar 12 14:00:01 nagios CROND[22321]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1)
Mar 12 14:00:01 nagios CROND[22323]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Mar 12 14:00:01 nagios CROND[22326]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok)
Mar 12 14:00:01 nagios CROND[22322]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1)
Mar 12 14:00:01 nagios CROND[22324]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1)
Mar 12 14:00:01 nagios CROND[22325]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/deadpool.php > /usr/local/nagiosxi/var/deadpool.log 2>&1)
Mar 12 14:00:01 nagios CROND[22327]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1)
Mar 12 14:00:01 nagios CROND[22328]: (cacti) CMD (/usr/bin/php /usr/share/cacti/poller.php > /dev/null 2>&1)
[root@nagios sgd]# chage -l nagios
Last password change : Sep 23, 2011
Password expires : never
Password inactive : never
Account expires : never
Minimum number of days between password change : 0
Maximum number of days between password change : 99999
Number of days of warning before password expires : 7
[root@nagios sgd]# chage -l apache
Last password change : Sep 22, 2011
Password expires : never
Password inactive : never
Account expires : never
Minimum number of days between password change : -1
Maximum number of days between password change : -1
Number of days of warning before password expires : -1
[root@nagios sgd]# grep nag /etc/group
nagios:x:501:nagios,apache
nagcmd:x:502:nagios,apache
[root@nagios sgd]# exit
exit
Thanks.
Re: Filesystem filled up, now Nagios XI is not fully functio
Posted: Thu Mar 12, 2015 4:36 pm
by lmiltchev
OK, I didn't see these cron entries in the "NagiosXI.debug.session.txt" that you initially posted.
Do you still have errors in the mysqld.log?
Run the following commands and show us the output:
Code: Select all
cd /usr/local/nagiosxi/scripts
./reconfigure_nagios.sh