Request was not processed in a timely manner

[arnab.roy]

Guys,

We are having the exact same issue, I wish didnt do the upgrade now! something is seriously broken. I have the output you last requested from our system

KARMA[nagiosxi]> chage -l nagios
Last password change : Jun 29, 2011
Password expires : never
Password inactive : never
Account expires : never
Minimum number of days between password change : 0
Maximum number of days between password change : 99999
Number of days of warning before password expires : 7
KARMA[nagiosxi]> cat /etc/group | grep nag
nagios500:nagios,apache
nagcmd501:nagios,apache
KARMA[nagiosxi]> service nagios status
nagios (pid 19957) is running...
KARMA[nagiosxi]> service ndo2db status
ndo2db (pid 4603) is running...
KARMA[nagiosxi]> service mysqld status
mysqld (pid 4225) is running...
KARMA[nagiosxi]> service postgresql status
postmaster (pid 23172 22958 22951 22945 22935 16732 9092 8973 7257 7243 7238 7233 7226 7222 7217 7195 7190 4392 4391 4390 4387 4321) is running...
KARMA[nagiosxi]> service crond status
crond (pid 4406) is running...
KARMA[nagiosxi]>

Rebooted re-upgraded what not! Nothing helps.

[slansing]

Can you two please share the output of the following immediately after running a command from the UI that produces this error:

Code: Select all

tail -20 /var/log/httpd/error_log

Can you also run the vacuum on postgres by pasting the following into the CLI?:

Code: Select all

 psql nagiosxi nagiosxi
 VACUUM;
 VACUUM ANALYZE;
 VACUUM FULL;
 \q

Please share the output at the end.

[arnab.roy]

Their is nothing getting logged when I do running tail on the apache error log . The interesting thing is it is actually doing the action your asking it to do i.e. say disable notification .

[lmiltchev]

@arnab.roy

Are you using SSL with this particular Nagios XI instance?

[sreinhardt]

Yes SSL with self signed cert.

[slansing]

Can you let us know which green check marks you have? You can mouse over them to reveal what they are. Thanks! We are working on reproducing this on our end.

Also, can you run and report the output of the following:

Code: Select all

tail -20 /var/log/mysqld.log

Code: Select all

service ndo2db restart

[sreinhardt]

tail -20 /var/log/mysqld.log
130527 0:36:25 [Note] /usr/libexec/mysqld: Shutdown complete

130527 00:36:25 mysqld ended

130527 00:39:37 mysqld started
130527 0:39:39 InnoDB: Started; log sequence number 0 43675
130527 0:39:40 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.95' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
130529 18:07:23 [Note] /usr/libexec/mysqld: Normal shutdown

130529 18:07:23 InnoDB: Starting shutdown...
130529 18:07:24 InnoDB: Shutdown completed; log sequence number 0 43675
130529 18:07:24 [Note] /usr/libexec/mysqld: Shutdown complete

130529 18:07:24 mysqld ended

130529 18:09:41 mysqld started
130529 18:09:42 InnoDB: Started; log sequence number 0 43675
130529 18:09:43 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.95' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution

ndo2db restarts just fine. The checks are being processed just fine , thank god for that.

The enabled ticks are :

Monitoring Engine Running

Performance Grapher is running

Database Backend is running

So it looks like , we cannot read anything of Nagios Core on the front end, i.e. Host checks , notification status etc. This again ties in with the fact that we cant see the status of Nagios core like process id etc.

Hope that helps.

Many Thanks
Arnab

EDIT: Just to add to that the generic symptom seems to be broken ajax calls. As the Quick Find searches are no longer bringing up the suggested list. This is making increasingly towards a permission issue somewhere.

[sreinhardt]

Can you run the following and return results please. We are narrowing this down on our end with a bunch of internal testing but want to verify that you are seeing similar things.

Code: Select all

tail -20 /var/log/cron
cat /etc/cron.d/nagiosxi

[arnab.roy]

tail -20 /var/log/cron

Code: Select all

May 29 22:35:01 karma crond[31275]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok)
May 29 22:35:01 karma crond[31277]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/deadpool.php > /usr/local/nagiosxi/var/deadpool.log 2>&1)
May 29 22:35:01 karma crond[31280]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1)
May 29 22:35:01 karma crond[31282]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1)
May 29 22:35:01 karma crond[31290]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1)
May 29 22:35:01 karma crond[31293]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1)
May 29 22:35:01 karma crond[31305]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
May 29 22:35:01 karma crond[31308]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
May 29 22:35:01 karma crond[31317]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1)
May 29 22:35:01 karma crond[31328]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1)
May 29 22:35:01 karma crond[31331]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1)
May 29 22:36:01 karma crond[32152]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1)
May 29 22:36:01 karma crond[32154]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1)
May 29 22:36:01 karma crond[32158]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1)
May 29 22:36:01 karma crond[32159]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1)
May 29 22:36:01 karma crond[32164]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
May 29 22:36:01 karma crond[32165]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1)
May 29 22:36:01 karma crond[32163]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
May 29 22:36:01 karma crond[32176]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1)
You have new mail in /var/spool/mail/root

KARMA[scripts]> cat /etc/cron.d/nagiosxi
# /etc/cron.d/nagiosxi: crontab fragment for nagiosxi

Code: Select all

# Backup MySQL & PostgreSQL Databases
0   7 * * * root   /root/scripts/automysqlbackup
0   8 * * * root   /root/scripts/autopostgresqlbackup

*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1
*/5 * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
01  * * * * nagios /usr/local/nagiosxi/cron/recurringdowntime.pl > /usr/local/nagiosxi/var/recurringdowntime.log 2>&1
*/5 * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/deadpool.php > /usr/local/nagiosxi/var/deadpool.log 2>&1

As requested

[scottwilkerson]

and one more

Code: Select all

ps -ef |grep nagiosxi/cron/