Your request was not processed in a timely manner... 2018

[BenCowan]

I know this is probably not a new topic, but I've been spinning my wheels looking at posts from 2013, and haven't found anything that has helped thus far. I seem to have broken my NagiosXI web interface. I get "Your request was not processed in a timely manner. It may still execute, as the server may be temporarily busy." However, the requests never complete. An example would be forcing a check, or disabling notifications, etc...

I am running XI 5.4.12, on CentOS 7.4.

Luckily, Nagios core works okay. I was playing with Puppet and I know at one point, I may have screwed up permissions and my sudoers config, but I have run several scripts from /usr/local/nagiosxi/scripts and did my best to fix the permissions and restore the nagios sudo settings, but I don't even know for sure if that's what the problem was. Getting desperate...

[npolovenko]

Hello, @BenCowan. Here's my sudoers file so you can compare, expesially please verify the bottom part with all Nagios entries:

Code: Select all

cat /etc/sudoers
## Sudoers allows particular users to run various commands as
## the root user, without needing the root password.
##
## Examples are provided at the bottom of the file for collections
## of related commands, which can then be delegated out to particular
## users or groups.
##
## This file must be edited with the 'visudo' command.

## Host Aliases
## Groups of machines. You may prefer to use hostnames (perhaps using
## wildcards for entire domains) or IP addresses instead.
# Host_Alias     FILESERVERS = fs1, fs2
# Host_Alias     MAILSERVERS = smtp, smtp2

## User Aliases
## These aren't often necessary, as you can use regular groups
## (ie, from files, LDAP, NIS, etc) in this file - just use %groupname
## rather than USERALIAS
# User_Alias ADMINS = jsmith, mikem


## Command Aliases
## These are groups of related commands...

## Networking
# Cmnd_Alias NETWORKING = /sbin/route, /sbin/ifconfig, /bin/ping, /sbin/dhclient, /usr/bin/net, /sbin/iptables, /usr/bin/rfcomm, /usr/bin/wvdial, /sbin/iwconfig, /sbin/mii-tool

## Installation and management of software
# Cmnd_Alias SOFTWARE = /bin/rpm, /usr/bin/up2date, /usr/bin/yum

## Services
# Cmnd_Alias SERVICES = /sbin/service, /sbin/chkconfig, /usr/bin/systemctl start, /usr/bin/systemctl stop, /usr/bin/systemctl reload, /usr/bin/systemctl restart, /usr/bin/systemctl status, /usr/bin/systemctl enable, /usr/bin/systemctl disable

## Updating the locate database
# Cmnd_Alias LOCATE = /usr/bin/updatedb

## Storage
# Cmnd_Alias STORAGE = /sbin/fdisk, /sbin/sfdisk, /sbin/parted, /sbin/partprobe, /bin/mount, /bin/umount

## Delegating permissions
# Cmnd_Alias DELEGATING = /usr/sbin/visudo, /bin/chown, /bin/chmod, /bin/chgrp

## Processes
# Cmnd_Alias PROCESSES = /bin/nice, /bin/kill, /usr/bin/kill, /usr/bin/killall

## Drivers
# Cmnd_Alias DRIVERS = /sbin/modprobe

# Defaults specification

#
# Refuse to run if unable to disable echo on the tty.
#
Defaults   !visiblepw

#
# Preserving HOME has security implications since many programs
# use it when searching for configuration files. Note that HOME
# is already set when the the env_reset option is enabled, so
# this option is only effective for configurations where either
# env_reset is disabled or HOME is present in the env_keep list.
#
Defaults    always_set_home

Defaults    env_reset
Defaults    env_keep =  "COLORS DISPLAY HOSTNAME HISTSIZE KDEDIR LS_COLORS"
Defaults    env_keep += "MAIL PS1 PS2 QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE"
Defaults    env_keep += "LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES"
Defaults    env_keep += "LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE"
Defaults    env_keep += "LC_TIME LC_ALL LANGUAGE LINGUAS _XKB_CHARSET XAUTHORITY"

#
# Adding HOME to env_keep may enable a user to run unrestricted
# commands via sudo.
#
# Defaults   env_keep += "HOME"

Defaults    secure_path = /sbin:/bin:/usr/sbin:/usr/bin

## Next comes the main part: which users can run what software on
## which machines (the sudoers file can be shared between multiple
## systems).
## Syntax:
##
##      user    MACHINE=COMMANDS
##
## The COMMANDS section may have other options added to it.
##
## Allow root to run any commands anywhere
root    ALL=(ALL)       ALL

## Allows members of the 'sys' group to run networking, software,
## service management apps and more.
# %sys ALL = NETWORKING, SOFTWARE, SERVICES, STORAGE, DELEGATING, PROCESSES, LOCATE, DRIVERS

## Allows people in group wheel to run all commands
%wheel  ALL=(ALL)       ALL

## Same thing without a password
# %wheel        ALL=(ALL)       NOPASSWD: ALL

## Allows members of the users group to mount and unmount the
## cdrom as root
# %users  ALL=/sbin/mount /mnt/cdrom, /sbin/umount /mnt/cdrom

## Allows members of the users group to shutdown this system
# %users  localhost=/sbin/shutdown -h now

## Read drop-in files from /etc/sudoers.d (the # here does not mean a comment)
#includedir /etc/sudoers.d
User_Alias      NAGIOSXI=nagios
User_Alias              NAGIOSXIWEB=apache
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios start
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios stop
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios restart
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios reload
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios status
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios checkconfig
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db start
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db stop
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db restart
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db reload
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db status
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd start
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd stop
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd restart
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd reload
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd status
NAGIOSXI ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/html/includes/components/autodiscovery/scripts/autodiscover_new.php *
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/html/includes/components/profile/getprofile.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/upgrade_to_latest.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/change_timezone.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/manage_services.sh *
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/reset_config_perms.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/backup_xi.sh *
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/messages
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/httpd/error_log
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/mysqld.log
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/html/includes/components/autodiscovery/scripts/autodiscover_new.php *
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/html/includes/components/profile/getprofile.sh
NAGIOSXIWEB ALL = NOPASSWD:/etc/init.d/snmptt restart
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/scripts/repair_databases.sh
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/scripts/manage_services.sh *

Have you upgraded the XI recently, and if yes from what version?

Please run the following commands in order to eliminate ndo2db issues:

Code: Select all

service nagios stop
service ndo2db stop
mv /usr/local/nagios/var/ndo2db.lock /usr/local/nagios/var/ndo2db.lock.bak
mv /usr/local/nagios/var/ndo.sock /usr/local/nagios/var/ndo.sock.bak
service ndo2db start
service nagios start

If you still have problems after running the above commands please send in your system profile.
To send us your system profile. Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file, upload it to a cloud storage of your choice and share a link with me via personal message.
After you send it please post something in this thread to bring it back up in the support queue.

[BenCowan]

My sudoers file looks right. No luck with bouncing ndo2db and nagios. I just sent you a link via private message. Thanks.

[npolovenko]

@BenCowan, Please run the following script as a sudo user and let me know if you get any errors during the process.

Code: Select all

/usr/local/nagiosxi/scripts/repair_databases.sh

Also, please remove the /usr/local/nagiosxi/var/event_handler.lock file, and then restart Nagios from the command line using:

Code: Select all

service nagios restart

Profile was received and shared with the support team

[BenCowan]

Did not see any obvious errors. Same results after restart.

[npolovenko]

@BenCowan, What is the output of the follwoing command:

Code: Select all

ls -l /usr/local/nagios/var/service-perfdata

[BenCowan]

[root@warpath bcowan]# ls -l /usr/local/nagios/var/service-perfdata
-rw-r--r--. 1 apache nagios 13611 Mar 6 14:13 /usr/local/nagios/var/service-perfdata

[lmiltchev]

In some larger deployments of nagios, starting/stopping/restarting the nagios service may result in timeouts. See how you can resolve the issue here:

https://support.nagios.com/kb/article.php?id=172

Similar issues can be caused by crond not running (or not having permissions to run as nagios user). Also, we've seen cases when nagios or apache user has expired. To rule out these causes, and to see if apache is added to the nagios group, run the commands below, and show the output in code wraps:

Code: Select all

service crond stop
killall crond
service crond start
service crond status
tail -n 100 /var/log/cron
chage -l nagios
chage -l apache
grep nag /etc/group

In addition to this, we will need to check a few more things. We found the following error in your profile:

[Wed Mar 07 20:17:08.427626 2018] [:error] [pid 31174] [client 127.0.0.1:54216] PHP Warning: pg_pconnect(): Unable to connect to PostgreSQL server: could not connect to server: Connection refused\n\tIs the server running on host "localhost" (::1) and accepting\n\tTCP/IP connections on port 5432?\ncould not connect to server: Connection refused\n\tIs the server running on host "localhost" (127.0.0.1) and accepting\n\tTCP/IP connections on port 5432? in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-postgres64.inc.php on line 699

Is postgresql running? Can you view the tables?

Run the commands below, and show the output:

Code: Select all

service postgresql restart
service postgresql status
echo '\d' | psql nagiosxi nagiosxi

Note: It is possible that puppet changed some of the permissions, which is also causing some issues:

[root@warpath bcowan]# ls -l /usr/local/nagios/var/service-perfdata /usr/local/nagios/var/host-perfdata
-rw-r--r--. 1 apache nagios 13611 Mar 6 14:13 /usr/local/nagios/var/service-perfdata

Let's change the permissions of host-perfdata and service-perdata to nagios:

Code: Select all

chown nagios.nagios /usr/local/nagios/var/host-perfdata /usr/local/nagios/var/service-perfdata

tail: cannot open '/usr/local/nagios/var/nagios.log\r' for reading: No such file or directory

What are the permissions of the "nagios.log", and "nagiosql" scripts?

Code: Select all

ls -l /usr/local/nagios/var/nagios.log
ls -l /usr/local/nagiosxi/scripts/nagiosql*

[BenCowan]

Changed perms per instructions. No change. Here are the listings requested:

# ls -l /usr/local/nagios/var/nagios.log
-rw-r--r-- 1 nagios nagios 852018 Mar 9 09:20 /usr/local/nagios/var/nagios.log
# ls -l /usr/local/nagiosxi/scripts/nagiosql*
-rwxrwxr-x. 1 nagios nagios 161 Mar 8 12:57 /usr/local/nagiosxi/scripts/nagiosql.cookies
-rwxrwxr-x. 1 nagios nagios 283701 Jan 17 11:55 /usr/local/nagiosxi/scripts/nagiosql_defaults.sql
-rwxrwxr-x. 1 nagios nagios 895 Jan 17 11:55 /usr/local/nagiosxi/scripts/nagiosql_delete_contact.php
-rwxrwxr-x. 1 nagios nagios 40685 Dec 12 2014 /usr/local/nagiosxi/scripts/nagiosql.delete.host
-rwxrwxr-x. 1 nagios nagios 1937 Jan 17 11:55 /usr/local/nagiosxi/scripts/nagiosql_delete_host.php
-rwxrwxr-x. 1 nagios nagios 497 Jan 17 11:55 /usr/local/nagiosxi/scripts/nagiosql_delete_object.sh
-rwxrwxr-x. 1 nagios nagios 43756 May 28 2014 /usr/local/nagiosxi/scripts/nagiosql.delete.service
-rwxrwxr-x. 1 nagios nagios 2766 Jan 17 11:55 /usr/local/nagiosxi/scripts/nagiosql_delete_service.php
-rwxrwxr-x. 1 nagios nagios 34230 Jan 29 2016 /usr/local/nagiosxi/scripts/nagiosql.delete.timeperiod
-rwxrwxr-x. 1 nagios nagios 896 Jan 17 11:55 /usr/local/nagiosxi/scripts/nagiosql_delete_timeperiod.php
-rwxrwxr-x. 1 nagios nagios 7400 Jun 9 2014 /usr/local/nagiosxi/scripts/nagiosql.export.additional
-rwxrwxr-x. 1 nagios nagios 1272 Jan 17 11:55 /usr/local/nagiosxi/scripts/nagiosql_exportall.php
-rwxrwxr-x. 1 nagios nagios 21486 Mar 8 12:57 /usr/local/nagiosxi/scripts/nagiosql.export.monitoring
-rwxrwxr-x. 1 nagios nagios 1296 Jan 17 11:55 /usr/local/nagiosxi/scripts/nagiosql_importall.php
-rwxrwxr-x. 1 nagios nagios 77290 Jan 17 11:55 /usr/local/nagiosxi/scripts/nagiosql.import.monitoring
-rwxrwxr-x. 1 nagios nagios 39455 Mar 8 12:57 /usr/local/nagiosxi/scripts/nagiosql.login
-rwxrwxr-x. 1 nagios nagios 1681 Jan 17 11:55 /usr/local/nagiosxi/scripts/nagiosql_login.php
-rwxrwxr-x. 1 nagios nagios 2247 Jan 17 11:55 /usr/local/nagiosxi/scripts/nagiosql_snapshot.sh
-rwxrwxr-x. 1 nagios nagios 200 Jan 17 11:55 /usr/local/nagiosxi/scripts/nagiosql_trim_backups.sh

[scottwilkerson]

What was the output of this?

Code: Select all

tail -n 100 /var/log/cron
chage -l nagios
chage -l apache
grep nag /etc/group

and this?

Is postgresql running? Can you view the tables?

Run the commands below, and show the output:

Code: Select all

service postgresql restart
service postgresql status
echo '\d' | psql nagiosxi nagiosxi

Also with your reply send your /etc/php.ini I believe you may have a sourceguardian line in there that should be commented out