Nagios XI Configuration Random Configuration Error

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
JakeHatMacys
Posts: 281
Joined: Thu Sep 25, 2014 3:21 pm

Re: Nagios XI Configuration Random Configuration Error

Post by JakeHatMacys »

jdalrymple wrote:
JakeHatMacys wrote:Especially now that I did a restore to a clean config from 4/6 and it's still in the bad state.
What is a bad state? Is the monitoring engine still offline? When you say you restored from 4/6, did you do a full restore, or just reverted to a config snapshot? It sounds like full restore - which should negate any possible database issues.

Can you post the output of:

Code: Select all

/usr/local/nagiosxi/scripts/reconfigure_nagios.sh
Yeah the monitoring engine is still down, and I ran the restore script on a back up from 4/6:

/usr/local/nagiosxi/scripts/restore_xi.sh

And basically still no change.

Contenst of /usr/local/nagiosxi/scripts/reconfigure_nagios.sh:

Code: Select all

#!/bin/sh
# Copyright (c) 2008-2014 Nagios Enterprises, LLC.  All rights reserved.
# $Id: reconfigure_nagios.sh 2854 2014-08-23 18:29:04Z swilkerson $

# exit codes:
#       1       config verification failed
#       2       nagiosql login failed
#       3       nagiosql import failed
#       4       reset_config_perms failed
#       5       nagiosql_exportall.php failed (write configs failed)
#       6       /etc/init.d/nagios restart failed
#       7       db_connect failed
#


# Import data to NagiosQL
./import_nagiosql.sh
ret=$?
if [ $ret -gt 0 ]; then
        exit $ret
fi

# Restart Nagios
./restart_nagios_with_export.sh

ret=$?
if [ $ret -gt 0 ]; then
        exit $ret
fi

exit 0
Also now when I run the Write Config gauntlet from top to bottom everything is fine but I get an error on the restart...
Capture.JPG
You do not have the required permissions to view the files attached to this post.
JakeHatMacys
Posts: 281
Joined: Thu Sep 25, 2014 3:21 pm

Re: Nagios XI Configuration Random Configuration Error

Post by JakeHatMacys »

Been digging around some of the logs to find more information to help.

Contents of /usr/local/nagiosxi/var/cmdsubsys.log

Code: Select all

more /usr/local/nagiosxi/var/cmdsubsys.log
PROCESSING COMMAND ID 383...
PROCESS COMMAND: CMD=11, DATA=a:1:{i:0;s:0:"";}
CMDLINE=/usr/local/nagiosxi/scripts/manage_services.sh start nagios
/usr/local/nagiosxi/scripts/manage_services.sh: fork: retry: Resource temporarily unavailable
Starting nagios:
Nagios Core 4.0.8
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 08-12-2014
License: GPL

Website: http://www.nagios.org
Reading configuration data...
   Read main config file okay...
   Read object config files okay...

Running pre-flight check on configuration data...

Checking objects...
Error: There are no services defined!
        Checked 0 services.
Error: There are no hosts defined!
        Checked 0 hosts.
        Checked 5 host groups.
        Checked 0 service groups.
        Checked 10 contacts.
        Checked 2 contact groups.
        Checked 122 commands.
        Checked 16 time periods.
        Checked 0 host escalations.
        Checked 0 service escalations.
Checking for circular paths...
        Checked 0 hosts
        Checked 0 service dependencies
        Checked 0 host dependencies
        Checked 16 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 0
Total Errors:   2

***> One or more problems was encountered while running the pre-flight check...

     Check your configuration file(s) to ensure that they contain valid
     directives and data defintions.  If you are upgrading from a previous
     version of Nagios, you should be aware that some variables/definitions
     may have been removed or modified in this version.  Make sure to read
     the HTML documentation regarding the config files, as well as the
     'Whats New' section to find out what has changed.

OUTPUT=
RETURNCODE=8
PROCESSING COMMAND ID 384...
PROCESS COMMAND: CMD=17, DATA=
APPLYING NAGIOSCORE CONFIG...
CMDLINE=cd /usr/local/nagiosxi/scripts && ./reconfigure_nagios.sh
URL: http://localhost/nagiosxi/includes/components/ccm/
CMDLINE
--2015-04-08 16:11:04--  http://localhost/nagiosxi/includes/components/ccm/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `nagiosql.login'

     0K ..........                                              731K=0.01s

2015-04-08 16:11:04 (731 KB/s) - `nagiosql.login' saved [10831]

/usr/bin/wget --save-cookies nagiosql.cookies --keep-session-cookies http://localhost/nagiosxi/includes/components/ccm/
--no-check-certificate --post-data 'submit=Login&hidelog=true&loginSubmitted=true&username=nagiosxi&password=n@gweb' -O
nagiosql.loginLOGIN SUCCESSFUL!
IMPORTING CONFIG FILES...URL: http://localhost/nagiosxi/includes/components/ccm/
Array
(
)
RESETTING PERMS
SETUID ROOT OK
URL: http://localhost/nagiosxi/includes/components/ccm/
CMDLINE
--2015-04-08 16:11:05--  http://localhost/nagiosxi/includes/components/ccm/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `nagiosql.login'

     0K ..........                                              744K=0.01s

2015-04-08 16:11:05 (744 KB/s) - `nagiosql.login' saved [10831]

/usr/bin/wget --save-cookies nagiosql.cookies --keep-session-cookies http://localhost/nagiosxi/includes/components/ccm/
--no-check-certificate --post-data 'submit=Login&hidelog=true&loginSubmitted=true&username=nagiosxi&password=n@gweb' -O
nagiosql.loginLOGIN SUCCESSFUL!
URL: http://localhost/nagiosxi/includes/components/ccm/
CMDLINE:
/usr/bin/wget --load-cookies=nagiosql.cookies http://localhost/nagiosxi/includes/components/ccm/ --no-check-certificate
--post-data 'cmd=apply&type=writeConfig' -O nagiosql.export.monitoring
--2015-04-08 16:11:06--  http://localhost/nagiosxi/includes/components/ccm/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `nagiosql.export.monitoring'

     0K .......... .......... .......... .......... ..........  151
    50K .......... .......... .......... .......... ..........  134M
   100K .......... .......... .......... .......... ..........  153M
   150K .......... .......... .......... .......... ..........  164M
   200K .......... .......... .......... .......... ..........  129M
   250K .......... .......... .......... .......... ..........  171M
   300K .......... ...                                         1.11M=5m39s

2015-04-08 16:16:44 (948 B/s) - `nagiosql.export.monitoring' saved [320984]

WRITE CONFIGS SUCCESSFUL!
./restart_nagios_with_export.sh: fork: retry: Resource temporarily unavailable
./restart_nagios_with_export.sh: fork: retry: Resource temporarily unavailable
./restart_nagios_with_export.sh: fork: retry: Resource temporarily unavailable
./restart_nagios_with_export.sh: fork: retry: Resource temporarily unavailable
./restart_nagios_with_export.sh: fork: Resource temporarily unavailable
OUTPUT=WRITE CONFIGS SUCCESSFUL!
RETURNCODE=254
cmerchant
Posts: 546
Joined: Wed Sep 24, 2014 11:19 am

Re: Nagios XI Configuration Random Configuration Error

Post by cmerchant »

Could you run the following to see the nagiosql Database count of hosts and services.

Code: Select all

echo 'select count(*) from nagios_hosts;select count(*) from nagios_services;' | mysql -u root -pnagiosxi nagios -t
also could show us the output of the following:

Code: Select all

ps -efw
df -h
df -i
JakeHatMacys
Posts: 281
Joined: Thu Sep 25, 2014 3:21 pm

Re: Nagios XI Configuration Random Configuration Error

Post by JakeHatMacys »

cmerchant wrote:Could you run the following to see the nagiosql Database count of hosts and services.

Code: Select all

echo 'select count(*) from nagios_hosts;select count(*) from nagios_services;' | mysql -u root -pnagiosxi nagios -t
also could show us the output of the following:

Code: Select all

ps -efw
df -h
df -i
Sure

Code: Select all

 echo 'select count(*) from nagios_hosts;select count(*) from nagios_services;' | mysql -u root -pnagiosxi nagios -t
+----------+
| count(*) |
+----------+
|     2048 |
+----------+
+----------+
| count(*) |
+----------+
|     5380 |
+----------+
PS -efw (Everything I could capture after the command)

Code: Select all

nagios   28798 28795  0 Apr08 pts/368  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=310 [email protected] iosta
nagios   28800 28797  0 Apr08 pts/370  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=310 [email protected] iosta
nagios   28816 26260  0 Apr07 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no
nagios   28818 28816  0 Apr07 pts/109  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] df -v
nagios   28819     1  0 Apr08 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.48.7.238
nagios   28820 28819  0 Apr08 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no
nagios   28821 28820  0 Apr08 pts/191  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat
nagios   28830     1  0 Apr08 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.120.38.2
nagios   28831 28830  0 Apr08 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no
nagios   28832 28831  0 Apr08 pts/280  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iosta
nagios   28944     1  0 Apr08 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_FS_space_linux_by_sshpass.sh 11.120.39.146
nagios   28946 28944  0 Apr08 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no
nagios   28947 28946  0 Apr08 pts/265  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=310 [email protected] iosta
nagios   28979     1  0 Apr08 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.120.38.2
nagios   28980 28979  0 Apr08 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no
nagios   28981 28980  0 Apr08 pts/113  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iosta
nagios   29450     1  0  2014 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.48.27.87
nagios   29451 29450  0  2014 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no
nagios   29452 29451  0  2014 pts/14   00:00:00 ssh -q -o StrictHostKeyChecking=no [email protected] iostat -c 5 2 |  tr -s ' ' ';
nagios   29525     1  0 Apr07 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_FS_space_linux_by_sshpass.sh 11.48.27.45
nagios   29526 29525  0 Apr07 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no
nagios   29527 29526  0 Apr07 pts/220  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=310 [email protected] iostat
nagios   29630 26956  0 Mar17 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no
nagios   29632 29630  0 Mar17 pts/43   00:00:00 ssh -q -o StrictHostKeyChecking=no [email protected] df -v
nagios   29690     1  0 Apr07 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.48.45.6
nagios   29691 29690  0 Apr07 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no
nagios   29692 29691  0 Apr07 pts/246  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -
apache   29716 21552  0 Apr08 ?        00:06:33 /usr/sbin/httpd
nagios   29981     1  0 Apr08 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.48.221.7
nagios   29986 29981  0 Apr08 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no
nagios   29987 29986  0 Apr08 pts/273  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat
root     30014     1  0 Apr02 ?        00:00:00 sudo su - root
root     30015  2474  0 Apr02 ?        00:00:04 pmmasterd -ar
nagios   30227     1  0 Apr07 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.48.24.13
nagios   30228 30227  0 Apr07 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no
nagios   30229 30228  0 Apr07 pts/124  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat
root     30298  2539  0 Jan15 ?        00:00:00 sshd: root@pts/29
root     30318 30298  0 Jan15 pts/29   00:00:00 -bash
nagios   30556     1  0 Mar17 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.48.168.2
nagios   30557 30556  0 Mar17 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no
nagios   30558 30557  0 Mar17 pts/59   00:00:00 ssh -q -o StrictHostKeyChecking=no [email protected] iostat -c 5 2 |  tr -s ' '
nagios   30577     1  0 Apr08 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.48.39.19
nagios   30578 30577  0 Apr08 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no
nagios   30579 30578  0 Apr08 pts/148  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat
nagios   30629     1  0 Mar20 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_FS_space_linux_by_sshpass.sh 11.48.115.33
nagios   30636 30629  0 Mar20 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no
nagios   30647 30636  0 Mar20 pts/71   00:00:00 ssh -q -o StrictHostKeyChecking=no [email protected] iostat -c 5 2 |  tr -s ' ' '
nagios   30653     1  0 Apr08 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_FS_space_linux_by_sshpass.sh 11.48.221.70
nagios   30654 30653  0 Apr08 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no
nagios   30655 30654  0 Apr08 pts/293  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=310 [email protected] iostat
nagios   30662     1  0 Apr07 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_FS_space_linux_by_sshpass.sh 11.120.39.146
nagios   30672 30662  0 Apr07 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no
nagios   30676 30672  0 Apr07 pts/217  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=310 [email protected] iosta
nagios   30738     1  0 Apr07 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.48.45.11
nagios   30739 30738  0 Apr07 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no
nagios   30740 30739  0 Apr07 pts/161  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat
nagios   30761     1  0  2014 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.48.27.87
nagios   30766 30761  0  2014 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no
nagios   30768 30766  0  2014 pts/52   00:00:00 ssh -q -o StrictHostKeyChecking=no [email protected] iostat -c 5 2 |  tr -s ' ' ';
nagios   30874     1  0 Apr08 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.48.221.1
nagios   30876 30874  0 Apr08 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no
nagios   30877 30876  0 Apr08 pts/214  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iosta
nagios   30967     1  0 Apr08 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.120.114.
nagios   30968 30967  0 Apr08 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no
nagios   30969 30968  0 Apr08 pts/153  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iosta
nagios   30970     1  0 Apr08 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.48.115.3
nagios   30971 30970  0 Apr08 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no
nagios   30972 30971  0 Apr08 pts/166  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat
postgres 30983 21530  0 Apr08 ?        00:00:07 postgres: nagiosxi nagiosxi [local] idle
nagios   31037     1  0 Apr08 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.48.27.42
nagios   31040 31037  0 Apr08 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no
nagios   31041 31040  0 Apr08 pts/178  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat
root     31195 30014  0 Apr02 ?        00:00:00 sudo su - root
root     31196 31195  0 Apr02 ?        00:00:00 su - root
root     31198 31196  0 Apr02 ?        00:00:00 -bash
root     31210 28576  0 Apr02 ?        00:00:00 sudo su - root
root     31213 31210  0 Apr02 ?        00:00:00 su - root
root     31241 31213  0 Apr02 ?        00:00:00 -bash
nagios   31248     1  0 Apr08 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.48.27.41
nagios   31249 31248  0 Apr08 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no
nagios   31250 31249  0 Apr08 pts/257  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat
nagios   31257     1  0 Apr08 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.48.115.3
nagios   31258 31257  0 Apr08 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no
nagios   31259 31258  0 Apr08 pts/122  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat
root     31334     1  0 Apr02 ?        00:00:00 sudo su - root
root     31335  2474  0 Apr02 ?        00:00:04 pmmasterd -ar
nagios   31341     1  0 Apr07 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.48.21.11
nagios   31342 31341  0 Apr07 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no
nagios   31343 31342  0 Apr07 pts/204  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat
nagios   31485     1  0 Jan07 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_FS_space_linux_by_sshpass.sh 11.120.114.58
nagios   31500 31485  0 Jan07 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no
nagios   31511 31500  0 Jan07 pts/60   00:00:00 ssh -q -o StrictHostKeyChecking=no [email protected] iostat -c 5 2 |  tr -s ' '
nagios   31980     1  0 Jan29 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_FS_space_linux_by_sshpass.sh 11.48.45.116
nagios   31997 31980  0 Jan29 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no
nagios   32007 31997  0 Jan29 pts/91   00:00:00 ssh -q -o StrictHostKeyChecking=no [email protected] iostat -c 5 2 |  tr -s ' ' '
nagios   32035     1  0 Mar17 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_FS_space_linux_by_sshpass.sh 11.122.80.14
nagios   32037 32035  0 Mar17 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no
nagios   32038 32037  0 Mar17 pts/93   00:00:00 ssh -q -o StrictHostKeyChecking=no [email protected] iostat -c 5 2 |  tr -s ' ' '
nagios   32058     1  0 Mar17 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_FS_space_linux_by_sshpass.sh 11.120.114.58
nagios   32059 32058  0 Mar17 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no
nagios   32060 32059  0 Mar17 pts/111  00:00:00 ssh -q -o StrictHostKeyChecking=no [email protected] iostat -c 5 2 |  tr -s ' '
nagios   32097     1  0  2014 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.120.114.
nagios   32104 32097  0  2014 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no
nagios   32107 32104  0  2014 pts/19   00:00:00 ssh -q -o StrictHostKeyChecking=no [email protected] iostat -c 5 2 |  tr -s ' '
nagios   32130     1  0 Apr08 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_FS_space_linux_by_sshpass.sh 11.120.113.102
nagios   32131 32130  0 Apr08 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no
nagios   32132 32131  0 Apr08 pts/425  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=310 [email protected] iost
nagios   32133     1  0 Apr07 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.48.45.6
nagios   32134 32133  0 Apr07 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no
nagios   32135 32134  0 Apr07 pts/205  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -
nagios   32550     1  0 Apr08 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.48.21.12
nagios   32551 32550  0 Apr08 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no
nagios   32552 32551  0 Apr08 pts/382  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat
root     32615     1  0 Mar06 ?        00:00:00 /bin/sh /opt/OV/HPBsmIntNagios/genint/bin/genintd.sh /opt/OV/HPBsmIntNagios/genint/c
nagios   32617     1  0 Apr07 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.48.115.3
root     32619 32615  0 Mar06 ?        04:47:21 /opt/OV/HPBsmIntNagios/genint/bin/genintd /opt/OV/HPBsmIntNagios/genint/conf/nagios_
nagios   32622 32617  0 Apr07 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no
nagios   32624 32622  0 Apr07 pts/145  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat
df -h

Code: Select all

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/localvg00-lv_slash
                       17G  6.3G  9.2G  41% /
tmpfs                  48G  1.0M   48G   1% /dev/shm
/dev/mapper/localvg00-lv_archive
                      4.0G  194M  3.6G   6% /archive
/dev/sda1             248M   38M  198M  17% /boot
/dev/mapper/localvg00-lv_home
                      4.0G  1.4G  2.4G  37% /home
/dev/mapper/localvg00-lv_opt
                       16G  427M   15G   3% /opt
/dev/mapper/localvg00-lv_tmp
                      3.0G  652M  2.2G  23% /tmp
/dev/mapper/localvg00-lv_usr
                      9.9G  2.4G  7.1G  25% /usr
/dev/mapper/localvg00-lv_usr_local
                       56G   13G   41G  24% /usr/local
/dev/mapper/localvg00-lv_var
                      9.9G  4.7G  4.8G  50% /var
df -I:

Code: Select all

Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/localvg00-lv_slash
                     1081344    7362 1073982    1% /
tmpfs                12381499       2 12381497    1% /dev/shm
/dev/mapper/localvg00-lv_archive
                      262144     225  261919    1% /archive
/dev/sda1              65536      39   65497    1% /boot
/dev/mapper/localvg00-lv_home
                      262144    1090  261054    1% /home
/dev/mapper/localvg00-lv_opt
                     1048576    2907 1045669    1% /opt
/dev/mapper/localvg00-lv_tmp
                      196608    7808  188800    4% /tmp
/dev/mapper/localvg00-lv_usr
                      655360   78731  576629   13% /usr
/dev/mapper/localvg00-lv_usr_local
                     3670016   35415 3634601    1% /usr/local
/dev/mapper/localvg00-lv_var
                      655360    5317  650043    1% /var
cmerchant
Posts: 546
Joined: Wed Sep 24, 2014 11:19 am

Re: Nagios XI Configuration Random Configuration Error

Post by cmerchant »

What is the status of the The Monitoring engine? Is everything "green"?

Can you show us the permissions on the following:

Code: Select all

ls -lR /usr/local/nagios/etc/

Code: Select all

grep nag /etc/group

Code: Select all

ps -efw|grep [n]agios
If you can repeat the process of Write Config, Verify Config (ONLY), then go to the command line and do the following:

Code: Select all

cd /usr/local/nagiosxi/scripts
./reconfigure_nagios.sh
JakeHatMacys
Posts: 281
Joined: Thu Sep 25, 2014 3:21 pm

Re: Nagios XI Configuration Random Configuration Error

Post by JakeHatMacys »

cmerchant wrote:What is the status of the The Monitoring engine? Is everything "green"?

Can you show us the permissions on the following:

Code: Select all

ls -lR /usr/local/nagios/etc/

Code: Select all

grep nag /etc/group

Code: Select all

ps -efw|grep [n]agios
If you can repeat the process of Write Config, Verify Config (ONLY), then go to the command line and do the following:

Code: Select all

cd /usr/local/nagiosxi/scripts
./reconfigure_nagios.sh
Everything is red now...
Capture.JPG
ls -lR /usr/local/nagios/etc/

Code: Select all

 ls -lR /usr/local/nagios/etc/
/usr/local/nagios/etc/:
total 376
-rw-rw-r-- 1 apache nagios  2132 Apr  1 15:38 cgi.cfg
-rw-r--r-- 1 apache nagios 25651 Apr  9 09:14 commands.cfg
-rw-r--r-- 1 apache nagios  1106 Apr  9 09:14 contactgroups.cfg
-rw-r--r-- 1 apache nagios  4758 Apr  9 09:14 contacts.cfg
-rw-r--r-- 1 apache nagios  1675 Apr  9 09:14 contacttemplates.cfg
drwxrwsr-x 7 apache nagios  4096 Jan 29 16:57 etc_back2
-rw-r--r-- 1 apache nagios   817 Apr  9 09:14 hostdependencies.cfg
-rw-r--r-- 1 apache nagios   819 Apr  9 09:14 hostescalations.cfg
-rw-r--r-- 1 apache nagios   837 Apr  9 09:14 hostextinfo.cfg
-rw-r--r-- 1 apache nagios  1475 Apr  9 09:14 hostgroups.cfg
drwsrwsr-x 2 apache nagios 86016 Apr  9 09:14 hosts
-rw-r--r-- 1 apache nagios 15516 Apr  9 09:14 hosttemplates.cfg
drwsrwsr-x 2 apache nagios  4096 Apr  1 15:38 import
-rwxrwxr-x 1 apache nagios  5629 Mar  5 17:24 nagios.cfg
-rwxrwxr-x 1 apache nagios  5597 Dec  4 19:36 nagios.cfg.xi
-rw-rw-r-- 1 apache nagios  2229 Dec  4 19:17 ndo2db.cfg
-rw-rw-r-- 1 apache nagios  4827 Dec  4 19:17 ndomod.cfg
-rw-rw-r-- 1 apache nagios  7989 Mar 25 14:00 nrpe.cfg
-rw-rw-r-- 1 apache nagios  5345 Dec  4 19:17 nsca.cfg
drwxrwsr-x 4 apache nagios  4096 Jan 29 16:57 pnp
-rw-rw-r-- 1 apache nagios     0 Mar 17 22:23 recurringdowntime.cfg
-rwxrwxr-x 1 apache nagios   210 Dec  4 19:15 resource.cfg
-rwxrwxr-x 1 apache nagios   210 Dec  4 19:36 resource.cfg.xi
-rw-rw-r-- 1 apache nagios  1627 Dec  4 19:17 send_nsca.cfg
-rw-r--r-- 1 apache nagios   823 Apr  9 09:14 servicedependencies.cfg
-rw-r--r-- 1 apache nagios   825 Apr  9 09:14 serviceescalations.cfg
-rw-r--r-- 1 apache nagios   843 Apr  9 09:14 serviceextinfo.cfg
-rw-r--r-- 1 apache nagios   813 Apr  9 09:14 servicegroups.cfg
drwsrwsr-x 2 apache nagios 86016 Apr  9 09:14 services
-rw-r--r-- 1 apache nagios 22867 Apr  9 09:14 servicetemplates.cfg
drwsrwsr-x 2 apache nagios  4096 Apr  8 15:49 static
-rw-r--r-- 1 apache nagios  7247 Apr  9 09:14 timeperiods.cfg

/usr/local/nagios/etc/etc_back2:
total 32
drwxrwsr-x 2 apache nagios 4096 Dec  4 19:46 hosts
drwxrwsr-x 2 apache nagios 4096 Dec  4 19:31 import
-rwxrwxr-x 1 apache nagios 5597 Dec  4 19:31 nagios.cfg.xi
drwxrwsr-x 4 apache nagios 4096 Jan 29 16:57 pnp
-rwxrwxr-x 1 apache nagios  210 Dec  4 19:31 resource.cfg.xi
drwxrwsr-x 2 apache nagios 4096 Dec  4 19:46 services
drwxrwsr-x 2 apache nagios 4096 Dec  4 19:46 static

/usr/local/nagios/etc/etc_back2/hosts:
total 0

/usr/local/nagios/etc/etc_back2/import:
total 0

/usr/local/nagios/etc/etc_back2/pnp:
total 40
-rw-rw-r-- 1 apache nagios 6952 Dec  4 19:31 background.pdf
drwxrwsr-x 2 apache nagios 4096 Jan 29 16:57 check_commands
-rw-rw-r-- 1 apache nagios 3417 Dec  4 19:31 config.php
-rw-rw-r-- 1 apache nagios 4127 Dec  4 19:31 npcd.cfg-sample
drwxrwsr-x 2 apache nagios 4096 Jan 29 16:57 pages
-rw-rw-r-- 1 apache nagios   63 Dec  4 19:31 pnp4nagios_release
-rw-rw-r-- 1 apache nagios  793 Dec  4 19:31 process_perfdata.cfg-sample
-rw-rw-r-- 1 apache nagios  524 Dec  4 19:31 rra.cfg-sample

/usr/local/nagios/etc/etc_back2/pnp/check_commands:
total 4
-rw-rw-r-- 1 apache nagios 755 Dec  4 19:31 check_nwstat.cfg-sample

/usr/local/nagios/etc/etc_back2/pnp/pages:
total 4
-rw-rw-r-- 1 apache nagios 324 Dec  4 19:31 web_traffic.cfg-sample

/usr/local/nagios/etc/etc_back2/services:
total 0

/usr/local/nagios/etc/etc_back2/static:
total 0

/usr/local/nagios/etc/hosts:
total 0

/usr/local/nagios/etc/import:
total 0

/usr/local/nagios/etc/pnp:
total 52
-rw-rw-r-- 1 apache nagios 6952 Jan 19 13:24 background.pdf
drwxrwsr-x 2 apache nagios 4096 Jan 29 16:57 check_commands
-rw-rw-r-- 1 apache nagios 4076 Jan 19 13:24 config.php
-rw-rw-r-- 1 apache nagios 3090 Jan 19 13:24 npcd.cfg
-rw-rw-r-- 1 apache nagios 4127 Dec  4 19:17 npcd.cfg-sample
drwxrwsr-x 2 apache nagios 4096 Jan 29 16:57 pages
-rw-rw-r-- 1 apache nagios   63 Jan 19 13:24 pnp4nagios_release
-rw-rw-r-- 1 apache nagios  793 Dec  4 19:17 process_perfdata.cfg
-rw-rw-r-- 1 apache nagios  793 Dec  4 19:17 process_perfdata.cfg-sample
-rw-rw-r-- 1 apache nagios  524 Dec  4 19:17 rra.cfg
-rw-rw-r-- 1 apache nagios  524 Dec  4 19:17 rra.cfg-sample

/usr/local/nagios/etc/pnp/check_commands:
total 4
-rw-rw-r-- 1 apache nagios 755 Dec  4 19:17 check_nwstat.cfg-sample

/usr/local/nagios/etc/pnp/pages:
total 4
-rw-rw-r-- 1 apache nagios 324 Dec  4 19:17 web_traffic.cfg-sample

/usr/local/nagios/etc/services:
total 0

/usr/local/nagios/etc/static:
total 8
-rwxrwxr-x 1 apache nagios  878 Feb  6 15:21 xiobjects.cfg
-rwxrwxr-x 1 apache nagios 4002 Feb  6 15:21 xitemplates.cfg
-rwxrwxr-x 1 apache nagios    0 Feb  6 15:21 xitest.cfg
[root@esu1l268 ~]# grep nag /etc/group
nagios:x:59996:nagios,apache
nagcmd:x:59997:nagios,apache

ps -efw|grep [n]agios

Code: Select all

nagios   29527 29526  0 Apr07 pts/220  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=310 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   29630 26956  0 Mar17 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no [email protected] df -v
nagios   29632 29630  0 Mar17 pts/43   00:00:00 ssh -q -o StrictHostKeyChecking=no [email protected] df -v
nagios   29690     1  0 Apr07 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.48.45.6
nagios   29691 29690  0 Apr07 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   29692 29691  0 Apr07 pts/246  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   29981     1  0 Apr08 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.48.221.75
nagios   29986 29981  0 Apr08 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   29987 29986  0 Apr08 pts/273  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   30227     1  0 Apr07 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.48.24.134
nagios   30228 30227  0 Apr07 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   30229 30228  0 Apr07 pts/124  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   30556     1  0 Mar17 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.48.168.224
nagios   30557 30556  0 Mar17 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   30558 30557  0 Mar17 pts/59   00:00:00 ssh -q -o StrictHostKeyChecking=no [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   30577     1  0 Apr08 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.48.39.198
nagios   30578 30577  0 Apr08 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   30579 30578  0 Apr08 pts/148  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   30629     1  0 Mar20 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_FS_space_linux_by_sshpass.sh 11.48.115.33
nagios   30636 30629  0 Mar20 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   30647 30636  0 Mar20 pts/71   00:00:00 ssh -q -o StrictHostKeyChecking=no [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   30653     1  0 Apr08 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_FS_space_linux_by_sshpass.sh 11.48.221.70
nagios   30654 30653  0 Apr08 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=310 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   30655 30654  0 Apr08 pts/293  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=310 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   30662     1  0 Apr07 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_FS_space_linux_by_sshpass.sh 11.120.39.146
nagios   30672 30662  0 Apr07 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=310 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   30676 30672  0 Apr07 pts/217  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=310 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   30738     1  0 Apr07 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.48.45.115
nagios   30739 30738  0 Apr07 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   30740 30739  0 Apr07 pts/161  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   30761     1  0  2014 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.48.27.87
nagios   30766 30761  0  2014 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   30768 30766  0  2014 pts/52   00:00:00 ssh -q -o StrictHostKeyChecking=no [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   30783  2930  0 14:02 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.120.38.228
nagios   30787 30783  0 14:02 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   30789 30787  0 14:02 pts/335  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   30874     1  0 Apr08 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.48.221.116
nagios   30876 30874  0 Apr08 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   30877 30876  0 Apr08 pts/214  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   30967     1  0 Apr08 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.120.114.58
nagios   30968 30967  0 Apr08 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   30969 30968  0 Apr08 pts/153  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   30970     1  0 Apr08 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.48.115.32
nagios   30971 30970  0 Apr08 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   30972 30971  0 Apr08 pts/166  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
postgres 30983 21530  0 Apr08 ?        00:00:08 postgres: nagiosxi nagiosxi [local] idle
nagios   31037     1  0 Apr08 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.48.27.42
nagios   31040 31037  0 Apr08 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   31041 31040  0 Apr08 pts/178  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   31248     1  0 Apr08 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.48.27.41
nagios   31249 31248  0 Apr08 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   31250 31249  0 Apr08 pts/257  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   31257     1  0 Apr08 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.48.115.32
nagios   31258 31257  0 Apr08 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   31259 31258  0 Apr08 pts/122  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   31341     1  0 Apr07 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.48.21.118
nagios   31342 31341  0 Apr07 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   31343 31342  0 Apr07 pts/204  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   31485     1  0 Jan07 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_FS_space_linux_by_sshpass.sh 11.120.114.58
nagios   31500 31485  0 Jan07 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   31511 31500  0 Jan07 pts/60   00:00:00 ssh -q -o StrictHostKeyChecking=no [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   31696  2961  0 14:02 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_FS_space_AIX_by_sshpass.sh 11.48.25.162
nagios   31698 31696  0 14:02 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -t 5 2 | sed -e 's/,/./g'|tr -s ' ' ';' | tail -1
nagios   31702 31698  0 14:02 pts/392  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -t 5 2 | sed -e 's/,/./g'|tr -s ' ' ';' | tail -1
nagios   31784  2929  0 14:02 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_FS_space_AIX_by_sshpass.sh 11.122.16.42
nagios   31786 31784  0 14:02 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -t 5 2 | sed -e 's/,/./g'|tr -s ' ' ';' | tail -1
nagios   31787 31786  0 14:02 pts/358  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -t 5 2 | sed -e 's/,/./g'|tr -s ' ' ';' | tail -1
nagios   31788  2931  0 14:02 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_FS_space_AIX_by_sshpass.sh 11.120.113.52
nagios   31789 31788  0 14:02 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -t 5 2 | sed -e 's/,/./g'|tr -s ' ' ';' | tail -1
nagios   31790 31789  0 14:02 pts/420  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -t 5 2 | sed -e 's/,/./g'|tr -s ' ' ';' | tail -1
nagios   31953  2941  0 14:02 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_FS_space_AIX_by_sshpass.sh 11.48.114.50
nagios   31954 31953  0 14:02 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -t 5 2 | sed -e 's/,/./g'|tr -s ' ' ';' | tail -1
nagios   31955 31954  0 14:02 pts/463  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -t 5 2 | sed -e 's/,/./g'|tr -s ' ' ';' | tail -1
nagios   31980     1  0 Jan29 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_FS_space_linux_by_sshpass.sh 11.48.45.116
nagios   31997 31980  0 Jan29 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   32007 31997  0 Jan29 pts/91   00:00:00 ssh -q -o StrictHostKeyChecking=no [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   32035     1  0 Mar17 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_FS_space_linux_by_sshpass.sh 11.122.80.14
nagios   32037 32035  0 Mar17 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   32038 32037  0 Mar17 pts/93   00:00:00 ssh -q -o StrictHostKeyChecking=no [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   32058     1  0 Mar17 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_FS_space_linux_by_sshpass.sh 11.120.114.58
nagios   32059 32058  0 Mar17 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   32060 32059  0 Mar17 pts/111  00:00:00 ssh -q -o StrictHostKeyChecking=no [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   32097     1  0  2014 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.120.114.58
nagios   32104 32097  0  2014 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   32107 32104  0  2014 pts/19   00:00:00 ssh -q -o StrictHostKeyChecking=no [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   32130     1  0 Apr08 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_FS_space_linux_by_sshpass.sh 11.120.113.102
nagios   32131 32130  0 Apr08 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=310 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   32132 32131  0 Apr08 pts/425  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=310 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   32133     1  0 Apr07 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.48.45.6
nagios   32134 32133  0 Apr07 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   32135 32134  0 Apr07 pts/205  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   32550     1  0 Apr08 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.48.21.126
nagios   32551 32550  0 Apr08 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   32552 32551  0 Apr08 pts/382  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
root     32615     1  0 Mar06 ?        00:00:00 /bin/sh /opt/OV/HPBsmIntNagios/genint/bin/genintd.sh /opt/OV/HPBsmIntNagios/genint/conf/nagios_server.conf
nagios   32617     1  0 Apr07 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.48.115.37
root     32619 32615  0 Mar06 ?        04:47:54 /opt/OV/HPBsmIntNagios/genint/bin/genintd /opt/OV/HPBsmIntNagios/genint/conf/nagios_server.conf
nagios   32622 32617  0 Apr07 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   32624 32622  0 Apr07 pts/145  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   32724  2960  0 14:02 ?        00:00:00 /bin/sh /usr/local/nagios/libexec/check_read_only_fs_linux_by_sshpass.sh 11.120.113.16
nagios   32725 32724  0 14:02 ?        00:00:00 /usr/local/nagios/sshpass-1.05/sshpass -pzzzzzzz ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
nagios   32726 32725  0 14:02 pts/415  00:00:00 ssh -q -o StrictHostKeyChecking=no -o ConnectTimeout=300 [email protected] iostat -c 5 2 |  tr -s ' ' ';' | sed '/^$/d' | tail -1
[root@esu1l268 ~]#
Reconfigure is still running but it looks like it's stuck on this:

./reconfigure_nagios.sh

Another reconfigure process is still running, sleeping...

NM it finally went through. Shows all the warning for no contacts and then this:

Code: Select all

        Checked 2048 hosts.
        Checked 5 host groups.
        Checked 0 service groups.
        Checked 10 contacts.
        Checked 2 contact groups.
        Checked 122 commands.
        Checked 16 time periods.
        Checked 0 host escalations.
        Checked 0 service escalations.
Checking for circular paths...
        Checked 2048 hosts
        Checked 0 service dependencies
        Checked 0 host dependencies
        Checked 16 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 7419
Total Errors:   0

Things look okay - No serious problems were detected during the pre-flight check
RET: 0
Running configuration check...
Stopping nagios:. done.
Starting nagios: done.
And after running this still no changes :/ still red
You do not have the required permissions to view the files attached to this post.
cmerchant
Posts: 546
Joined: Wed Sep 24, 2014 11:19 am

Re: Nagios XI Configuration Random Configuration Error

Post by cmerchant »

Noticed the redness on your monitoring engine status... I think in order to have the apply configuration work, we need to have nagios, ndo2db restarted.
But I would check the health of your database services as well.

Code: Select all

service nagios stop
service nagios status
service ndo2db stop
service ndo2db status
service mysqld stop
service mysqld status
service mysqld start
tail -100 /var/log/mysqld.log
service ndo2db start
tail -100 /var/log/messages
service nagios start
tail -100 /usr/local/nagios/var/nagios.log
after all of that, could you verify the status of the monitoring engine status.

Code: Select all

cat /usr/local/nagiosxi/var/sysstat.log
JakeHatMacys
Posts: 281
Joined: Thu Sep 25, 2014 3:21 pm

Re: Nagios XI Configuration Random Configuration Error

Post by JakeHatMacys »

cmerchant wrote:Noticed the redness on your monitoring engine status... I think in order to have the apply configuration work, we need to have nagios, ndo2db restarted.
But I would check the health of your database services as well.

Code: Select all

service nagios stop
service nagios status
service ndo2db stop
service ndo2db status
service mysqld stop
service mysqld status
service mysqld start
tail -100 /var/log/mysqld.log
service ndo2db start
tail -100 /var/log/messages
service nagios start
tail -100 /usr/local/nagios/var/nagios.log
after all of that, could you verify the status of the monitoring engine status.

Code: Select all

cat /usr/local/nagiosxi/var/sysstat.log
Did all that, and still Red. Applying config just clocks endlessly now.

This is the result of cat /usr/local/nagiosxi/var/sysstat.log:

Code: Select all

            [min_execution_time] => 0.00127
            [max_execution_time] => 60.017
            [avg_execution_time] => 8.1480894702602
        )

)
PHP Warning:  exec(): Unable to fork [/usr/bin/uptime | sed s/,//g | awk -F'average: ' '{  print $2 }'] in /usr/local/nagiosxi/cron/sysstat.php on line 198
PHP Notice:  Undefined offset: 0 in /usr/local/nagiosxi/cron/sysstat.php on line 201
PHP Notice:  Undefined offset: 1 in /usr/local/nagiosxi/cron/sysstat.php on line 205
PHP Notice:  Undefined offset: 2 in /usr/local/nagiosxi/cron/sysstat.php on line 206
LOAD:
Array
(
    [load1] =>
    [load5] =>
    [load15] =>
)
PHP Warning:  exec(): Unable to fork [/usr/bin/free -m | head --lines=2 | tail --lines=1 | awk '{ print $2,$3,$4,$5,$6,$7}'] in /usr/local/nagiosxi/cron/sysstat.php on line 220
PHP Notice:  Undefined offset: 0 in /usr/local/nagiosxi/cron/sysstat.php on line 223
PHP Notice:  Undefined offset: 1 in /usr/local/nagiosxi/cron/sysstat.php on line 227
PHP Notice:  Undefined offset: 2 in /usr/local/nagiosxi/cron/sysstat.php on line 228
PHP Notice:  Undefined offset: 3 in /usr/local/nagiosxi/cron/sysstat.php on line 229
PHP Notice:  Undefined offset: 4 in /usr/local/nagiosxi/cron/sysstat.php on line 230
PHP Notice:  Undefined offset: 5 in /usr/local/nagiosxi/cron/sysstat.php on line 231
MEMORY:
Array
(
    [total] =>
    [used] =>
    [free] =>
    [shared] =>
    [buffers] =>
    [cached] =>
)
PHP Warning:  exec(): Unable to fork [/usr/bin/free -m | tail --lines=1 | awk '{ print $2,$3,$4}'] in /usr/local/nagiosxi/cron/sysstat.php on line 244
PHP Notice:  Undefined offset: 0 in /usr/local/nagiosxi/cron/sysstat.php on line 247
PHP Notice:  Undefined offset: 1 in /usr/local/nagiosxi/cron/sysstat.php on line 251
PHP Notice:  Undefined offset: 2 in /usr/local/nagiosxi/cron/sysstat.php on line 252
SWAP:
Array
(
    [total] =>
    [used] =>
    [free] =>
)
PHP Warning:  exec(): Unable to fork [/usr/bin/iostat -c 5 2 | tail --lines=2 | head --lines=1 | awk '{ print $1,$2,$3,$4,$5,$6 }'] in /usr/local/nagiosxi/cron/sysstat.php on line 265
PHP Notice:  Undefined offset: 0 in /usr/local/nagiosxi/cron/sysstat.php on line 268
PHP Notice:  Undefined offset: 1 in /usr/local/nagiosxi/cron/sysstat.php on line 272
PHP Notice:  Undefined offset: 2 in /usr/local/nagiosxi/cron/sysstat.php on line 273
PHP Notice:  Undefined offset: 3 in /usr/local/nagiosxi/cron/sysstat.php on line 274
PHP Notice:  Undefined offset: 4 in /usr/local/nagiosxi/cron/sysstat.php on line 275
PHP Notice:  Undefined offset: 5 in /usr/local/nagiosxi/cron/sysstat.php on line 276
IOSTAT:
Array
(
    [user] =>
    [nice] =>
    [system] =>
    [iowait] =>
    [steal] =>
    [idle] =>
)
.DB BACKEND:
Array
(
    [last_checkin] => 2015-04-08 14:40:59
    [bytes_processed] => 18156270
    [entries_processed] => 38119
    [connect_time] => 2015-04-08 14:36:55
    [disconnect_time] => 0000-00-00 00:00:00
)
CMDLINE=/etc/init.d/nagios status
PHP Warning:  system(): Unable to fork [/etc/init.d/nagios status] in /usr/local/nagiosxi/cron/sysstat.php on line 159
OUTPUT=
RETURNCODE=-1
CMDLINE=/etc/init.d/npcd status
PHP Warning:  system(): Unable to fork [/etc/init.d/npcd status] in /usr/local/nagiosxi/cron/sysstat.php on line 159
OUTPUT=
RETURNCODE=-1
CMDLINE=/etc/init.d/ndo2db status
PHP Warning:  system(): Unable to fork [/etc/init.d/ndo2db status] in /usr/local/nagiosxi/cron/sysstat.php on line 159
OUTPUT=
RETURNCODE=-1
DAEMONS:
Array
(
    [nagioscore] => Array
        (
            [daemon] => nagios
            [output] =>
            [return_code] => -1
            [status] => 1
        )

    [pnp] => Array
        (
            [daemon] => npcd
            [output] =>
            [return_code] => -1
            [status] => 1
        )

    [ndoutils] => Array
        (
            [daemon] => ndo2db
            [output] =>
            [return_code] => -1
            [status] => 1
        )

)
CORE STATS:
Array
(
    [activehostchecks] => Array
        (
            [1min] => 235
            [5min] => 1144
            [15min] => 1144
        )

    [passivehostchecks] => Array
        (
            [1min] => 0
            [5min] => 0
            [15min] => 0
        )

    [activeservicechecks] => Array
        (
            [1min] => 341
            [5min] => 1440
            [15min] => 1440
        )

    [passiveservicechecks] => Array
        (
            [1min] => 0
            [5min] => 0
            [15min] => 0
        )

    [activehostcheckperf] => Array
        (
            [min_latency] => 0
            [max_latency] => 1.72661
            [avg_latency] => 0.03091142578125
            [min_execution_time] => 0.00167
            [max_execution_time] => 10.00291
            [avg_execution_time] => 2.8069380517578
        )

    [activeservicecheckperf] => Array
        (
            [min_latency] => 0
            [max_latency] => 7.754
            [avg_latency] => 0.0664435724907063
            [min_execution_time] => 0.00127
            [max_execution_time] => 60.017
            [avg_execution_time] => 8.1498299758364
        )

)
PHP Warning:  exec(): Unable to fork [/usr/bin/uptime | sed s/,//g | awk -F'average: ' '{  print $2 }'] in /usr/local/nagiosxi/cron/sysstat.php on line 198
PHP Notice:  Undefined offset: 0 in /usr/local/nagiosxi/cron/sysstat.php on line 201
PHP Notice:  Undefined offset: 1 in /usr/local/nagiosxi/cron/sysstat.php on line 205
PHP Notice:  Undefined offset: 2 in /usr/local/nagiosxi/cron/sysstat.php on line 206
LOAD:
Array
(
    [load1] =>
    [load5] =>
    [load15] =>
)
PHP Warning:  exec(): Unable to fork [/usr/bin/free -m | head --lines=2 | tail --lines=1 | awk '{ print $2,$3,$4,$5,$6,$7}'] in /usr/local/nagiosxi/cron/sysstat.php on line 220
PHP Notice:  Undefined offset: 0 in /usr/local/nagiosxi/cron/sysstat.php on line 223
PHP Notice:  Undefined offset: 1 in /usr/local/nagiosxi/cron/sysstat.php on line 227
PHP Notice:  Undefined offset: 2 in /usr/local/nagiosxi/cron/sysstat.php on line 228
PHP Notice:  Undefined offset: 3 in /usr/local/nagiosxi/cron/sysstat.php on line 229
PHP Notice:  Undefined offset: 4 in /usr/local/nagiosxi/cron/sysstat.php on line 230
PHP Notice:  Undefined offset: 5 in /usr/local/nagiosxi/cron/sysstat.php on line 231
MEMORY:
Array
(
    [total] =>
    [used] =>
    [free] =>
    [shared] =>
    [buffers] =>
    [cached] =>
)
PHP Warning:  exec(): Unable to fork [/usr/bin/free -m | tail --lines=1 | awk '{ print $2,$3,$4}'] in /usr/local/nagiosxi/cron/sysstat.php on line 244
PHP Notice:  Undefined offset: 0 in /usr/local/nagiosxi/cron/sysstat.php on line 247
PHP Notice:  Undefined offset: 1 in /usr/local/nagiosxi/cron/sysstat.php on line 251
PHP Notice:  Undefined offset: 2 in /usr/local/nagiosxi/cron/sysstat.php on line 252
SWAP:
Array
(
    [total] =>
    [used] =>
    [free] =>
)
PHP Warning:  exec(): Unable to fork [/usr/bin/iostat -c 5 2 | tail --lines=2 | head --lines=1 | awk '{ print $1,$2,$3,$4,$5,$6 }'] in /usr/local/nagiosxi/cron/sysstat.php on line 265
PHP Notice:  Undefined offset: 0 in /usr/local/nagiosxi/cron/sysstat.php on line 268
PHP Notice:  Undefined offset: 1 in /usr/local/nagiosxi/cron/sysstat.php on line 272
PHP Notice:  Undefined offset: 2 in /usr/local/nagiosxi/cron/sysstat.php on line 273
PHP Notice:  Undefined offset: 3 in /usr/local/nagiosxi/cron/sysstat.php on line 274
PHP Notice:  Undefined offset: 4 in /usr/local/nagiosxi/cron/sysstat.php on line 275
PHP Notice:  Undefined offset: 5 in /usr/local/nagiosxi/cron/sysstat.php on line 276
IOSTAT:
Array
(
    [user] =>
    [nice] =>
    [system] =>
    [iowait] =>
    [steal] =>
    [idle] =>
)
.Done
[root@esu1l268 ~]#

tailing the ndo2db:

Code: Select all

141211  9:10:24  InnoDB: Completed initialization of buffer pool
141211  9:10:24  InnoDB: Started; log sequence number 0 44263
141211  9:10:24 [Note] Event Scheduler: Loaded 0 events
141211  9:10:24 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.67'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
141212 15:33:10 [Note] /usr/libexec/mysqld: Normal shutdown

141212 15:33:10 [Note] Event Scheduler: Purging the queue. 0 events
141212 15:33:12  InnoDB: Starting shutdown...
141212 15:33:16  InnoDB: Shutdown completed; log sequence number 0 44273
141212 15:33:16 [Note] /usr/libexec/mysqld: Shutdown complete

141212 15:33:16 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
141212 15:33:17 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
141212 15:33:17  InnoDB: Initializing buffer pool, size = 8.0M
141212 15:33:17  InnoDB: Completed initialization of buffer pool
141212 15:33:17  InnoDB: Started; log sequence number 0 44273
141212 15:33:17 [Note] Event Scheduler: Loaded 0 events
141212 15:33:17 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.67'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
141212 15:48:29 [Note] /usr/libexec/mysqld: Normal shutdown

141212 15:48:29 [Note] Event Scheduler: Purging the queue. 0 events
141212 15:48:31  InnoDB: Starting shutdown...
141212 15:48:33  InnoDB: Shutdown completed; log sequence number 0 44273
141212 15:48:33 [Note] /usr/libexec/mysqld: Shutdown complete

141212 15:48:33 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
141212 15:54:33 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
141212 15:54:33  InnoDB: Initializing buffer pool, size = 8.0M
141212 15:54:33  InnoDB: Completed initialization of buffer pool
141212 15:54:33  InnoDB: Started; log sequence number 0 44273
141212 15:54:33 [Note] Event Scheduler: Loaded 0 events
141212 15:54:33 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.67'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
150119  8:06:38 [Note] /usr/libexec/mysqld: Normal shutdown

150119  8:06:38 [Note] Event Scheduler: Purging the queue. 0 events
150119  8:06:40  InnoDB: Starting shutdown...
150119  8:06:44  InnoDB: Shutdown completed; log sequence number 0 44283
150119  8:06:44 [Note] /usr/libexec/mysqld: Shutdown complete

150119 08:06:44 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150119 08:06:44 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150119  8:06:44  InnoDB: Initializing buffer pool, size = 8.0M
150119  8:06:44  InnoDB: Completed initialization of buffer pool
150119  8:06:44  InnoDB: Started; log sequence number 0 44283
150119  8:06:44 [Note] Event Scheduler: Loaded 0 events
150119  8:06:44 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.67'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
150320 14:23:45 [Note] /usr/libexec/mysqld: Normal shutdown

150320 14:23:45 [Note] Event Scheduler: Purging the queue. 0 events
150320 14:23:47  InnoDB: Starting shutdown...
150320 14:23:51  InnoDB: Shutdown completed; log sequence number 0 44293
150320 14:23:51 [Note] /usr/libexec/mysqld: Shutdown complete

150320 14:23:51 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150320 14:23:52 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150320 14:23:52  InnoDB: Initializing buffer pool, size = 8.0M
150320 14:23:52  InnoDB: Completed initialization of buffer pool
150320 14:23:52  InnoDB: Started; log sequence number 0 44293
150320 14:23:52 [Note] Event Scheduler: Loaded 0 events
150320 14:23:52 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.67'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
150408 14:36:44 [Note] /usr/libexec/mysqld: Normal shutdown

150408 14:36:44 [Note] Event Scheduler: Purging the queue. 0 events
150408 14:36:46  InnoDB: Starting shutdown...
150408 14:36:49  InnoDB: Shutdown completed; log sequence number 0 44303
150408 14:36:49 [Note] /usr/libexec/mysqld: Shutdown complete

150408 14:36:49 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150408 14:36:49 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150408 14:36:49  InnoDB: Initializing buffer pool, size = 8.0M
150408 14:36:49  InnoDB: Completed initialization of buffer pool
150408 14:36:49  InnoDB: Started; log sequence number 0 44303
150408 14:36:49 [Note] Event Scheduler: Loaded 0 events
150408 14:36:49 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.67'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
150409 18:46:43 [Note] /usr/libexec/mysqld: Normal shutdown

150409 18:46:43 [Note] Event Scheduler: Purging the queue. 0 events
150409 18:46:45  InnoDB: Starting shutdown...
150409 18:46:46  InnoDB: Shutdown completed; log sequence number 0 44303
150409 18:46:46 [Note] /usr/libexec/mysqld: Shutdown complete

150409 18:46:46 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150409 18:47:00 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150409 18:47:00  InnoDB: Initializing buffer pool, size = 8.0M
150409 18:47:00  InnoDB: Completed initialization of buffer pool
150409 18:47:00  InnoDB: Started; log sequence number 0 44303
150409 18:47:00 [Note] Event Scheduler: Loaded 0 events
150409 18:47:00 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.67'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
[root@esu1l268 scripts]# service ndo2db start
Starting ndo2db: done.
[root@esu1l268 scripts]# tail -100 /var/log/messages
Apr  9 18:45:02 esu1l268 nagios: wproc: Core Worker 22208: job 5131 (pid=834): Dormant child reaped
Apr  9 18:45:02 esu1l268 nagios: wproc: Core Worker 22211: job 5131 (pid=890) timed out. Killing it
Apr  9 18:45:02 esu1l268 nagios: wproc: CHECK job 5131 from worker Core Worker 22211 timed out after 60.01s
Apr  9 18:45:02 esu1l268 nagios: wproc:   host=esu1l318; service=check_FS_space_linux_by_sshpass;
Apr  9 18:45:02 esu1l268 nagios: wproc:   early_timeout=1; exited_ok=0; wait_status=0; error_code=62;
Apr  9 18:45:02 esu1l268 nagios: Warning: Check of service 'check_FS_space_linux_by_sshpass' on host 'esu1l                  318' timed out after 60.007s!
Apr  9 18:45:02 esu1l268 nagios: wproc: Core Worker 22211: job 5131 (pid=890): Dormant child reaped
Apr  9 18:45:03 esu1l268 nagios: wproc: Core Worker 22245: job 5131 (pid=1150) timed out. Killing it
Apr  9 18:45:03 esu1l268 nagios: wproc: CHECK job 5131 from worker Core Worker 22245 timed out after 60.02s
Apr  9 18:45:03 esu1l268 nagios: wproc:   host=esu1l321; service=check_FS_space_linux_by_sshpass;
Apr  9 18:45:03 esu1l268 nagios: wproc:   early_timeout=1; exited_ok=0; wait_status=0; error_code=62;
Apr  9 18:45:03 esu1l268 nagios: Warning: Check of service 'check_FS_space_linux_by_sshpass' on host 'esu1l                  321' timed out after 60.017s!
Apr  9 18:45:03 esu1l268 nagios: wproc: Core Worker 22245: job 5131 (pid=1150): Dormant child reaped
Apr  9 18:45:04 esu1l268 nagios: wproc: Core Worker 22200: job 5132 (pid=1245) timed out. Killing it
Apr  9 18:45:04 esu1l268 nagios: wproc: CHECK job 5132 from worker Core Worker 22200 timed out after 60.01s
Apr  9 18:45:04 esu1l268 nagios: wproc:   host=esu1l325; service=check_FS_space_linux_by_sshpass;
Apr  9 18:45:04 esu1l268 nagios: wproc:   early_timeout=1; exited_ok=0; wait_status=0; error_code=62;
Apr  9 18:45:04 esu1l268 nagios: Warning: Check of service 'check_FS_space_linux_by_sshpass' on host 'esu1l                  325' timed out after 60.007s!
Apr  9 18:45:04 esu1l268 nagios: wproc: Core Worker 22200: job 5132 (pid=1245): Dormant child reaped
Apr  9 18:45:07 esu1l268 nagios: wproc: Core Worker 22207: job 5133 (pid=2386) timed out. Killing it
Apr  9 18:45:07 esu1l268 nagios: wproc: CHECK job 5133 from worker Core Worker 22207 timed out after 60.01s
Apr  9 18:45:07 esu1l268 nagios: wproc:   host=esu1l327; service=check_FS_space_linux_by_sshpass;
Apr  9 18:45:07 esu1l268 nagios: wproc:   early_timeout=1; exited_ok=0; wait_status=0; error_code=62;
Apr  9 18:45:07 esu1l268 nagios: Warning: Check of service 'check_FS_space_linux_by_sshpass' on host 'esu1l                  327' timed out after 60.006s!
Apr  9 18:45:07 esu1l268 nagios: wproc: Core Worker 22207: job 5133 (pid=2386): Dormant child reaped
Apr  9 18:45:09 esu1l268 nagios: wproc: Core Worker 22244: job 5133 (pid=2900) timed out. Killing it
Apr  9 18:45:09 esu1l268 nagios: wproc: CHECK job 5133 from worker Core Worker 22244 timed out after 60.01s
Apr  9 18:45:09 esu1l268 nagios: wproc:   host=esu1l326; service=check_FS_space_linux_by_sshpass;
Apr  9 18:45:09 esu1l268 nagios: wproc:   early_timeout=1; exited_ok=0; wait_status=0; error_code=62;
Apr  9 18:45:09 esu1l268 nagios: Warning: Check of service 'check_FS_space_linux_by_sshpass' on host 'esu1l                  326' timed out after 60.006s!
Apr  9 18:45:09 esu1l268 nagios: wproc: Core Worker 22244: job 5133 (pid=2900): Dormant child reaped
Apr  9 18:45:13 esu1l268 nagios: wproc: Core Worker 22208: job 5135 (pid=3757) timed out. Killing it
Apr  9 18:45:13 esu1l268 nagios: wproc: CHECK job 5135 from worker Core Worker 22208 timed out after 60.01s
Apr  9 18:45:13 esu1l268 nagios: wproc:   host=esu2v262; service=Check for read only File system on linux;
Apr  9 18:45:13 esu1l268 nagios: wproc:   early_timeout=1; exited_ok=0; wait_status=0; error_code=62;
Apr  9 18:45:13 esu1l268 nagios: Warning: Check of service 'Check for read only File system on linux' on ho                  st 'esu2v262' timed out after 60.005s!
Apr  9 18:45:13 esu1l268 nagios: wproc: Core Worker 22208: job 5135 (pid=3757): Dormant child reaped
Apr  9 18:45:16 esu1l268 vasd[2315]: _create_rule_from_local_file: Cannot locate <FEDERATED.FDS\UxRG_esu1l2                  68>(UxRG_esu1l268) from file </etc/opt/quest/vas/users.allow>, error: <2>. Resolution will be attempted aga                  in in 30 seconds.
Apr  9 18:45:25 esu1l268 nagios: wproc: Core Worker 22248: job 5140 (pid=7581) timed out. Killing it
Apr  9 18:45:25 esu1l268 nagios: wproc: CHECK job 5140 from worker Core Worker 22248 timed out after 60.01s
Apr  9 18:45:25 esu1l268 nagios: wproc:   host=scj3dbadm01; service=check_FS_space_Solaris_by_sshpass;
Apr  9 18:45:25 esu1l268 nagios: wproc:   early_timeout=1; exited_ok=0; wait_status=0; error_code=62;
Apr  9 18:45:25 esu1l268 nagios: Warning: Check of service 'check_FS_space_Solaris_by_sshpass' on host 'scj                  3dbadm01' timed out after 60.006s!
Apr  9 18:45:25 esu1l268 nagios: wproc: Core Worker 22248: job 5140 (pid=7581): Dormant child reaped
Apr  9 18:45:38 esu1l268 nagios: wproc: Core Worker 22219: job 5145 (pid=11115) timed out. Killing it
Apr  9 18:45:38 esu1l268 nagios: wproc: CHECK job 5145 from worker Core Worker 22219 timed out after 60.01s
Apr  9 18:45:38 esu1l268 nagios: wproc:   host=esu2v660; service=Check for read only File system on linux;
Apr  9 18:45:38 esu1l268 nagios: wproc:   early_timeout=1; exited_ok=0; wait_status=0; error_code=62;
Apr  9 18:45:38 esu1l268 nagios: Warning: Check of service 'Check for read only File system on linux' on ho                  st 'esu2v660' timed out after 60.006s!
Apr  9 18:45:38 esu1l268 nagios: wproc: Core Worker 22219: job 5145 (pid=11115): Dormant child reaped
Apr  9 18:45:39 esu1l268 nagios: wproc: Core Worker 22237: job 5145 (pid=11234) timed out. Killing it
Apr  9 18:45:39 esu1l268 nagios: wproc: CHECK job 5145 from worker Core Worker 22237 timed out after 60.01s
Apr  9 18:45:39 esu1l268 nagios: wproc:   host=esu2v219; service=Check for read only File system on linux;
Apr  9 18:45:39 esu1l268 nagios: wproc:   early_timeout=1; exited_ok=0; wait_status=0; error_code=62;
Apr  9 18:45:39 esu1l268 nagios: Warning: Check of service 'Check for read only File system on linux' on ho                  st 'esu2v219' timed out after 60.008s!
Apr  9 18:45:39 esu1l268 nagios: wproc: Core Worker 22237: job 5145 (pid=11234): Dormant child reaped
Apr  9 18:45:41 esu1l268 nagios: wproc: Core Worker 22205: job 5146 (pid=11753) timed out. Killing it
Apr  9 18:45:41 esu1l268 nagios: wproc: CHECK job 5146 from worker Core Worker 22205 timed out after 60.01s
Apr  9 18:45:41 esu1l268 nagios: wproc:   host=sun21; service=check_FS_space_Solaris_by_sshpass;
Apr  9 18:45:41 esu1l268 nagios: wproc:   early_timeout=1; exited_ok=0; wait_status=0; error_code=62;
Apr  9 18:45:41 esu1l268 nagios: Warning: Check of service 'check_FS_space_Solaris_by_sshpass' on host 'sun                  21' timed out after 60.008s!
Apr  9 18:45:41 esu1l268 nagios: wproc: Core Worker 22205: job 5146 (pid=11753): Dormant child reaped
Apr  9 18:45:47 esu1l268 vasd[2315]: _create_rule_from_local_file: Cannot locate <FEDERATED.FDS\UxRG_esu1l2                  68>(UxRG_esu1l268) from file </etc/opt/quest/vas/users.allow>, error: <2>. Resolution will be attempted aga                  in in 30 seconds.
Apr  9 18:45:51 esu1l268 nagios: wproc: Core Worker 22200: job 5150 (pid=13976) timed out. Killing it
Apr  9 18:45:51 esu1l268 nagios: wproc: CHECK job 5150 from worker Core Worker 22200 timed out after 60.01s
Apr  9 18:45:51 esu1l268 nagios: wproc:   host=esu1l316; service=check_FS_space_linux_by_sshpass;
Apr  9 18:45:51 esu1l268 nagios: wproc:   early_timeout=1; exited_ok=0; wait_status=0; error_code=62;
Apr  9 18:45:51 esu1l268 nagios: Warning: Check of service 'check_FS_space_linux_by_sshpass' on host 'esu1l                  316' timed out after 60.006s!
Apr  9 18:45:51 esu1l268 nagios: wproc: Core Worker 22200: job 5150 (pid=13976): Dormant child reaped
Apr  9 18:45:53 esu1l268 nagios: wproc: Core Worker 22234: job 5150 (pid=14403) timed out. Killing it
Apr  9 18:45:53 esu1l268 nagios: wproc: CHECK job 5150 from worker Core Worker 22234 timed out after 60.01s
Apr  9 18:45:53 esu1l268 nagios: wproc:   host=esu1l044; service=check_FS_space_linux_by_sshpass;
Apr  9 18:45:53 esu1l268 nagios: wproc:   early_timeout=1; exited_ok=0; wait_status=0; error_code=62;
Apr  9 18:45:53 esu1l268 nagios: Warning: Check of service 'check_FS_space_linux_by_sshpass' on host 'esu1l                  044' timed out after 60.007s!
Apr  9 18:45:53 esu1l268 nagios: wproc: Core Worker 22234: job 5150 (pid=14403): Dormant child reaped
Apr  9 18:45:55 esu1l268 nagios: wproc: Core Worker 22215: job 5151 (pid=14765) timed out. Killing it
Apr  9 18:45:55 esu1l268 nagios: wproc: CHECK job 5151 from worker Core Worker 22215 timed out after 60.01s
Apr  9 18:45:55 esu1l268 nagios: wproc:   host=esu1l332; service=check_FS_space_linux_by_sshpass;
Apr  9 18:45:55 esu1l268 nagios: wproc:   early_timeout=1; exited_ok=0; wait_status=0; error_code=62;
Apr  9 18:45:55 esu1l268 nagios: Warning: Check of service 'check_FS_space_linux_by_sshpass' on host 'esu1l                  332' timed out after 60.007s!
Apr  9 18:45:55 esu1l268 nagios: wproc: Core Worker 22215: job 5151 (pid=14765): Dormant child reaped
Apr  9 18:45:59 esu1l268 nagios: wproc: Core Worker 22236: job 5152 (pid=15707) timed out. Killing it
Apr  9 18:45:59 esu1l268 nagios: wproc: CHECK job 5152 from worker Core Worker 22236 timed out after 60.01s
Apr  9 18:45:59 esu1l268 nagios: wproc:   host=esu3v128; service=Check for read only File system on linux;
Apr  9 18:45:59 esu1l268 nagios: wproc:   early_timeout=1; exited_ok=0; wait_status=0; error_code=62;
Apr  9 18:45:59 esu1l268 nagios: Warning: Check of service 'Check for read only File system on linux' on ho                  st 'esu3v128' timed out after 60.005s!
Apr  9 18:45:59 esu1l268 nagios: wproc: Core Worker 22236: job 5152 (pid=15707): Dormant child reaped
Apr  9 18:46:01 esu1l268 nagios: Caught SIGTERM, shutting down...
Apr  9 18:46:01 esu1l268 nagios: Successfully shutdown... (PID=22192)
Apr  9 18:46:01 esu1l268 nagios: Event broker module 'NERD' deinitialized successfully.
Apr  9 18:46:01 esu1l268 nagios: HP BSM Integration: Event Broker Module has been successfully unloaded.
Apr  9 18:46:01 esu1l268 nagios: Event broker module '/opt/OV/HPBsmIntNagios/lib64/libbsmintneb4.so' deinit                  ialized successfully.
Apr  9 18:46:01 esu1l268 nagios: ndomod: Shutdown complete.
Apr  9 18:46:01 esu1l268 nagios: Event broker module '/usr/local/nagios/bin/ndomod.o' deinitialized success                  fully.
Apr  9 18:46:17 esu1l268 vasd[2315]: _create_rule_from_local_file: Cannot locate <FEDERATED.FDS\UxRG_esu1l2                  68>(UxRG_esu1l268) from file </etc/opt/quest/vas/users.allow>, error: <2>. Resolution will be attempted aga                  in in 30 seconds.
Apr  9 18:46:32 esu1l268 snmpd[2527]: Connection from UDP: [11.48.116.70]:48920->[11.48.4.85]
Apr  9 18:46:48 esu1l268 vasd[2315]: _create_rule_from_local_file: Cannot locate <FEDERATED.FDS\UxRG_esu1l2                  68>(UxRG_esu1l268) from file </etc/opt/quest/vas/users.allow>, error: <2>. Resolution will be attempted aga                  in in 30 seconds.
Apr  9 18:47:19 esu1l268 vasd[2315]: _create_rule_from_local_file: Cannot locate <FEDERATED.FDS\UxRG_esu1l2                  68>(UxRG_esu1l268) from file </etc/opt/quest/vas/users.allow>, error: <2>. Resolution will be attempted aga                  in in 30 seconds.
Apr  9 18:47:42 esu1l268 sshd[24527]: Accepted keyboard-interactive/pam for b161172 from 11.48.70.73 port 2                  5197 ssh2
Apr  9 18:47:49 esu1l268 vasd[2315]: _create_rule_from_local_file: Cannot locate <FEDERATED.FDS\UxRG_esu1l2                  68>(UxRG_esu1l268) from file </etc/opt/quest/vas/users.allow>, error: <2>. Resolution will be attempted aga                  in in 30 seconds.
[root@esu1l268 scripts]#
tail -100 /usr/local/nagios/var/nagios.log:

Code: Select all

[root@esu1l268 ~]# tail -100 /usr/local/nagios/var/nagios.log
[1428620223] wproc:   host=esu2v805; service=Check for read only File system on linux;
[1428620223] wproc:   early_timeout=1; exited_ok=0; wait_status=0; error_code=62;
[1428620223] Warning: Check of service 'Check for read only File system on linux' on host 'esu2v805' timed out after 60.011s!
[1428620223] wproc: Core Worker 24762: job 139 (pid=7841): Dormant child reaped
[1428620224] wproc: Core Worker 24753: job 139 (pid=7879) timed out. Killing it
[1428620224] wproc: CHECK job 139 from worker Core Worker 24753 timed out after 60.01s
[1428620224] wproc:   host=esu2v823; service=Check for read only File system on linux;
[1428620224] wproc:   early_timeout=1; exited_ok=0; wait_status=0; error_code=62;
[1428620224] Warning: Check of service 'Check for read only File system on linux' on host 'esu2v823' timed out after 60.010s!
[1428620224] wproc: Core Worker 24753: job 139 (pid=7879): Dormant child reaped
[1428620230] wproc: Core Worker 24752: job 142 (pid=8419) timed out. Killing it
[1428620230] wproc: CHECK job 142 from worker Core Worker 24752 timed out after 60.01s
[1428620230] wproc:   host=esu2v974; service=Check for read only File system on linux;
[1428620230] wproc:   early_timeout=1; exited_ok=0; wait_status=0; error_code=62;
[1428620230] Warning: Check of service 'Check for read only File system on linux' on host 'esu2v974' timed out after 60.006s!
[1428620230] wproc: Core Worker 24752: job 142 (pid=8419): Dormant child reaped
[1428620233] wproc: Core Worker 24748: job 143 (pid=8602) timed out. Killing it
[1428620233] wproc: CHECK job 143 from worker Core Worker 24748 timed out after 60.01s
[1428620233] wproc:   host=esu1l321; service=Check for read only File system on linux;
[1428620233] wproc:   early_timeout=1; exited_ok=0; wait_status=0; error_code=62;
[1428620233] Warning: Check of service 'Check for read only File system on linux' on host 'esu1l321' timed out after 60.013s!
[1428620233] wproc: Core Worker 24748: job 143 (pid=8602): Dormant child reaped
[1428620240] SERVICE ALERT: scj3-appadm01;Ping;CRITICAL;SOFT;1;CRITICAL - 11.48.37.140: rta nan, lost 100%
[1428620240] HP BSM Integration: Forwarding event for service 'Ping' (scj3-appadm01)...
[1428620242] wproc: Core Worker 24774: job 146 (pid=9496) timed out. Killing it
[1428620242] wproc: CHECK job 146 from worker Core Worker 24774 timed out after 60.01s
[1428620242] wproc:   host=esu2v653; service=Check for read only File system on linux;
[1428620242] wproc:   early_timeout=1; exited_ok=0; wait_status=0; error_code=62;
[1428620242] Warning: Check of service 'Check for read only File system on linux' on host 'esu2v653' timed out after 60.008s!
[1428620242] wproc: Core Worker 24774: job 146 (pid=9496): Dormant child reaped
[1428620242] wproc: Core Worker 24779: job 146 (pid=9621) timed out. Killing it
[1428620242] wproc: CHECK job 146 from worker Core Worker 24779 timed out after 60.01s
[1428620242] wproc:   host=esu4v110; service=Check for read only File system on linux;
[1428620242] wproc:   early_timeout=1; exited_ok=0; wait_status=0; error_code=62;
[1428620242] Warning: Check of service 'Check for read only File system on linux' on host 'esu4v110' timed out after 60.006s!
[1428620242] wproc: Core Worker 24779: job 146 (pid=9621): Dormant child reaped
[1428620242] wproc: Core Worker 24745: job 147 (pid=9743) timed out. Killing it
[1428620242] wproc: CHECK job 147 from worker Core Worker 24745 timed out after 60.01s
[1428620242] wproc:   host=esu4v128; service=Check for read only File system on linux;
[1428620242] wproc:   early_timeout=1; exited_ok=0; wait_status=0; error_code=62;
[1428620242] Warning: Check of service 'Check for read only File system on linux' on host 'esu4v128' timed out after 60.007s!
[1428620242] wproc: Core Worker 24745: job 147 (pid=9743): Dormant child reaped
[1428620243] wproc: Core Worker 24755: job 147 (pid=9769) timed out. Killing it
[1428620243] wproc: CHECK job 147 from worker Core Worker 24755 timed out after 60.01s
[1428620243] wproc:   host=esu4v133; service=Check for read only File system on linux;
[1428620243] wproc:   early_timeout=1; exited_ok=0; wait_status=0; error_code=62;
[1428620243] Warning: Check of service 'Check for read only File system on linux' on host 'esu4v133' timed out after 60.006s!
[1428620243] wproc: Core Worker 24755: job 147 (pid=9769): Dormant child reaped
[1428620246] wproc: Core Worker 24768: job 148 (pid=10422) timed out. Killing it
[1428620246] wproc: CHECK job 148 from worker Core Worker 24768 timed out after 60.01s
[1428620246] wproc:   host=ibm104p01; service=check_FS_space_AIX_by_sshpass;
[1428620246] wproc:   early_timeout=1; exited_ok=0; wait_status=0; error_code=62;
[1428620246] Warning: Check of service 'check_FS_space_AIX_by_sshpass' on host 'ibm104p01' timed out after 60.008s!
[1428620246] wproc: Core Worker 24768: job 148 (pid=10422): Dormant child reaped
[1428620247] wproc: Core Worker 24774: job 148 (pid=10558) timed out. Killing it
[1428620247] wproc: CHECK job 148 from worker Core Worker 24774 timed out after 60.01s
[1428620247] wproc:   host=ibm104p04; service=check_FS_space_AIX_by_sshpass;
[1428620247] wproc:   early_timeout=1; exited_ok=0; wait_status=0; error_code=62;
[1428620247] Warning: Check of service 'check_FS_space_AIX_by_sshpass' on host 'ibm104p04' timed out after 60.005s!
[1428620247] wproc: Core Worker 24774: job 148 (pid=10558): Dormant child reaped
[1428620250] HOST ALERT: scj3-appadm01;DOWN;SOFT;1;CRITICAL - 11.48.37.140: rta nan, lost 100%
[1428620250] HP BSM Integration: Forwarding event for host 'scj3-appadm01'...
[1428620250] wproc: Core Worker 24779: job 149 (pid=11077) timed out. Killing it
[1428620250] wproc: CHECK job 149 from worker Core Worker 24779 timed out after 60.01s
[1428620250] wproc:   host=ibm70p7; service=check_FS_space_AIX_by_sshpass;
[1428620250] wproc:   early_timeout=1; exited_ok=0; wait_status=0; error_code=62;
[1428620250] Warning: Check of service 'check_FS_space_AIX_by_sshpass' on host 'ibm70p7' timed out after 60.010s!
[1428620250] wproc: Core Worker 24779: job 149 (pid=11077): Dormant child reaped
[1428620251] wproc: Core Worker 24747: job 150 (pid=11197) timed out. Killing it
[1428620251] wproc: CHECK job 150 from worker Core Worker 24747 timed out after 60.01s
[1428620251] wproc:   host=ibm78p02; service=check_FS_space_AIX_by_sshpass;
[1428620251] wproc:   early_timeout=1; exited_ok=0; wait_status=0; error_code=62;
[1428620251] Warning: Check of service 'check_FS_space_AIX_by_sshpass' on host 'ibm78p02' timed out after 60.008s!
[1428620251] wproc: Core Worker 24747: job 150 (pid=11197): Dormant child reaped
[1428620251] wproc: Core Worker 24775: job 150 (pid=11204) timed out. Killing it
[1428620251] wproc: CHECK job 150 from worker Core Worker 24775 timed out after 60.01s
[1428620251] wproc:   host=ibm78p11; service=check_FS_space_AIX_by_sshpass;
[1428620251] wproc:   early_timeout=1; exited_ok=0; wait_status=0; error_code=62;
[1428620251] Warning: Check of service 'check_FS_space_AIX_by_sshpass' on host 'ibm78p11' timed out after 60.007s!
[1428620251] wproc: Core Worker 24775: job 150 (pid=11204): Dormant child reaped
[1428620297] wproc: Core Worker 24766: job 159 (pid=21904) timed out. Killing it
[1428620297] wproc: CHECK job 159 from worker Core Worker 24766 timed out after 60.01s
[1428620297] wproc:   host=scr1db02; service=check_FS_space_Solaris_by_sshpass;
[1428620297] wproc:   early_timeout=1; exited_ok=0; wait_status=0; error_code=62;
[1428620297] Warning: Check of service 'check_FS_space_Solaris_by_sshpass' on host 'scr1db02' timed out after 60.009s!
[1428620297] wproc: Core Worker 24766: job 159 (pid=21904): Dormant child reaped
[1428620297] SERVICE ALERT: scj3-appadm01;Ping;CRITICAL;SOFT;2;CRITICAL - 11.48.37.140: rta nan, lost 100%
[1428620307] HOST ALERT: scj3-appadm01;DOWN;SOFT;2;CRITICAL - 11.48.37.140: rta nan, lost 100%
[1428620316] wproc: Core Worker 24758: job 162 (pid=26873) timed out. Killing it
[1428620316] wproc: CHECK job 162 from worker Core Worker 24758 timed out after 60.01s
[1428620316] wproc:   host=ibmpci00p02; service=check_FS_space_AIX_by_sshpass;
[1428620316] wproc:   early_timeout=1; exited_ok=0; wait_status=0; error_code=62;
[1428620316] Warning: Check of service 'check_FS_space_AIX_by_sshpass' on host 'ibmpci00p02' timed out after 60.011s!
[1428620316] wproc: Core Worker 24758: job 162 (pid=26873): Dormant child reaped
[1428620321] wproc: Core Worker 24776: job 162 (pid=27858) timed out. Killing it
[1428620321] wproc: CHECK job 162 from worker Core Worker 24776 timed out after 60.01s
[1428620321] wproc:   host=esu2v221; service=Check for read only File system on linux;
[1428620321] wproc:   early_timeout=1; exited_ok=0; wait_status=0; error_code=62;
[1428620321] Warning: Check of service 'Check for read only File system on linux' on host 'esu2v221' timed out after 60.015s!
[1428620321] wproc: Core Worker 24776: job 162 (pid=27858): Dormant child reaped
[root@esu1l268 ~]#
JakeHatMacys
Posts: 281
Joined: Thu Sep 25, 2014 3:21 pm

Re: Nagios XI Configuration Random Configuration Error

Post by JakeHatMacys »

And just curious do you guys ever do any remote sessions? Would that be a viable option here? (I know we recently upgraded our support package to include Phone Support) While I was thinking about just rolling back to a good configuration snapshot I'd really like to find out what has this hosed up.

We were supposed to go live on this server Monday. If this were to happen while we were live in production it'd be a huge problem for us.

And to be clear Core is still running, so I'm assuming we're still functioning but XI is hosed:
Capture.JPG
You do not have the required permissions to view the files attached to this post.
tmcdonald
Posts: 9117
Joined: Mon Sep 23, 2013 8:40 am

Re: Nagios XI Configuration Random Configuration Error

Post by tmcdonald »

We certainly do perform remote sessions, but we usually only do those once a ticket has been opened (I do not recall anyone having done a remote for a forum thread). If you want to move this to a ticket, please email [email protected] and someone will be assigned to the ticket. Make sure to provide a descriptive email subject, and link to this thread in the body of the email.

If that is not something you want to do, please reply to this thread indicating such and we will continue here.
Former Nagios employee
Locked