GUI Issues - Applying updates and System Status info

[gazoo1997]

To whom it may concern:

Since upgrading to Nagios XI 5.2.3 I have been having several issues with the GUI. Two of the most pressing are:

1. Applying updates from the GUI - This fails with an array of issues (running reconfigure_nagios.sh works just fine and finds no configuration issues)
2. System Status shows "No Data" for XI System Component Status and Server Statistics. Hover over "Green Check" icon displays "Could not Read Program Data".

We have repaired the databases with no issues and are at a loss as to what to do next.

I have combined the issues in one topic because I feel that they are somehow related due to the timing of when they were discovered.

Configuration:
RHEL6.7
64BIT
Manual Install of Nagios XI 5.2.3
No special configurations

[lmiltchev]

1. Applying updates from the GUI - This fails with an array of issues (running reconfigure_nagios.sh works just fine and finds no configuration issues)

When you say "Applying updates from the GUI..." you mean modifying objects in the CCM, and applying configuration, correct?

2. System Status shows "No Data" for XI System Component Status and Server Statistics. Hover over "Green Check" icon displays "Could not Read Program Data".

Can you show us screenshots of these errors in the GUI, and a tail of the mysql log, and the apache error log in code wraps?

Code: Select all

tail -100 /var/log/mysqld.log
tail -10  /var/log/httpd/error_log

[gazoo1997]

When you say "Applying updates from the GUI..." you mean modifying objects in the CCM, and applying configuration, correct?

Yes that is correct.

Can you show us screenshots of these errors in the GUI, and a tail of the mysql log, and the apache error log in code wraps?

Here are the screen shots and logs.

CheckConfiguration-CommandLine.png

httpd_error_log.rtf

See next reply for the rest of the attachments.

[gazoo1997]

mysqld.log

servicedependencies.rtf

[gazoo1997]

ApplyConfiguration-GUI.png

ProgramData.png

[dwhitfield]

You have many errors about crashed tables in the mysqd.log, so let's start with repairing your database.

Code: Select all

/usr/local/nagiosxi/scripts/repair_databases.sh

After the database is repaired, check to see if you can run "reconfigure_nagios.sh" from the CLI successfully:

Code: Select all

cd /usr/local/nagiosxi/scripts
./reconfigure_nagios.sh

[ssax]

In addition to lmiltchev's post, please attach your /etc/sudoers file and also the output of these commands:

Code: Select all

ps aux | grep nagios.cfg
chage -l nagios

Did you do anything else with the upgrade like setup SSL or change any of the apache configs at all (hardening/ciphers)?

Thank you

[gazoo1997]

We have attempted the repair_databases.sh script many times recently to try and resolve this. I ran it again and captured the mysqld.log again during the run as well as ran a reconfigure_nagios.sh.

160301 13:52:10 [Note] /usr/libexec/mysqld: Normal shutdown

160301 13:52:10 [Note] Event Scheduler: Purging the queue. 0 events
160301 13:52:12 InnoDB: Starting shutdown...
160301 13:52:15 InnoDB: Shutdown completed; log sequence number 0 44233
160301 13:52:15 [Note] /usr/libexec/mysqld: Shutdown complete

160301 13:52:15 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
160301 13:54:04 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
160301 13:54:04 InnoDB: Initializing buffer pool, size = 8.0M
160301 13:54:04 InnoDB: Completed initialization of buffer pool
160301 13:54:04 InnoDB: Started; log sequence number 0 44233
160301 13:54:04 [Note] Event Scheduler: Loaded 0 events
160301 13:54:04 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
160301 13:54:05 [Note] /usr/libexec/mysqld: Normal shutdown

160301 13:54:05 [Note] Event Scheduler: Purging the queue. 0 events
160301 13:54:07 InnoDB: Starting shutdown...
160301 13:54:09 InnoDB: Shutdown completed; log sequence number 0 44233
160301 13:54:09 [Note] /usr/libexec/mysqld: Shutdown complete

160301 13:54:09 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
160301 13:54:11 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
160301 13:54:11 InnoDB: Initializing buffer pool, size = 8.0M
160301 13:54:11 InnoDB: Completed initialization of buffer pool
160301 13:54:11 InnoDB: Started; log sequence number 0 44233
160301 13:54:11 [Note] Event Scheduler: Loaded 0 events
160301 13:54:11 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
160301 13:54:12 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
160301 13:54:12 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
160301 13:54:14 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
160301 13:54:14 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
160301 13:54:14 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
160301 13:54:14 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
160301 13:54:14 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
160301 13:54:14 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
160301 13:54:14 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
160301 13:54:14 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
160301 13:54:15 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
160301 13:54:15 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
160301 13:54:15 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired


_____________________________________________________________________________________________________________________________________________



WRITE CONFIGS SUCCESSFUL!
OUTPUT:
Nagios Core 4.1.1
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 08-19-2015
License: GPL

Website: https://www.nagios.org
Reading configuration data...
Read main config file okay...
Read object config files okay...

Running pre-flight check on configuration data...

Checking objects...
Checked 2062 services.
Checked 267 hosts.
Checked 9 host groups.
Checked 3 service groups.
Checked 10 contacts.
Checked 3 contact groups.
Checked 135 commands.
Checked 17 time periods.
Checked 0 host escalations.
Checked 0 service escalations.
Checking for circular paths...
Checked 267 hosts
Checked 0 service dependencies
Checked 0 host dependencies
Checked 17 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 0
Total Errors: 0

Things look okay - No serious problems were detected during the pre-flight check
RET: 0
Running configuration check...
Stopping nagios:. done.
Starting nagios: done.

[gazoo1997]

This was not an upgrade but was a fresh install of 5.2.3 and the configuration has not changed since. At one point all of this was working correctly.

ps aux | grep nagios.cfg
nagios 686 0.8 0.0 42352 7744 ? Ss 13:38 0:11 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios 705 0.0 0.0 42024 6080 ? S 13:38 0:00 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
root 3830 0.0 0.0 103312 908 pts/3 S+ 13:59 0:00 grep nagios.cfg
nagios 19526 1.5 0.0 38284 7124 ? Ss 13:55 0:03 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios 19969 0.0 0.0 37768 6060 ? S 13:55 0:00 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg

chage -l nagios
Last password change : Nov 17, 2015
Password expires : May 15, 2016
Password inactive : never
Account expires : never
Minimum number of days between password change : 7
Maximum number of days between password change : 180
Number of days of warning before password expires : 7

# This file is managed by Chef.
# Do NOT modify this file directly.

Defaults !visiblepw
Defaults env_reset
Defaults env_keep = "COLORS DISPLAY HOSTNAME HISTSIZE INPUTRC KDEDIR LS_COLORS"
Defaults env_keep += "MAIL PS1 PS2 QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE"
Defaults env_keep += "LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES"
Defaults env_keep += "LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE"
Defaults env_keep += "LC_TIME LC_ALL LANGUAGE LINGUAS _XKB_CHARSET XAUTHORITY"
Defaults env_keep += "HOME"
Defaults always_set_home
Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin

# User privilege specification
root ALL=(ALL) ALL


ec2-user ALL=(ALL) NOPASSWD:ALL

# Members of the group 'admin' may gain root privileges
%admin ALL=(ALL) NOPASSWD:ALL
# Members of the group 'wheel' may gain root privileges
%wheel ALL=(ALL) NOPASSWD:ALL
# Members of the group 'Linux\ Admin' may gain root privileges
%Linux\ Admin ALL=(ALL) NOPASSWD:ALL
# Members of the group 'Linux\ Service' may gain root privileges
%Linux\ Service ALL=(ALL) NOPASSWD:ALL
# This file is managed by Chef.
# Do NOT modify this file directly.

Defaults !visiblepw
Defaults env_reset
Defaults env_keep = "COLORS DISPLAY HOSTNAME HISTSIZE INPUTRC KDEDIR LS_COLORS"
Defaults env_keep += "MAIL PS1 PS2 QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE"
Defaults env_keep += "LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES"
Defaults env_keep += "LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE"
Defaults env_keep += "LC_TIME LC_ALL LANGUAGE LINGUAS _XKB_CHARSET XAUTHORITY"
Defaults env_keep += "HOME"
Defaults always_set_home
Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin

# User privilege specification
root ALL=(ALL) ALL


ec2-user ALL=(ALL) NOPASSWD:ALL

# Members of the group 'admin' may gain root privileges
%admin ALL=(ALL) NOPASSWD:ALL
# Members of the group 'wheel' may gain root privileges
%wheel ALL=(ALL) NOPASSWD:ALL
# Members of the group 'Linux\ Admin' may gain root privileges
%Linux\ Admin ALL=(ALL) NOPASSWD:ALL
# Members of the group 'Linux\ Service' may gain root privileges
%Linux\ Service ALL=(ALL) NOPASSWD:ALL

[ssax]

Even though it's not the problem, let's get rid of the password expiration on the nagios account so it doesn't become one:

Code: Select all

passwd nagios
passwd -d nagios
chage -I -1 -m 0 -M 99999 -E -1 nagios

Your /etc/sudoers file is also not proper, please add these to the bottom:

Code: Select all

User_Alias      NAGIOSXI=nagios
User_Alias              NAGIOSXIWEB=apache
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios start
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios stop
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios restart
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios reload
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios status
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios checkconfig
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db start
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db stop
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db restart
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db reload
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db status
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd start
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd stop
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd restart
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd reload
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd status
NAGIOSXI ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/html/includes/components/autodiscovery/scripts/autodiscover_new.php *
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/html/includes/components/profile/getprofile.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/upgrade_to_latest.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/change_timezone.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/manage_services.sh *
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/reset_config_perms.sh
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/messages
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/httpd/error_log
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/mysqld.log
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/html/includes/components/autodiscovery/scripts/autodiscover_new.php *
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/html/includes/components/profile/getprofile.sh
NAGIOSXIWEB ALL = NOPASSWD:/etc/init.d/snmptt restart
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/scripts/repair_databases.sh
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/scripts/manage_services.sh *

You also have too many nagios processes running, after fixing the above problems, run these commands:

Code: Select all

service nagios stop
killall -9 nagios
service nagios start
service httpd restart

Let us know if that resolves the issue.