Nagios Stats Wizard Slow

[andyb4u]

Hi,

We recently upgraded two Nagios XI systems from Version 5-4-13 to Version 5-6-14.

We have service checks on each XI server generated using the Nagiostats Wizard. The config Wizard says it is up to date v1.2.3

One server seems to be very slow to run the checks.

For example if I run:

/usr/bin/php /usr/local/nagios/libexec/check_nagios_performance.php ahc15

On one server it can take 15-20 seconds to return.

If I run the same command on the other server it returns in 1 second.

I tried to compare the php.ini settings on each server and they are identical.

Are there any ways I can try and troubleshoot it's taking so long to run?

[ssax]

Please send me a copy of your profile, you can download it from Admin > System Profile > Download Profile button from the system that is taking a long time.

Additionally, please send the output of these commands (as root) on that system:
- NOTE: You may need to adjust the -h 127.0.0.1, the -uroot, and -pnagiosxi in the first command if your DB is offloaded to another server and/or you've changed the root mysql password

Code: Select all

echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -h 127.0.0.1 -uroot -pnagiosxi --table

This next command may fail, that's okay, not all systems have postgresql:

Code: Select all

echo "SELECT relname as Table, pg_size_pretty(pg_total_relation_size(relid)) As Size, pg_size_pretty(pg_total_relation_size(relid) - pg_relation_size(relid)) as ExternalSize FROM pg_catalog.pg_statio_user_tables ORDER BY pg_total_relation_size(relid) DESC;" | psql nagiosxi nagiosxi

[andyb4u]

Hi,

I've PM'd the profile and MySQL query results to you. We don't use postgresql.

[ssax]

The mysql results aren't attached to the PM, please resend them. Profile received.

[ssax]

One thing I see from the profile is that you have you have multiple ad/ldap components installed (which will need to be remediated).

You will need to do this:

First, set nagiosadmin to a local account (if it isn't already) and logout/log back into the system with it.

Then, disable active directory authentication in Admin > Manage Components > Active Directory > Settings, double check Admin > Manage Components > LDAP Auth > Settings is disabled as well.

Then setup the new component (copying the info from the old) in Admin > LDAP / AD Integration.
-- See below if you have questions

https://assets.nagios.com/downloads/nag ... ios-XI.pdf

See here as well if you have issues:

https://support.nagios.com/kb/article/a ... n-600.html

Once you've done that and tested login functionality you can run these commands to get rid of the old components (which can cause slowness):

Code: Select all

rm -rf /usr/local/nagiosxi/html/includes/components/active_directory
rm -rf /usr/local/nagiosxi/html/includes/components/ldapauth

[andyb4u]

Hi,

I've resent the output of the MySQL query.

We currently don't use any Active Directory integration. We only use local accounts.

I inherited this system with those two components installed but they are both disabled. Could they still be causing issues if they are disabled?

The other XI server which isn't experiencing the slowness has these 2 components installed and disabled too.

[ssax]

They should not exist on your XI systems running XI 5+, please remove them:

Code: Select all

rm -rf /usr/local/nagiosxi/html/includes/components/active_directory
rm -rf /usr/local/nagiosxi/html/includes/components/ldapauth

Does this take 15 to 20 seconds when running this command directly? I want to see if it's something in the PHP script or if it's the program it's calling:

Code: Select all

/usr/local/nagios/bin/nagiostats -m -D ';' -d NUMOACTHSTCHECKS15M,NUMSACTHSTCHECKS15M,NUMACTHSTCHECKS15MOK

How many total hosts/services do they have on each system?

Do you have high IO wait on the system it takes longer on?
- See the bold column from the top command output

[root@xid ~]# top
top - 16:42:56 up 13 min, 1 user, load average: 0.19, 0.39, 0.30
Tasks: 150 total, 2 running, 148 sleeping, 0 stopped, 0 zombie
%Cpu(s): 48.4 us, 0.0 sy, 0.0 ni, 51.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st

[andyb4u]

Hi,

I will get those two components removed. It just has to go through a change process first.

I did find that running:

Code: Select all

/usr/local/nagios/bin/nagiostats -m -D ';' -d NUMOACTHSTCHECKS15M,NUMSACTHSTCHECKS15M,NUMACTHSTCHECKS15MOK

Returns straight away on the system which is having the slowness issue with the PHP script.

The system that is slow in returning the PHP script has:
548 hosts
3998 services

The system that is not slow in returning the PHP script has:
1547 hosts
13914 services

The IO wait is around 0.1 on the system that is slow to return the PHP script:

top - 09:01:48 up 16 days, 18:38, 1 user, load average: 0.30, 0.56, 0.70
Tasks: 311 total, 2 running, 309 sleeping, 0 stopped, 0 zombie
%Cpu(s): 7.5 us, 1.0 sy, 0.0 ni, 91.4 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 st
KiB Mem : 16266188 total, 1740748 free, 1283812 used, 13241628 buff/cache
KiB Swap: 5177340 total, 5102844 free, 74496 used. 13818580 avail Mem

[ssax]

Looking at the PHP code it doesn't do much but calling the command and parsing it, what are your /etc/php.ini settings for memory_limit?

[andyb4u]

It's the same on both systems:

memory_limit = 512M