Page 1 of 3
Nagios Stats Wizard Slow
Posted: Tue Aug 11, 2020 8:07 am
by andyb4u
Hi,
We recently upgraded two Nagios XI systems from Version 5-4-13 to Version 5-6-14.
We have service checks on each XI server generated using the Nagiostats Wizard. The config Wizard says it is up to date v1.2.3
One server seems to be very slow to run the checks.
For example if I run:
/usr/bin/php /usr/local/nagios/libexec/check_nagios_performance.php ahc15
On one server it can take 15-20 seconds to return.
If I run the same command on the other server it returns in 1 second.
I tried to compare the php.ini settings on each server and they are identical.
Are there any ways I can try and troubleshoot it's taking so long to run?
Re: Nagios Stats Wizard Slow
Posted: Tue Aug 11, 2020 6:40 pm
by ssax
Please send me a copy of your profile, you can download it from Admin > System Profile > Download Profile button from the system that is taking a long time.
Additionally, please send the output of these commands (as root) on that system:
- NOTE: You may need to adjust the -h 127.0.0.1, the -uroot, and -pnagiosxi in the first command if your DB is offloaded to another server and/or you've changed the root mysql password
Code: Select all
echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -h 127.0.0.1 -uroot -pnagiosxi --table
This next command may fail, that's okay, not all systems have postgresql:
Code: Select all
echo "SELECT relname as Table, pg_size_pretty(pg_total_relation_size(relid)) As Size, pg_size_pretty(pg_total_relation_size(relid) - pg_relation_size(relid)) as ExternalSize FROM pg_catalog.pg_statio_user_tables ORDER BY pg_total_relation_size(relid) DESC;" | psql nagiosxi nagiosxi
Re: Nagios Stats Wizard Slow
Posted: Wed Aug 12, 2020 7:36 am
by andyb4u
Hi,
I've PM'd the profile and MySQL query results to you. We don't use postgresql.
Re: Nagios Stats Wizard Slow
Posted: Wed Aug 12, 2020 2:23 pm
by ssax
The mysql results aren't attached to the PM, please resend them. Profile received.
Re: Nagios Stats Wizard Slow
Posted: Wed Aug 12, 2020 2:27 pm
by ssax
One thing I see from the profile is that you have you have multiple ad/ldap components installed (which will need to be remediated).
You will need to do this:
First, set nagiosadmin to a local account (if it isn't already) and logout/log back into the system with it.
Then, disable active directory authentication in Admin > Manage Components > Active Directory > Settings, double check Admin > Manage Components > LDAP Auth > Settings is disabled as well.
Then setup the new component (copying the info from the old) in Admin > LDAP / AD Integration.
-- See below if you have questions
https://assets.nagios.com/downloads/nag ... ios-XI.pdf
See here as well if you have issues:
https://support.nagios.com/kb/article/a ... n-600.html
Once you've done that and tested login functionality you can run these commands to get rid of the old components (
which can cause slowness):
Code: Select all
rm -rf /usr/local/nagiosxi/html/includes/components/active_directory
rm -rf /usr/local/nagiosxi/html/includes/components/ldapauth
Re: Nagios Stats Wizard Slow
Posted: Thu Aug 13, 2020 2:39 am
by andyb4u
Hi,
I've resent the output of the MySQL query.
We currently don't use any Active Directory integration. We only use local accounts.
I inherited this system with those two components installed but they are both disabled. Could they still be causing issues if they are disabled?
The other XI server which isn't experiencing the slowness has these 2 components installed and disabled too.
Re: Nagios Stats Wizard Slow
Posted: Thu Aug 13, 2020 4:47 pm
by ssax
They should not exist on your XI systems running XI 5+, please remove them:
Code: Select all
rm -rf /usr/local/nagiosxi/html/includes/components/active_directory
rm -rf /usr/local/nagiosxi/html/includes/components/ldapauth
Does this take 15 to 20 seconds when running this command directly? I want to see if it's something in the PHP script or if it's the program it's calling:
Code: Select all
/usr/local/nagios/bin/nagiostats -m -D ';' -d NUMOACTHSTCHECKS15M,NUMSACTHSTCHECKS15M,NUMACTHSTCHECKS15MOK
How many total hosts/services do they have on each system?
Do you have high IO wait on the system it takes longer on?
- See the bold column from the top command output
[root@xid ~]# top
top - 16:42:56 up 13 min, 1 user, load average: 0.19, 0.39, 0.30
Tasks: 150 total, 2 running, 148 sleeping, 0 stopped, 0 zombie
%Cpu(s): 48.4 us, 0.0 sy, 0.0 ni, 51.6 id,
0.0 wa, 0.0 hi, 0.0 si, 0.0 st
Re: Nagios Stats Wizard Slow
Posted: Fri Aug 14, 2020 3:20 am
by andyb4u
Hi,
I will get those two components removed. It just has to go through a change process first.
I did find that running:
Code: Select all
/usr/local/nagios/bin/nagiostats -m -D ';' -d NUMOACTHSTCHECKS15M,NUMSACTHSTCHECKS15M,NUMACTHSTCHECKS15MOK
Returns straight away on the system which is having the slowness issue with the PHP script.
The system that is slow in returning the PHP script has:
548 hosts
3998 services
The system that is not slow in returning the PHP script has:
1547 hosts
13914 services
The IO wait is around 0.1 on the system that is slow to return the PHP script:
top - 09:01:48 up 16 days, 18:38, 1 user, load average: 0.30, 0.56, 0.70
Tasks: 311 total, 2 running, 309 sleeping, 0 stopped, 0 zombie
%Cpu(s): 7.5 us, 1.0 sy, 0.0 ni, 91.4 id,
0.0 wa, 0.0 hi, 0.1 si, 0.0 st
KiB Mem : 16266188 total, 1740748 free, 1283812 used, 13241628 buff/cache
KiB Swap: 5177340 total, 5102844 free, 74496 used. 13818580 avail Mem
Re: Nagios Stats Wizard Slow
Posted: Fri Aug 14, 2020 4:27 pm
by ssax
Looking at the PHP code it doesn't do much but calling the command and parsing it, what are your /etc/php.ini settings for memory_limit?
Re: Nagios Stats Wizard Slow
Posted: Mon Aug 17, 2020 1:48 am
by andyb4u
It's the same on both systems:
memory_limit = 512M