Nagios XI slowness

[RIDS_I2MP]

Hello,

We haven't added the space yet. We need to add it. below is the output:

[root@vmaz-nagiosxi ~]# echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -h 127.0.0.1 -uroot -pnagiosxi --table
mysql: [Warning] Using a password on the command line interface can be insecure.
+--------------------------------------------+------------+
| Table | Size in MB |
+--------------------------------------------+------------+
| nagios_acknowledgements | 0.00 |
| nagios_commands | 0.03 |
| nagios_commenthistory | 1.93 |
| nagios_comments | 0.00 |
| nagios_configfiles | 0.01 |
| nagios_configfilevariables | 0.01 |
| nagios_contact_addresses | 0.00 |
| nagios_contact_notificationcommands | 0.01 |
| nagios_contactgroup_members | 0.00 |
| nagios_contactgroups | 0.00 |
| nagios_contactnotificationmethods | 0.77 |
| nagios_contactnotifications | 0.54 |
| nagios_contacts | 0.00 |
| nagios_contactstatus | 0.00 |
| nagios_customvariables | 1.31 |
| nagios_customvariablestatus | 1.32 |
| nagios_dbversion | 0.00 |
| nagios_downtimehistory | 0.00 |
| nagios_eventhandlers | 0.48 |
| nagios_externalcommands | 0.00 |
| nagios_flappinghistory | 0.35 |
| nagios_host_contactgroups | 0.00 |
| nagios_host_contacts | 0.00 |
| nagios_host_parenthosts | 0.00 |
| nagios_hostchecks | 9.07 |
| nagios_hostdependencies | 0.00 |
| nagios_hostescalation_contactgroups | 0.00 |
| nagios_hostescalation_contacts | 0.00 |
| nagios_hostescalations | 0.00 |
| nagios_hostgroup_members | 0.00 |
| nagios_hostgroups | 0.00 |
| nagios_hosts | 1.40 |
| nagios_hoststatus | 2.53 |
| nagios_instances | 0.00 |
| nagios_logentries | 2166.60 |
| nagios_notifications | 1600.06 |
| nagios_objects | 2.47 |
| nagios_processevents | 0.29 |
| nagios_programstatus | 0.00 |
| nagios_runtimevariables | 0.00 |
| nagios_scheduleddowntime | 0.00 |
| nagios_service_contactgroups | 0.00 |
| nagios_service_contacts | 0.02 |
| nagios_service_parentservices | 0.00 |
| nagios_servicechecks | 5.18 |
| nagios_servicedependencies | 0.00 |
| nagios_serviceescalation_contactgroups | 0.00 |
| nagios_serviceescalation_contacts | 0.00 |
| nagios_serviceescalations | 0.00 |
| nagios_servicegroup_members | 0.01 |
| nagios_servicegroups | 0.00 |
| nagios_services | 5.12 |
| nagios_servicestatus | 10.63 |
| nagios_statehistory | 42.93 |
| nagios_systemcommands | 0.06 |
| nagios_timedeventqueue | 0.00 |
| nagios_timedevents | 0.00 |
| nagios_timeperiod_timeranges | 0.01 |
| nagios_timeperiods | 0.00 |
| tbl_command | 0.06 |
| tbl_contact | 0.03 |
| tbl_contactgroup | 0.03 |
| tbl_contacttemplate | 0.03 |
| tbl_domain | 0.03 |
| tbl_host | 0.63 |
| tbl_hostdependency | 0.03 |
| tbl_hostescalation | 0.03 |
| tbl_hostextinfo | 0.03 |
| tbl_hostgroup | 0.03 |
| tbl_hosttemplate | 0.03 |
| tbl_info | 0.17 |
| tbl_lnkContactToCommandHost | 0.02 |
| tbl_lnkContactToCommandService | 0.02 |
| tbl_lnkContactToContactgroup | 0.02 |
| tbl_lnkContactToContacttemplate | 0.02 |
| tbl_lnkContactToVariabledefinition | 0.02 |
| tbl_lnkContactgroupToContact | 0.02 |
| tbl_lnkContactgroupToContactgroup | 0.02 |
| tbl_lnkContacttemplateToCommandHost | 0.02 |
| tbl_lnkContacttemplateToCommandService | 0.02 |
| tbl_lnkContacttemplateToContactgroup | 0.02 |
| tbl_lnkContacttemplateToContacttemplate | 0.02 |
| tbl_lnkContacttemplateToVariabledefinition | 0.02 |
| tbl_lnkHostToContact | 0.02 |
| tbl_lnkHostToContactgroup | 0.02 |
| tbl_lnkHostToHost | 0.02 |
| tbl_lnkHostToHostgroup | 0.02 |
| tbl_lnkHostToHosttemplate | 0.19 |
| tbl_lnkHostToVariabledefinition | 0.23 |
| tbl_lnkHostdependencyToHost_DH | 0.02 |
| tbl_lnkHostdependencyToHost_H | 0.02 |
| tbl_lnkHostdependencyToHostgroup_DH | 0.02 |
| tbl_lnkHostdependencyToHostgroup_H | 0.02 |
| tbl_lnkHostescalationToContact | 0.02 |
| tbl_lnkHostescalationToContactgroup | 0.02 |
| tbl_lnkHostescalationToHost | 0.02 |
| tbl_lnkHostescalationToHostgroup | 0.02 |
| tbl_lnkHostgroupToHost | 0.02 |
| tbl_lnkHostgroupToHostgroup | 0.02 |
| tbl_lnkHosttemplateToContact | 0.02 |
| tbl_lnkHosttemplateToContactgroup | 0.02 |
| tbl_lnkHosttemplateToHost | 0.02 |
| tbl_lnkHosttemplateToHostgroup | 0.02 |
| tbl_lnkHosttemplateToHosttemplate | 0.02 |
| tbl_lnkHosttemplateToVariabledefinition | 0.02 |
| tbl_lnkServiceToContact | 0.02 |
| tbl_lnkServiceToContactgroup | 0.02 |
| tbl_lnkServiceToHost | 0.48 |
| tbl_lnkServiceToHostgroup | 0.02 |
| tbl_lnkServiceToServicegroup | 0.02 |
| tbl_lnkServiceToServicetemplate | 1.52 |
| tbl_lnkServiceToVariabledefinition | 0.44 |
| tbl_lnkServicedependencyToHost_DH | 0.02 |
| tbl_lnkServicedependencyToHost_H | 0.02 |
| tbl_lnkServicedependencyToHostgroup_DH | 0.02 |
| tbl_lnkServicedependencyToHostgroup_H | 0.02 |
| tbl_lnkServicedependencyToService_DS | 0.02 |
| tbl_lnkServicedependencyToService_S | 0.02 |
| tbl_lnkServicedependencyToServicegroup_DS | 0.02 |
| tbl_lnkServicedependencyToServicegroup_S | 0.02 |
| tbl_lnkServiceescalationToContact | 0.02 |
| tbl_lnkServiceescalationToContactgroup | 0.02 |
| tbl_lnkServiceescalationToHost | 0.02 |
| tbl_lnkServiceescalationToHostgroup | 0.02 |
| tbl_lnkServiceescalationToService | 0.02 |
| tbl_lnkServiceescalationToServicegroup | 0.02 |
| tbl_lnkServicegroupToService | 0.02 |
| tbl_lnkServicegroupToServicegroup | 0.02 |
| tbl_lnkServicetemplateToContact | 0.02 |
| tbl_lnkServicetemplateToContactgroup | 0.02 |
| tbl_lnkServicetemplateToHost | 0.02 |
| tbl_lnkServicetemplateToHostgroup | 0.02 |
| tbl_lnkServicetemplateToServicegroup | 0.02 |
| tbl_lnkServicetemplateToServicetemplate | 0.02 |
| tbl_lnkServicetemplateToVariabledefinition | 0.02 |
| tbl_lnkTimeperiodToTimeperiod | 0.02 |
| tbl_logbook | 0.02 |
| tbl_mainmenu | 0.02 |
| tbl_permission | 0.02 |
| tbl_permission_inactive | 0.02 |
| tbl_service | 3.52 |
| tbl_servicedependency | 0.03 |
| tbl_serviceescalation | 0.03 |
| tbl_serviceextinfo | 0.03 |
| tbl_servicegroup | 0.03 |
| tbl_servicetemplate | 0.03 |
| tbl_session | 0.02 |
| tbl_session_locks | 0.02 |
| tbl_settings | 0.03 |
| tbl_submenu | 0.02 |
| tbl_timedefinition | 0.02 |
| tbl_timeperiod | 0.03 |
| tbl_user | 0.03 |
| tbl_variabledefinition | 1.52 |
| xi_auditlog | 75.64 |
| xi_auth_tokens | 0.03 |
| xi_cmp_ccm_backups | 0.02 |
| xi_cmp_favorites | 0.03 |
| xi_cmp_nagiosbpi_backups | 0.02 |
| xi_cmp_trapdata | 0.03 |
| xi_cmp_trapdata_log | 0.03 |
| xi_commands | 0.02 |
| xi_deploy_agents | 0.02 |
| xi_deploy_jobs | 0.02 |
| xi_eventqueue | 0.03 |
| xi_events | 0.52 |
| xi_meta | 6.16 |
| xi_mibs | 0.05 |
| xi_options | 0.03 |
| xi_sessions | 0.03 |
| xi_sysstat | 0.03 |
| xi_usermeta | 0.08 |
| xi_users | 0.03 |
+--------------------------------------------+------------+
[root@vmaz-nagiosxi ~]#



We can see many files consuming the data in below path:

[root@vmaz-nagiosxi AE-DIP-WOJOECOM-AP]# du -sh /usr/local/nagios/share/perfdata
11G /usr/local/nagios/share/perfdata
[root@vmaz-nagiosxi AE-DIP-WOJOECOM-AP]# cd /usr/local/nagios/share/perfdata
[root@vmaz-nagiosxi perfdata]# ll | wc -l
3100
[root@vmaz-nagiosxi perfdata]#


drwxrwxr-x. 2 nagios nagios 74 Aug 13 15:14 AE-D3-B7-LVMH-AP1
drwxrwxr-x. 2 nagios nagios 74 Aug 13 15:14 KW-360-SWA-AP01
drwxrwxr-x. 2 nagios nagios 74 Aug 13 15:14 SA-JED-5F-AP06
drwxrwxr-x. 2 nagios nagios 4096 Aug 13 15:14 LB-LDC-VPN-GW
drwxrwxr-x. 2 nagios nagios 74 Aug 13 15:14 AE-DFC-WOJ-AP01
drwxrwxr-x. 2 nagios nagios 74 Aug 13 15:14 AE-DM-IlGufo-AP1
drwxrwxr-x. 2 nagios nagios 74 Aug 13 15:14 AE-DIP-WOJOECOM-AP
drwxrwxr-x. 2 nagios nagios 74 Aug 13 15:14 KW-GATE-LOCCITANE-AP
[root@vmaz-nagiosxi perfdata]#


[root@vmaz-nagiosxi perfdata]# cd AE-DIP-WOJOECOM-AP
[root@vmaz-nagiosxi AE-DIP-WOJOECOM-AP]# ls -lrt
total 3012
-rw-rw-r--. 1 nagios nagios 4176 Aug 13 14:48 Ping.xml
-rw-rw-r--. 1 nagios nagios 1534768 Aug 13 14:48 Ping.rrd
-rw-rw-r--. 1 nagios nagios 4057 Aug 13 15:14 _HOST_.xml
-rw-rw-r--. 1 nagios nagios 1534768 Aug 13 15:14 _HOST_.rrd
[root@vmaz-nagiosxi AE-DIP-WOJOECOM-AP]#


Can we move those to "nagiosmon" partition.

[root@vmaz-nagiosxi perfdata]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 7.9G 0 7.9G 0% /dev
tmpfs 7.9G 0 7.9G 0% /dev/shm
tmpfs 7.9G 834M 7.1G 11% /run
tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup
/dev/sdb2 30G 25G 4.7G 85% /
/dev/sdb1 496M 109M 387M 22% /boot
/dev/sdb15 495M 6.8M 488M 2% /boot/efi
/dev/sdc1 32G 49M 30G 1% /mnt/resource
/dev/mapper/VG_MAIN-lv_nagios 55G 1.3G 51G 3% /nagiosmon
tmpfs 1.6G 0 1.6G 0% /run/user/1004
tmpfs 1.6G 0 1.6G 0% /run/user/1005
tmpfs 1.6G 0 1.6G 0% /run/user/1000
[root@vmaz-nagiosxi perfdata]#

[benjaminsmith]

Hi,

Did you notice any performance improvement after running the MySQL check and repair command? Please let me know. The table sizes from the SQL command do look fine. Can you post the output of the following command:

Code: Select all

echo "SELECT SCHEMA_NAME 'database', default_character_set_name 'charset', DEFAULT_COLLATION_NAME 'collation' FROM information_schema.SCHEMATA;" |mysql -t -u root -pnagiosxi

Also, how many hosts and services do you have on this system? The profile was missing this data, likely due to the db issues.

Let's get disk size increased on this system since running out of the space likely resulted in the database issues.

There maybe old RRD files from previous checks that are no longer need on the server. I would make a backup, and delete the old files. We have a script available to help with this process.

Nagios XI - Mass Delete Old RRD Files

Also the check the backup directory for old archives that are no longer needed, these can be quite large since the backup process will save everything.

Code: Select all

/store/backups/nagiosxi/

More tips for saving space on the following page.

Nagios XI - Running Out Of Disk Space On The Nagios XI VM

[RIDS_I2MP]

Hello,

Yes, we have added disk space now. Below is the output:

[root@vmaz-nagiosxi ~]# echo "SELECT SCHEMA_NAME 'database', default_character_set_name 'charset', DEFAULT_COLLATION_NAME 'collation' FROM information_schema.SCHEMATA;" |mysql -t -u root -pnagiosxi
mysql: [Warning] Using a password on the command line interface can be insecure.
+--------------------+---------+--------------------+
| database | charset | collation |
+--------------------+---------+--------------------+
| mysql | utf8mb4 | utf8mb4_0900_ai_ci |
| information_schema | utf8 | utf8_general_ci |
| performance_schema | utf8mb4 | utf8mb4_0900_ai_ci |
| sys | utf8mb4 | utf8mb4_0900_ai_ci |
| nagios | utf8 | utf8_general_ci |
| nagiosql | utf8 | utf8_general_ci |
| nagiosxi | utf8mb4 | utf8mb4_0900_ai_ci |
+--------------------+---------+--------------------+
[root@vmaz-nagiosxi ~]#


[root@vmaz-nagiosxi ~]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 7.9G 0 7.9G 0% /dev
tmpfs 7.9G 0 7.9G 0% /dev/shm
tmpfs 7.9G 221M 7.7G 3% /run
tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup
/dev/sdc2 158G 27G 131G 17% /
/dev/sdc1 496M 109M 387M 22% /boot
/dev/sdc15 495M 6.8M 488M 2% /boot/efi
/dev/mapper/VG_MAIN-lv_nagios 55G 1.3G 51G 3% /nagiosmon
/dev/sda1 32G 49M 30G 1% /mnt/resource
tmpfs 1.6G 0 1.6G 0% /run/user/1005
tmpfs 1.6G 0 1.6G 0% /run/user/1004
[root@vmaz-nagiosxi ~]#

Also, we will try deleting the old RRD files. I will update you on that.
For now, the slowness is same. We have added diskspace few hours back so maybe it will be fine after some time.

[benjaminsmith]

Hi,

Also, we will try deleting the old RRD files. I will update you on that.
For now, the slowness is same. We have added diskspace few hours back so maybe it will be fine after some time.

Sounds good. Let us know you notice any improvement.

Also, open up the following file:

Code: Select all

/usr/local/nagiosxi/html/config.inc.php

..and make sure ndo is set to UTF8 charset and restart the Apache webserver.

Code: Select all

    "ndoutils" => array(
        "dbtype" => 'mysql',
        "dbserver" => 'localhost',
        "user" => 'ndoutils',
        "db" => 'nagios',
        "charset" => "utf8",

Reststart httpd:

Code: Select all

systemctl restart httpd

[RIDS_I2MP]

Hello,

We have made the changes as per your suggestion and performance is somewhat okay now.
Still, we are facing issues with notifications. It takes 15-20 minutes for the notification data to load. Sometimes it doesn't even loads the data.

Other thing is, few services that we configured almost a week back are still in pending status. We have two Nagios XI instances/nodes. Both having the same issues. I am attaching pending service screenshot here.

[ssax]

Attach a fresh copy of your profile.

I think you're likely hitting a known issue:

There is a known issue on some systems running XI 5.7.2 where the host/service statuses don't update properly, there isn't currently a fix for this, development is actively working the issue and a fix is expected in XI 5.7.3.

As a workaround, please do this to downgrade NDO3 to NDO2DB (they are entirely different) and it should resolve your issues:

Take an XI backup first just in case:

https://assets.nagios.com/downloads/nag ... ios-XI.pdf

Code: Select all

systemctl stop nagios
cd /tmp
rm -rf /tmp/nagiosxi
wget https://assets.nagios.com/downloads/nagiosxi/5/xi-5.6.14.tar.gz
tar zxf xi-5.6.14.tar.gz
cd /tmp/nagiosxi
./init.sh
cd /tmp/nagiosxi/subcomponents/ndoutils
./install
systemctl enable ndo2db
# If you changed your DB user/pass, get it from /usr/local/nagios/etc/ndo.cfg and update your /usr/local/nagios/etc/ndo2db.cfg file with them before running the next command
systemctl start ndo2db

Then edit your /usr/local/nagios/etc/nagios.cfg and make sure this line is uncommented:

Code: Select all

broker_module=/usr/local/nagios/bin/ndomod.o config_file=/usr/local/nagios/etc/ndomod.cfg

Make sure this line is commented:

Code: Select all

#broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg

Then start the nagios service:

Code: Select all

systemctl start nagios

[RIDS_I2MP]

Hello,

The steps that you have provided, I am not sure if they will downgrade Nagios XI version as well.
Could you please confirm on that?

I am not only facing this particular issue, if I do any changes, eg. if I am removing a host, it takes almost 4-5 hours to reflect the same in GUI.

We are not able to monitor anything properly.

Could you please help us with a remote session?

Infact, we do have a booking today for MSSQL services issue. I request you to check it once.

[ssax]

It will not downgrade your XI version, it will only downgrade NDO3 back to NDO2DB.

We can take a look on the remote tomorrow.

[ssax]

There is a bug in XI 5.7+ with the new NDO on some systems, there is not currently a resolution for it, development is currently working the issue and a fix is expected to be released in the upcoming XI 5.7.3.

Here is the process we ran on your system: (you stated you needed to do this on the other system as well)

As a workaround, please do this to downgrade NDO3 to NDO2DB (they are entirely different) and it should resolve your issues:

Take an XI backup first just in case:

https://assets.nagios.com/downloads/nag ... ios-XI.pdf

Code: Select all

systemctl stop nagios
cd /tmp
rm -rf /tmp/nagiosxi
wget https://assets.nagios.com/downloads/nagiosxi/5/xi-5.6.14.tar.gz
tar zxf xi-5.6.14.tar.gz
cd /tmp/nagiosxi
./init.sh
cd /tmp/nagiosxi/subcomponents/ndoutils
./install
systemctl enable ndo2db
systemctl start ndo2db

Then edit your /usr/local/nagios/etc/nagios.cfg and add this line to the bottom:

Code: Select all

broker_module=/usr/local/nagios/bin/ndomod.o config_file=/usr/local/nagios/etc/ndomod.cfg

Make sure all other occurrences of this are commented out:

Code: Select all

#broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg

Then start the nagios service:

Code: Select all

systemctl start nagios

Once XI 5.7.3 is released it should automatically re-upgrade you to the new NDO3 (what you want).


Additionally, I saw some IO wait on your system, IO wait usually means the storage isn't fast enough to keep up with the system so you see CPU waiting on disk and it impacts your system quite a bit. Please check to see if you can increase the storage speed of your VM as it will cause slowness throughout the system and cause things to queue up waiting for storage (DB/checks/everything) and also cause crashing of tables.

We checked the DB table sizes with this command:

Code: Select all

echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -h 127.0.0.1 -uroot -pnagiosxi --table

Your nagios_logentries table was a little big which can impact the performance of your system when you have IO wait, here is the retention/cleanup information:

NOTE: You may need to adjust the -h 127.0.0.1, the -uroot, and -pnagiosxi in the commands if your DB is housed/stored/offloaded/contained on a different server and/or you've changed the root mysql password​

If you don't care about the data, or already have a backup, you can just truncate the tables which will essentially drop and recreate the table with zero data in it (removing all historical data for the respective reports):

nagios_logentries - Impacts Event Log report length

Code: Select all

mysql -uroot -pnagiosxi -h 127.0.0.1 -B nagios -e 'TRUNCATE TABLE nagios_logentries;'

nagios_statehistory - Impacts the State History report length

Code: Select all

mysql -uroot -pnagiosxi -h 127.0.0.1 -B nagios -e 'TRUNCATE TABLE nagios_statehistory;'

nagios_notifications - Impacts the Notifications report length

Code: Select all

mysql -uroot -pnagiosxi -h 127.0.0.1 -B nagios -e 'TRUNCATE TABLE nagios_notifications;'

These should technically work to clean the DB tables up manually (if the tables aren't crashed, if they ARE crashed, you will need to repair the database FIRST in order to run these queries):

nagios_logentries - Impacts Event Log report length

Code: Select all

mysql -uroot -pnagiosxi -h 127.0.0.1 -B nagios -e 'DELETE FROM nagios_logentries WHERE logentry_time <= (NOW() - INTERVAL 6 MONTH);'

nagios_statehistory - Impacts the State History report length

Code: Select all

mysql -uroot -pnagiosxi -h 127.0.0.1 -B nagios -e 'DELETE FROM nagios_statehistory WHERE state_time <= (NOW() - INTERVAL 6 MONTH);'

nagios_notifications - Impacts the Notifications report length

Code: Select all

mysql -uroot -pnagiosxi -h 127.0.0.1 -B nagios -e 'DELETE FROM nagios_notifications WHERE start_time <= (NOW() - INTERVAL 6 MONTH);'

Then you should go to Admin > Performance Settings > Databases tab and adjust ALL of the retention intervals to meet your business data policy standards to keep them cleaned up as these settings are for adjusting the retention on those DB tables.

I would lower them to the smallest possible level and utilize the XI backup/restore process and the Admin > Scheduled Backups process to offload the backups to another server. Since these XI backups contain database backups you can spin them up to grab the data and report on them if needed.

[RIDS_I2MP]

Hello,

I am getting below error while trying to truncate table:

[nagios@vmaz-nagiosxi ~]$ mysql -uroot -pnagiosxi -h 127.0.0.1 -B nagios -e 'TRUNCATE TABLE nagios_logentries;'
mysql: [Warning] Using a password on the command line interface can be insecure.

Could you please help us on it?

And, other suggestions that you had provided are already done. Still we can see Nagios XI is too much slow. Could you please help us to resolve it.
Let me know if we can connect again through remote session for the same.