Nagios Performance Graphs

[RebeccaIlene]

The performance graphs do not work in Nagios.

It throws an error "No performance graphs were found for this service. If you have just started monitoring this object then it may take up to 15 minutes for the performance graphs to appear.".

Can someone please suggest on this can be fixed?

[benjaminsmith]

Hi Rebecca,

Is this happening for all performance graphs or for only one host or server? To start, let's make sure the cron daemon is running on the system?

RHEL/CENT OS

Code: Select all

systemctl status crond.service

And then send us your system profile for us to review. Thanks!

To send us your system profile.
Login to the Nagios XI GUI using a web browser
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and share this in a private message and then reply to this post to bring it up in the queue.

Reference
Nagios XI - Debugging Bandwidth Performance Graphs

[RebeccaIlene]

Hi Benjamin,

Thanks for your reply. We have had space issues on the server and the files under /etc/mrtg/conf.d/ were deleted.

There are no files under that directory. Can this be the cause for the problem?

Also, here is the output for the time LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg command.

[root@hostname mrtg]# time LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg

real 0m1.586s
user 0m0.096s
sys 0m0.020s

Also confirming that there are no rrd files in /var/lib/mrtg.

Can you please suggest if this is the problem and how it can be fixed?

Regards,
Rebecca

[benjaminsmith]

Hi Rebecca,

Thanks for your reply. If you can send over the system profile then I'll check the logs to make the files are not spooling up.

To send us your system profile.
Login to the Nagios XI GUI using a web browser
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and share this in a private message and then reply to this post to bring it up in the queue.

Is this happening on all your your graphs? If it's specific services, let me know the name of those services.

Also, let's verify that the nagios user account is not expired. Thanks.

Code: Select all

chage -l nagios

[RebeccaIlene]

Hi Benjamin,

I have shared a copy of the profile with you.

Please check and let me know what we can do to fix this.

Regards,
Rebecca Murray

[benjaminsmith]

Hi Rebecca,

Thank you for the profile. It looks like the system is not processing performance data well due to high system load and corrupted database tables. Let's get the database repaired and adjust the settings for performance graphs.

Also, I would recommend adding more RAM and CPU resources on this server to decrease the load as well. It needs more resources
to handle the current check load 12,000 (host + services.)

https://assets.nagios.com/downloads/nag ... ements.pdf

1. Run the Database Repair scripts

Code: Select all

/usr/local/nagiosxi/scripts/repair_databases.sh

2. Increase the performance data timeout settings

You can increase the performance data processor's timeout range by editing:
/usr/local/nagios/etc/pnp/process_perfdata.cfg

Change TIMEOUT to:

Code: Select all

TIMEOUT = 20

3. Increase the max load threshold

You can increase this threshold by editing the following file:
/usr/local/nagios/etc/pnp/npcd.cfg

Code: Select all

load_threshold = 60

4. Do a full restart on the server.

Code: Select all

systemctl stop crond
systemctl stop npcd
systemctl stop nagios
systemctl stop ndo2db
pkill -9 -u nagios
for i in $(ipcs -q | grep nagios |awk '{print $2}'); do ipcrm -q $i; done
rm -rf /usr/local/nagiosxi/var/dbmaint.lock
rm -rf /usr/local/nagiosxi/var/event_handler.lock
rm -rf /usr/local/nagiosxi/scripts/reconfigure_nagios.lock
systemctl start npcd
systemctl start crond
systemctl start nagios
systemctl start ndo2db

Let the system run for a while and if you still having trouble with performance graphs, let's get a fresh profile and the output to the following SQL query.

Code: Select all

echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -uroot -pnagiosxi --table

Reference
Nagios XI Performance Graph Problems

[RebeccaIlene]

Hi Benjamin,

Have done the mentioned changes but that did not help.

Have sent you another copy of the profile. Can you please have a look?

Reagrds,
Rebecca

[RebeccaIlene]

+--------------------------------------------+------------+
| Table | Size in MB |
+--------------------------------------------+------------+
| nagios_acknowledgements | 0.00 |
| nagios_commands | 0.00 |
| nagios_commenthistory | 7.10 |
| nagios_comments | 0.01 |
| nagios_configfiles | 0.00 |
| nagios_configfilevariables | 0.01 |
| nagios_conninfo | 0.04 |
| nagios_contact_addresses | 0.00 |
| nagios_contact_notificationcommands | 0.00 |
| nagios_contactgroup_members | 0.00 |
| nagios_contactgroups | 0.00 |
| nagios_contactnotificationmethods | 89.46 |
| nagios_contactnotifications | 95.19 |
| nagios_contacts | 0.00 |
| nagios_contactstatus | 0.00 |
| nagios_customvariables | 0.00 |
| nagios_customvariablestatus | 0.01 |
| nagios_dbversion | 0.00 |
| nagios_downtimehistory | 0.42 |
| nagios_eventhandlers | 0.13 |
| nagios_externalcommands | 0.01 |
| nagios_flappinghistory | 2.30 |
| nagios_host_contactgroups | 0.00 |
| nagios_host_contacts | 0.00 |
| nagios_host_parenthosts | 0.00 |
| nagios_hostchecks | 0.00 |
| nagios_hostdependencies | 0.00 |
| nagios_hostescalation_contactgroups | 0.00 |
| nagios_hostescalation_contacts | 0.00 |
| nagios_hostescalations | 0.00 |
| nagios_hostgroup_members | 0.00 |
| nagios_hostgroups | 0.00 |
| nagios_hosts | 0.00 |
| nagios_hoststatus | 0.44 |
| nagios_instances | 0.00 |
| nagios_logentries | NULL |
| nagios_notifications | 72.38 |
| nagios_objects | 2.74 |
| nagios_processevents | 0.09 |
| nagios_programstatus | 0.00 |
| nagios_runtimevariables | 0.00 |
| nagios_scheduleddowntime | 0.00 |
| nagios_service_contactgroups | 0.00 |
| nagios_service_contacts | 0.00 |
| nagios_service_parentservices | 0.00 |
| nagios_servicechecks | 0.00 |
| nagios_servicedependencies | 0.00 |
| nagios_serviceescalation_contactgroups | 0.00 |
| nagios_serviceescalation_contacts | 0.00 |
| nagios_serviceescalations | 0.00 |
| nagios_servicegroup_members | 0.00 |
| nagios_servicegroups | 0.00 |
| nagios_services | 0.00 |
| nagios_servicestatus | 6.63 |
| nagios_statehistory | 220.98 |
| nagios_systemcommands | 0.04 |
| nagios_timedeventqueue | 0.00 |
| nagios_timedevents | 0.00 |
| nagios_timeperiod_timeranges | 0.00 |
| nagios_timeperiods | 0.00 |
| tbl_command | 0.11 |
| tbl_contact | 0.09 |
| tbl_contactgroup | 0.13 |
| tbl_contacttemplate | 0.03 |
| tbl_domain | 0.03 |
| tbl_host | 0.28 |
| tbl_hostdependency | 0.03 |
| tbl_hostescalation | 0.03 |
| tbl_hostextinfo | 0.03 |
| tbl_hostgroup | 0.03 |
| tbl_hosttemplate | 0.03 |
| tbl_info | 0.17 |
| tbl_lnkContactToCommandHost | 0.02 |
| tbl_lnkContactToCommandService | 0.02 |
| tbl_lnkContactToContactgroup | 0.02 |
| tbl_lnkContactToContacttemplate | 0.02 |
| tbl_lnkContactToVariabledefinition | 0.02 |
| tbl_lnkContactgroupToContact | 0.08 |
| tbl_lnkContactgroupToContactgroup | 0.02 |
| tbl_lnkContacttemplateToCommandHost | 0.02 |
| tbl_lnkContacttemplateToCommandService | 0.02 |
| tbl_lnkContacttemplateToContactgroup | 0.02 |
| tbl_lnkContacttemplateToContacttemplate | 0.02 |
| tbl_lnkContacttemplateToVariabledefinition | 0.02 |
| tbl_lnkHostToContact | 0.02 |
| tbl_lnkHostToContactgroup | 0.08 |
| tbl_lnkHostToHost | 0.02 |
| tbl_lnkHostToHostgroup | 0.08 |
| tbl_lnkHostToHosttemplate | 0.06 |
| tbl_lnkHostToVariabledefinition | 0.02 |
| tbl_lnkHostdependencyToHost_DH | 0.02 |
| tbl_lnkHostdependencyToHost_H | 0.02 |
| tbl_lnkHostdependencyToHostgroup_DH | 0.02 |
| tbl_lnkHostdependencyToHostgroup_H | 0.02 |
| tbl_lnkHostescalationToContact | 0.02 |
| tbl_lnkHostescalationToContactgroup | 0.02 |
| tbl_lnkHostescalationToHost | 0.02 |
| tbl_lnkHostescalationToHostgroup | 0.02 |
| tbl_lnkHostgroupToHost | 0.02 |
| tbl_lnkHostgroupToHostgroup | 0.02 |
| tbl_lnkHosttemplateToContact | 0.02 |
| tbl_lnkHosttemplateToContactgroup | 0.02 |
| tbl_lnkHosttemplateToHost | 0.02 |
| tbl_lnkHosttemplateToHostgroup | 0.02 |
| tbl_lnkHosttemplateToHosttemplate | 0.02 |
| tbl_lnkHosttemplateToVariabledefinition | 0.02 |
| tbl_lnkServiceToContact | 0.13 |
| tbl_lnkServiceToContactgroup | 0.38 |
| tbl_lnkServiceToHost | 0.33 |
| tbl_lnkServiceToHostgroup | 0.02 |
| tbl_lnkServiceToServicegroup | 0.36 |
| tbl_lnkServiceToServicetemplate | 1.52 |
| tbl_lnkServiceToVariabledefinition | 0.02 |
| tbl_lnkServicedependencyToHost_DH | 0.02 |
| tbl_lnkServicedependencyToHost_H | 0.02 |
| tbl_lnkServicedependencyToHostgroup_DH | 0.02 |
| tbl_lnkServicedependencyToHostgroup_H | 0.02 |
| tbl_lnkServicedependencyToService_DS | 0.02 |
| tbl_lnkServicedependencyToService_S | 0.02 |
| tbl_lnkServicedependencyToServicegroup_DS | 0.02 |
| tbl_lnkServicedependencyToServicegroup_S | 0.02 |
| tbl_lnkServiceescalationToContact | 0.02 |
| tbl_lnkServiceescalationToContactgroup | 0.02 |
| tbl_lnkServiceescalationToHost | 0.02 |
| tbl_lnkServiceescalationToHostgroup | 0.02 |
| tbl_lnkServiceescalationToService | 0.02 |
| tbl_lnkServiceescalationToServicegroup | 0.02 |
| tbl_lnkServicegroupToService | 0.08 |
| tbl_lnkServicegroupToServicegroup | 0.02 |
| tbl_lnkServicetemplateToContact | 0.02 |
| tbl_lnkServicetemplateToContactgroup | 0.02 |
| tbl_lnkServicetemplateToHost | 0.02 |
| tbl_lnkServicetemplateToHostgroup | 0.02 |
| tbl_lnkServicetemplateToServicegroup | 0.02 |
| tbl_lnkServicetemplateToServicetemplate | 0.02 |
| tbl_lnkServicetemplateToVariabledefinition | 0.02 |
| tbl_lnkTimeperiodToTimeperiod | 0.02 |
| tbl_logbook | 0.02 |
| tbl_mainmenu | 0.02 |
| tbl_permission | 0.02 |
| tbl_permission_inactive | 0.02 |
| tbl_service | 2.52 |
| tbl_servicedependency | 0.03 |
| tbl_serviceescalation | 0.03 |
| tbl_serviceextinfo | 0.03 |
| tbl_servicegroup | 0.03 |
| tbl_servicetemplate | 0.03 |
| tbl_session | 0.02 |
| tbl_session_locks | 0.02 |
| tbl_settings | 0.03 |
| tbl_submenu | 0.02 |
| tbl_timedefinition | 0.06 |
| tbl_timeperiod | 0.03 |
| tbl_user | 0.03 |
| tbl_variabledefinition | 0.02 |
| xi_auditlog | 16.58 |
| xi_auth_tokens | 0.03 |
| xi_cmp_trapdata | 0.03 |
| xi_cmp_trapdata_log | 0.03 |
| xi_commands | 0.02 |
| xi_eventqueue | 0.03 |
| xi_events | 0.56 |
| xi_meta | 7.34 |
| xi_mibs | 0.05 |
| xi_options | 0.03 |
| xi_sessions | 0.03 |
| xi_sysstat | 0.03 |
| xi_usermeta | 0.16 |
| xi_users | 0.03 |
+--------------------------------------------+------------+

Here is the output of the command provided.

[benjaminsmith]

Hi Rebecca,

Thank you for your patience. The log entries table is still corrupted, let's truncate that table and run the repair script once more.

1. Truncate logentries table

Code: Select all

mysql -uroot -pnagiosxi -h 127.0.0.1 -B nagios -e 'TRUNCATE TABLE nagios_logentries;'

2. Run the following command to repair the database again.

Code: Select all

/usr/local/nagiosxi/scripts/repair_databases.sh

The SQL query for the database table sizes should show a value of 0 instead of NULL. If successful, re-start and let the server run for a while, and let me know if it's starts to generate graphs again.

| nagios_logentries | NULL |

Also, did you increase the timeout settings /usr/local/nagios/etc/pnp/process_perfdata.cfg? If the database does not repair successfully, let's move this over to a support ticket as we may need to completely drop and re-build that table.

https://support.nagios.com/tickets/

[RebeccaIlene]

Hi Benjamin,

Yes the /usr/local/nagios/etc/pnp/process_perfdata.cfg has been updated with TIMEOUT=20.

I have run the Truncate logentries table, repaired the database and restarted the services but no difference. The graphs still do not work.

Regards,
Rebecca Murray