performance graphs stopped working

[jeephigh]

Hello,

I am running nagiosxi installed from the preconfigured vm image. The performance graphs no longer display any data. The plugin check_ping is returning performance data on each host as seen by looking at advanced tab in the host status detail. I can see the performance data being returned but graphs are not being displayed.

I have restarted the performance grapher and it restarts successfully.

Here is my npcd.log
[04-06-2016 16:33:31] NPCD: npcd Daemon (0.4.14) started with PID=21534
[04-06-2016 16:33:31] NPCD: Please have a look at 'npcd -V' to get license information
[04-06-2016 16:33:31] NPCD: HINT: load_threshold is enabled - ('10.000000')
[04-06-2016 16:33:52] NPCD: Caught Termination Signal - Hasta la vista... baby
[04-27-2016 13:40:05] NPCD: npcd Daemon (0.4.14) started with PID=1503
[04-27-2016 13:40:05] NPCD: Please have a look at 'npcd -V' to get license information
[04-27-2016 13:40:05] NPCD: HINT: load_threshold is enabled - ('10.000000')
[05-22-2016 19:53:19] NPCD: npcd Daemon (0.4.14) started with PID=1610
[05-22-2016 19:53:19] NPCD: Please have a look at 'npcd -V' to get license information
[05-22-2016 19:53:19] NPCD: HINT: load_threshold is enabled - ('10.000000')
[06-03-2016 11:19:44] NPCD: Caught Termination Signal - Hasta la vista... baby
[06-03-2016 11:19:44] NPCD: npcd Daemon (0.4.14) started with PID=1856
[06-03-2016 11:19:44] NPCD: Please have a look at 'npcd -V' to get license information
[06-03-2016 11:19:44] NPCD: HINT: load_threshold is enabled - ('10.000000')

I don't see much there unless the load_threshold is a problem. The nagiosxi server is not under any major load, in fact it is 92-94% idle. Only running simple checks on a small number of servers at this point.

Any Ideas? Any help would be much appreciated. We are a week or two out from purchasing an enterprise license but I would love to get this part working as we are adding more to this server and using what we have on it now.

[rkennedy]

The NPCD logs look fine -- nothing to worry about.

This is odd though, I'd like to look more into your system, can you PM a profile over? (Admin -> System Profile -> Download Profile)

[jeephigh]

Yeah sure thing, I have downloaded my system profile and I am ready to send it over as a PM. Do I need to add you somewhere to do that? I looked at your profile and couldn't see a PM option, sorry first post on this forum.

[rkennedy]

It actually will let you send a PM after you have two posts, so you should be able to send it over now. Please respond here once you have, and I'll take a look at it.

[jeephigh]

Oh I see why I can't PM. I am not authorized to PM until I have more Forum Presence. haha. Here is my system profile minus the network, which looks fine. I had to screenshot two images.

****Edit
I deleted the system profile images I uploaded and sent you file over PM

[rkennedy]

Got it.

Looking at your logs, I'm noticing a couple things -

Code: Select all

160522 19:59:49 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_systemcommands.MYI'; try to repair it
160522 20:00:01 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_systemcommands.MYI'; try to repair it
160522 20:00:01 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_systemcommands.MYI'; try to repair it

For this, try running through this document which should help - https://assets.nagios.com/downloads/nag ... tabase.pdf

Code: Select all

mv: `/usr/local/nagios/share/perfdata/rrcs-x-x-x-x.central.biz.rr.com/Optical_InErrors.xml' and `/usr/local/nagios/share/perfdata/rrcs-x-x-x-x.central.biz.rr.com/Optical_InErrors.xml' are the same file
mv: `/usr/local/nagios/share/perfdata/rrcs-x-x-x-x.central.biz.rr.com/Optical_MTU.rrd' and `/usr/local/nagios/share/perfdata/rrcs-x-x-x-x.central.biz.rr.com/Optical_MTU.rrd' are the same file
mv: `/usr/local/nagios/share/perfdata/rrcs-x-x-x-x.central.biz.rr.com/Optical_MTU.xml' and `/usr/local/nagios/share/perfdata/rrcs-x-x-x-x.central.biz.rr.com/Optical_MTU.xml' are the same file

Were any changes made to your system? The error is clear, just trying to figure out why this is in your apache logs.

[jeephigh]

Thank you I will take a look at that document.

Those logs seem to be complaining about sql errors, would this be affecting the perf graphs? As far as I can tell everything else is working as it should.

As for the apache logs, I did an snmp walk on a firewall just for testing purposes, I have since removed that host and its services.

[Box293]

Those logs seem to be complaining about sql errors, would this be affecting the perf graphs? As far as I can tell everything else is working as it should.

Code: Select all

160522 19:59:49 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_systemcommands.MYI'; try to repair it
160522 20:00:01 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_systemcommands.MYI'; try to repair it
160522 20:00:01 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_systemcommands.MYI'; try to repair it

The tables have commands that are used in the performance data processing.

If you can't fix the error using the database repair scripts, you will most likely need to restore from backup. One of these two guides should help you restore the nagios database:

https://support.nagios.com/kb/article.php?id=180
https://support.nagios.com/kb/article.php?id=492

The latter article will allow you to just restore the nagios database only.

[jeephigh]

I ran the database repair scripts successfully. Although the only errors in the MySQL logs are from 5-22-16. The performance graphs and the state history were working after that, but have since stopped working. I just noticed today that the state history reports only go up to a certain date and then stop showing. I did a tail on the MySQL logs and I am just not getting any errors even after restarting sql and running tasks. Could there be something else that is not configured correctly? Do you have any other ideas or questions for me to help me get the state history working again. Its funny because I have not changed anything since it was working, then it just stopped.

Any more help on this would be appreciated.

[jeephigh]

mysql_tail.PNG

Here is the tail on the sql log since running the repair script.