Missing monitors but still visible in config manager

[nms]

Hi,

We have a report which was set by placing some graphs on a dashboard. This report was being sent on a daily basis by mail. We noticed that one of the machines had some of its graphs not populating.
When checking the node I observed that some of the monitors were missing in the service detail, however when checking the config manager they were seen there and also activated. in order to be seen again in the service detail I opened one monitor and re-saved it, applied config and they were visible again.
Any reason why this could have happened?

[npolovenko]

Hello, @nms. Can you share a screenshot of the graph with the service detail?

Also, please send in your Nagios XI System Profile so I can review it?
To send us your system profile. Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and upload it to a cloud storage of your choice. You can share a link with me in a personal message.
After you upload the profile please post something in this thread to bring it up in the support queue.

**Profile received and shared with the support team.

[nms]

Hi,

The profile has been sent via pm.
Attached are some screenshots for the graph.

[lmiltchev]

Most probably you are seeing gaps in graphs because the load on the system exceeded the load_threshold value, specified in the npcd.cfg file.

[08-09-2018 06:23:21] NPCD: HINT: load_threshold is enabled - ('10.000000')
[08-12-2018 13:10:46] NPCD: WARN: MAX load reached: load 18.560000/10.000000 at i=0
[08-12-2018 13:11:01] NPCD: WARN: MAX load reached: load 14.900000/10.000000 at i=1
[08-12-2018 13:11:16] NPCD: WARN: MAX load reached: load 12.040000/10.000000 at i=1
[08-13-2018 06:26:18] NPCD: WARN: MAX load reached: load 10.770000/10.000000 at i=0
[08-15-2018 18:00:41] NPCD: WARN: MAX load reached: load 12.930000/10.000000 at i=0
[08-15-2018 18:00:56] NPCD: WARN: MAX load reached: load 10.290000/10.000000 at i=1
[08-16-2018 18:02:05] NPCD: WARN: MAX load reached: load 10.390000/10.000000 at i=0
[08-16-2018 18:02:36] NPCD: WARN: MAX load reached: load 10.790000/10.000000 at i=0
[08-17-2018 19:01:50] NPCD: WARN: MAX load reached: load 10.730000/10.000000 at i=0
[08-17-2018 19:02:22] NPCD: WARN: MAX load reached: load 10.490000/10.000000 at i=0
[08-17-2018 19:02:53] NPCD: WARN: MAX load reached: load 10.680000/10.000000 at i=0
[08-17-2018 20:20:48] NPCD: WARN: MAX load reached: load 22.960000/10.000000 at i=0
[08-17-2018 20:21:03] NPCD: WARN: MAX load reached: load 18.230000/10.000000 at i=1
[08-17-2018 20:21:18] NPCD: WARN: MAX load reached: load 14.340000/10.000000 at i=1
[08-17-2018 20:21:33] NPCD: WARN: MAX load reached: load 11.390000/10.000000 at i=1
[08-18-2018 01:59:28] NPCD: WARN: MAX load reached: load 10.030000/10.000000 at i=0
[08-18-2018 07:00:53] NPCD: WARN: MAX load reached: load 21.660000/10.000000 at i=0
[08-18-2018 07:01:08] NPCD: WARN: MAX load reached: load 17.690000/10.000000 at i=1
[08-18-2018 07:01:23] NPCD: WARN: MAX load reached: load 15.020000/10.000000 at i=1
[08-18-2018 07:01:38] NPCD: WARN: MAX load reached: load 12.070000/10.000000 at i=1
[08-25-2018 01:11:03] NPCD: WARN: MAX load reached: load 11.770000/10.000000 at i=0
[08-25-2018 14:35:06] NPCD: WARN: MAX load reached: load 10.520000/10.000000 at i=0
[08-27-2018 13:09:21] NPCD: Caught Termination Signal - Hasta la vista... baby

I would recommend that you increase this value, depending on your system resources (20.0 for a dual core CPU, 40,0 - for a quad core, etc.)

Read more on performance graph issues here:

https://support.nagios.com/kb/article/n ... ems-9.html

[npolovenko]

@nms, To answer your second question.

When checking the node I observed that some of the monitors were missing in the service detail, however when checking the config manager they were seen there and also activated. in order to be seen again in the service detail I opened one monitor and re-saved it, applied config and they were visible again.

Have you added these new hosts recently? Perhaps after adding these hosts, you did not click on the Apply Configuration the first time?

[nms]

Hi,

Our current system is a VM with 2 virtual sockets each with 4 cores per socket, that makes it 8 core.
I think with this current CPU configuration we can go with a threshold of 40.
Also looking at the CPU, currently it's quite comfortable, reaching 60% (user + system). Do you also propose a threshold of 40 or else start with a lower value?

For the second question no they have been added a long time ago and then graphs added to the report. We noticed this when checking the daily report that graphs for this particular node were not being graphed which led us to notice that they weren't visible in the service detail.

Rgds

[npolovenko]

@nms, Setting the threshold to 40 is a good idea. This doesn't mean NPCD will consume more resources automatically. All the threshold does is it temporarily stops the npcd daemon if the system load goes above the threshold level in order to better balance system resources.
This problem with disappearing services is unusual. Either some user did something to the service and did not save the changes, or you have a db corruption or ND02DB had some trouble communicating with the database. You could PM me your system profile and I'll take a look at your logs to see if anything stands out. Otherwise, you can wait to see if this happens again and if it does, generate the system profile right away and share it with us.

Profile was recieved and shared with the Support team.

[nms]

Hello

For the npcd I just increased the threshold to 40 today. I will keep an eye to see for anything unusual in the logs.
For the disappearance of the service, yes quite unusual, but I cannot really tell if a user using the admin did any changes with myself knowing (it's only 3 of us that are admins).
I uploaded the system profile just in case you find something unusual.

Rgds,

[npolovenko]

Thank you. I shared the profile with the support team.
Can you run the following command and when it's done upload the nagios.txt file from the /tmp folder?

ls -lR /var/nagiosramdisk > /tmp/nagios.txt

[nms]

Thanks.

File attached.