Nagios NCPA agent generating false downs

[mrjsokol]

Good morning,

I wanted to look into the NCPA service. When a site would go down, the npca service couldnt detect that the client was back online or the client would randomly go down and not go back online. I will DM the person that grabs the ticket a copy of the profile with the details of the specific client to look at that went down but has been online.

[vtrac]

Hi,
Greetings .....
Usually a specific NCPA check (for exampel "check_ncpa.py") do not check for host or service up/down. Those are done with a specific service defined like "PING" service for example.

When called "check_ncpa.py", it will goes ahead and execute the check regardless if your remote NCPA is up or down.

Yes, please PM me the profile.zip and details of hostName / serviceName of the one having issue.

Please also provide sample command(s) used with outputs of the trouble NCPA check, if possible.


Best Regards,
Vinh

[mrjsokol]

Hey Vtrac,

I have pm'ed you the details. Please do not post any client information IE name or IP address in chat. The machine in question can be referred as the client.

Thank you and have a great day,
Joseph

[vtrac]

Hi,
You have a database crashed issue, based on the log:

Code: Select all

210406 15:30:48 [ERROR] mysqld: Table './nagios/nagios_contactnotifications' is marked as crashed and should be repaired
210406 15:30:48 [Warning] Checking table:   './nagios/nagios_contactnotifications'
210406 15:30:49 [ERROR] mysqld: Table './nagios/nagios_contactnotificationmethods' is marked as crashed and should be repaired
210406 15:30:49 [Warning] Checking table:   './nagios/nagios_contactnotificationmethods'
210406 15:30:49 [ERROR] mysqld: Table './nagios/nagios_statehistory' is marked as crashed and should be repaired
210406 15:30:49 [Warning] Checking table:   './nagios/nagios_statehistory'

Please run the below command to repair your DB:

Code: Select all

/usr/local/nagiosxi/scripts/repair_databases.sh

Also, looking at the log:
[1623067863] HOST ALERT: CLIENT;DOWN;SOFT;1;CRITICAL: Host CLIENThas not checked in


Freshness checking:

Code: Select all

It's the responsibility of the external devices / applications to send the check results through, 
all Nagios XI does is wait for the passive check results. 

If Nagios XI hasn't heard from the passively monitored device / application for a specified amount of time then it can take
action. The most common action is to submit a check result to Nagios XI with a critical state, this ensures that
notifications are triggered and it appears as critical in the monitoring interface.

Based on the log, "CLIENT" has not checked in so Nagios XI will take action and submit result with Critical state.

Please check your remote "CLIENT" and see if your agent is running.


Best Regards,
Vinh

[mrjsokol]

I suspect the issues I am having are related to the database crashed issue. There are a few clients at the same site that are experiencing this issue currently. Once the database script is done running, I will let you know the result. If the client doesnt check in with nagios, usually I have to restart the service.

[mrjsokol]

Ive posted another PM about another down client. We will continue to monitor but please respond to the inquiry in the PM as the new client.

[vtrac]

Hi,
How are you doing?

Please check the passive log (below) and see if there is/are issue(s) with the NCPA listener.

Code: Select all

Windows:
C:\Program Files (x86)\Nagios\NCPA\var\log\

Unix:
/usr/local/ncpa/var/log/

Also, please update NCPA to the latest if you can. The latest released version is "2.3.1"

https://www.nagios.org/ncpa/#downloads


Best Regards,
Vinh

[mrjsokol]

Good morning,

I will look into that, for now we can close the ticket since I have a busy week and probably wont get to this until mid to next week. I will open a different forum post if needed unless you prefer me to respond on this same forum post.

[mrjsokol]

Last comment on this ticket. I did have time to look at something related to false down. I will pm you the details of " the new client" if you could tell me why this is reporting down when its up.

[mrjsokol]

Ignore last comment, it maybe a latency issue, looking into increasing the times potentially for down. TBD.