NCPA port 5693 is not listening but ncpa_listener was up

[sneha.irali]

Hi Team,

We have come across a instance where the ncpa_listerner was running but 5693 port was not listening locally on the machine also not connecting via Nagios server.

Server was rebooted recently and as of today the uptime was 2 days.

We did a restart of ncpa_listener service and the communication via 5693 was established. Can anyone help me understand what would be the reason for this kind of behavior.

[sneha.irali]

listener logs attached.

NCPA is installed on suse 15.1 server.

[vtrac]

Hi,
Here are what I see in the ncpa_listener.log:

Code: Select all

2021-04-12 19:16:46,534 38391 INFO ::ffff:10.160.113.239 - - [2021-04-12 19:16:46] "GET /api/disk/logical/%7Csrv/?token=NagiosXI%40Monitor&warning=80&critical=90&check=1 HTTP/1.1" 200 433 0.059141
2021-04-15 02:07:52,616 197287 INFO started
2021-04-15 02:07:55,076 197287 INFO Using SSL version TLSv1_2
2021-04-15 02:08:33,648 197287 INFO ::ffff:10.160.113.239 - - [2021-04-15 02:08:33] "GET /api/disk/logical/%7Cboot%7Cefi/?token=NagiosXI%40Monitor&warning=80&critical=90&check=1 HTTP/1.1" 200 404 0.063142
2021-04-15 02:08:37,032 197287 INFO ::ffff:10.160.113.239 - - [2021-04-15 02:08:37] "GET /api/disk/logical/%7Cvar%7Copt/?token=NagiosXI%40Monitor&warning=80&critical=90&check=1 HTTP/1.1" 200 433 0.055671
2021-04-15 02:08:37,355 197287 INFO ::ffff:10.160.113.239 - - [2021-04-15 02:08:37] "GET /api/disk/logical/%7Chome/?token=NagiosXI%40Monitor&warning=80&critical=90&check=1 HTTP/1.1" 200 433 0.055876
2021-04-15 02:08:38,740 197287 INFO ::ffff:10.160.113.239 - - [2021-04-15 02:08:38] "GET /api/disk/logical/%7Cvar%7Ccrash/?token=NagiosXI%40Monitor&warning=80&critical=90&check=1 HTTP/1.1" 200 433 0.066839
2021-04-15 02:08:44,616 197287 INFO stopped
2021-04-15 02:09:24,070 779553 INFO started
2021-04-15 02:09:24,113 779553 INFO Using SSL version TLSv1_2

Yes, looks like the listener has stopped receiving data on "2021-04-12 19:16:46" until it was started on "2021-04-15 02:07:52".

Based on the info provided, there is not a way to know what going on with your system.

Could you please check the "/var/log/messages" and see what happens at around "2021-04-12 19:16:46"?

I would also check with the Network and Admin group to see if there were any activities on that date and time.


Best Regards,
Vinh

[sneha.irali]

ok, I am checking on var/log/messages.

Talking about any activities then it was just a reboot of the server since the CPU utilization was 100%.

[vtrac]

Hi,
You has just mentioned in your last replied that CPU was at 100% before you rebooted the VM (machine).

Did you do the "top" command and see which process caused the issue?

I would also check the database's log and also the nagios's log:

Code: Select all

/var/log/mariadb/mariadb.log
/usr/local/nagios/var/nagios.log
/var/log/messages

If you have database crashed table, please run the below command to fix it:

Code: Select all

/usr/local/nagiosxi/scripts/repair_databases.sh

Best Regards,
Vinh

[sneha.irali]

Hi Vinh,

As checked the server was crashed and re-built on Friday after I post this query.

How do you want me to proceed further.

[vtrac]

Hi,
What OS is your NCPA runs on?

NCPA is just an agent, which has two running services (ncpa_listener & ncpa_passive).
Those services either in "running" or "stopped" and are very lite weights.

The reason you installed NCPA on your remote host was to monitor your running system and /or services that are running on that system.
Your running system might be a database server or mail server, for example.

However, the reason NCPA listener stopped (as you have seen from the log with no error) might be caused by all your other running services on that machine like backup that used up all the system resources.

Could you please check with your system-admin and see what installed on that machine?
Also, what is running on that machine.
See if you need to add more resources like CPU, memory or disk space increase.

If this is a UNIX machine, I would use the "top" command and see what is the state of your system.
Which running service(s) used up all your resources.
I would also check the "syslog.log" and the "/var/log/messages"


Best Regards,
Vinh