Page 1 of 2

Not able to monitor Apache and SSH services

Posted: Sat Feb 03, 2018 2:28 pm
by shamrozkadiwal
I have setup "check_http" "check_ssh"to monitor apache and ssh service on the server, but I have been getting CRITICAL - Socket timeout after 10/20/30 seconds error for both plugins. I have tried to increase the timer but still getting the same error. I also check the firewall and both services have been added.
Nagios Core is running on Centos 7
NRPE is running on Ubuntu

Code: Select all

[root@dj-mon ~]# /usr/local/nagios/libexec/check_http -H 10.167.176.43
CRITICAL - Socket timeout after 10 seconds

[root@dj-mon ~]# /usr/local/nagios/libexec/check_http -H 10.167.176.43 -S
CRITICAL - Socket timeout after 10 seconds

[root@dj-mon ~]# /usr/local/nagios/libexec/check_http -H 10.167.176.43 -S -t 20
CRITICAL - Socket timeout after 20 seconds

[root@dj-mon ~]# /usr/local/nagios/libexec/check_http -H 10.167.176.43 -S -p 443
CRITICAL - Socket timeout after 10 seconds

[root@dj-mon ~]# /usr/local/nagios/libexec/check_http -H 10.167.176.43 -S -p 80
CRITICAL - Socket timeout after 10 seconds

[root@dj-mon ~]# /usr/local/nagios/libexec/check_http -H 10.167.176.43 -S -p 8080
CRITICAL - Socket tieout after 10 seconds

[root@dj-mon ~]# /usr/local/nagios/libexec/check_http -H 10.167.176.43 -N -p 80
CRITICAL - Socket timeout after 10 seconds

[root@dj-mon ~]# /usr/local/nagios/libexec/check_http -H 10.167.176.43 -N -p 80 -t 1
CRITICAL - Socket timeout after 1 seconds

[root@dj-mon ~]# /usr/local/nagios/libexec/check_http -H 10.167.176.43 -N -p 80 -t 10
CRITICAL - Socket timeout after 10 seconds

[root@dj-mon ~]# /usr/local/nagios/libexec/check_http -H 10.167.176.43 -N -t 30 
CRITICAL - Socket timeout after 30 seconds

[root@dj-mon libexec]# /usr/local/nagios/libexec/check_ssh -H 10.167.176.43
CRITICAL - Socket timeout after 10 seconds

[root@dj-mon libexec]# /usr/local/nagios/libexec/check_ssh -H 10.167.176.43 -t 30
CRITICAL - Socket timeout after 30 seconds

Re: Not able to monitor Apache and SSH services

Posted: Mon Feb 05, 2018 12:39 pm
by npolovenko
Hello, @shamrozkadiwal. Are you able to nmap the server for open ports? Please show us the output of the following command:

Code: Select all

nmap 10.167.176.43

Re: Not able to monitor Apache and SSH services

Posted: Thu Feb 08, 2018 3:16 pm
by shamrozkadiwal

Code: Select all

ubuntuadmin@iusa-lin-db01:~$
ubuntuadmin@iusa-lin-db01:~$
ubuntuadmin@iusa-lin-db01:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:94:8e:ef brd ff:ff:ff:ff:ff:ff
    inet 10.167.176.43/24 brd 10.167.176.255 scope global ens160
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:fe94:8eef/64 scope link
       valid_lft forever preferred_lft forever
ubuntuadmin@iusa-lin-db01:~$
ubuntuadmin@iusa-lin-db01:~$
ubuntuadmin@iusa-lin-db01:~$ nmap 10.167.176.43

Starting Nmap 7.01 ( https://nmap.org ) at 2018-02-08 14:15 CST
Nmap scan report for iusa-lin-db01.lan.ismailiusa.org (10.167.176.43)
Host is up (0.00032s latency).
Not shown: 997 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
80/tcp   open  http
3306/tcp open  mysql

Nmap done: 1 IP address (1 host up) scanned in 0.14 seconds
ubuntuadmin@iusa-lin-db01:~$

Re: Not able to monitor Apache and SSH services

Posted: Thu Feb 08, 2018 4:33 pm
by dwhitfield
Please take a look at https://support.nagios.com/kb/article/n ... s-617.html and let us know if that does not help resolve the issue.

Re: Not able to monitor Apache and SSH services

Posted: Thu Feb 08, 2018 4:36 pm
by kyang
Are both servers online when running this check?

What version of check_http and check_ssh do you have?

Code: Select all

/usr/local/nagios/libexec/check_http -V
/usr/local/nagios/libexec/check_ssh -V
Can you run this towards your own Nagios server and not 10.167.176.43.

Code: Select all

/usr/local/nagios/libexec/check_http -H nagiosServerIP
Post the output thank you!

Re: Not able to monitor Apache and SSH services

Posted: Fri Feb 09, 2018 12:59 pm
by shamrozkadiwal
@kyang, Yes both servers are up all the time when these check runs.

Nagios Server (45.55.251.117)

Code: Select all

[root@dj-mon ~]# /usr/local/nagios/libexec/check_http -V
check_http v2.1.1 (nagios-plugins 2.1.1)
[root@dj-mon ~]# /usr/local/nagios/libexec/check_ssh -V
check_ssh v2.1.1 (nagios-plugins 2.1.1)
[root@dj-mon ~]#

Nagios Client (10.167.176.43)

Code: Select all

ubuntuadmin@iusa-lin-db01:~$ /usr/local/nagios/libexec/check_http -V
check_http v2.2.1.git (nagios-plugins 2.2.1)
ubuntuadmin@iusa-lin-db01:~$ /usr/local/nagios/libexec/check_ssh -V
check_ssh v2.2.1.git (nagios-plugins 2.2.1)
ubuntuadmin@iusa-lin-db01:~$

Nagios Server (45.55.251.117)

Code: Select all

[root@dj-mon ~]# /usr/local/nagios/libexec/check_http -H 45.55.251.117
HTTP OK: HTTP/1.1 200 OK - 664 bytes in 0.007 second response time |time=0.007303s;;;0.000000 size=664B;;;0

Re: Not able to monitor Apache and SSH services

Posted: Fri Feb 09, 2018 1:45 pm
by shamrozkadiwal
I saw this error when I tried to run "Re-schedule the next check of this service". I don't know this could be a help.

Re: Not able to monitor Apache and SSH services

Posted: Fri Feb 09, 2018 3:51 pm
by kyang
Could you create a new thread for that issue?

That way we are not troubleshooting 2 errors in one thread.

It would be less confusing for us techs as well. Thank you!

Can you run nmap 10.167.176.43 from your Nagios Server and not from the same server itself?.

Can you also run this from your Nagios Server, please post the output.

Code: Select all

/usr/local/nagios/libexec/check_http -H 10.167.176.43

Re: Not able to monitor Apache and SSH services

Posted: Fri Feb 09, 2018 8:14 pm
by shamrozkadiwal
@kyang, nmap is saying Host is down whereas It is up.

Code: Select all

[root@dj-mon ~]# /usr/local/nagios/libexec/check_http -H 10.167.176.43
CRITICAL - Socket timeout after 10 seconds
[root@dj-mon ~]#
[root@dj-mon ~]#
[root@dj-mon ~]# nmap 10.167.176.43

Starting Nmap 6.40 ( http://nmap.org ) at 2018-02-10 01:11 UTC
Note: Host seems down. If it is really up, but blocking our ping probes, try -Pn
Nmap done: 1 IP address (0 hosts up) scanned in 3.05 seconds
[root@dj-mon ~]#
[root@dj-mon ~]#

Re: Not able to monitor Apache and SSH services

Posted: Fri Feb 09, 2018 8:53 pm
by shamrozkadiwal
So I checked nrpe service on 10.167.176.43 and it was not runing.

Code: Select all

ubuntuadmin@iusa-lin-db01:/usr/local/nagios/var$ sudo systemctl status nrpe
● nrpe.service - Nagios Remote Plugin Executor
   Loaded: loaded (/lib/systemd/system/nrpe.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Fri 2018-02-09 19:41:40 CST; 7s ago
     Docs: http://www.nagios.org/documentation
  Process: 25027 ExecStopPost=/bin/rm -f /usr/local/nagios/var/nrpe.pid (code=exited, status=0/SUCCESS)
  Process: 25022 ExecStart=/usr/local/nagios/bin/nrpe -c /usr/local/nagios/etc/nrpe.cfg -f (code=exited, status=2)
 Main PID: 25022 (code=exited, status=2)

Feb 09 19:41:40 iusa-lin-db01 systemd[1]: Started Nagios Remote Plugin Executor.
Feb 09 19:41:40 iusa-lin-db01 systemd[1]: nrpe.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Feb 09 19:41:40 iusa-lin-db01 systemd[1]: nrpe.service: Unit entered failed state.
Feb 09 19:41:40 iusa-lin-db01 systemd[1]: nrpe.service: Failed with result 'exit-code'.
When I checked the logs (/usr/local/nagios/var/nrpe.log), it showed me this. There are three certificate errors, but not sure are those the reason to not able to check the services. One more weird thing that I am able to monitor other services on this server like Load, Partition, users and they all are working fine.

Code: Select all

[1518226046] parse_allowed_hosts: parsing the allowed host string >127.0.0.1,::1,45.55.251.117< to add to ACL list
[1518226046] parse_allowed_hosts: ADDING this record (127.0.0.1) to ACL list!
[1518226046] add_ipv4_to_acl: checking ip-address >127.0.0.1<
[1518226046] add_ipv4_to_acl: ip-address >127.0.0.1< correct, adding.
[1518226046] parse_allowed_hosts: Record added to ACL list!
[1518226046] parse_allowed_hosts: ADDING this record (::1) to ACL list!
[1518226046] parse_allowed_hosts: Record added to ACL list!
[1518226046] parse_allowed_hosts: ADDING this record (45.55.251.117) to ACL list!
[1518226046] add_ipv4_to_acl: checking ip-address >45.55.251.117<
[1518226046] add_ipv4_to_acl: ip-address >45.55.251.117< correct, adding.
[1518226046] parse_allowed_hosts: Record added to ACL list!
[1518226046] Showing ACL lists for both IP and DOMAIN acl's:
[1518226046] IP ACL: 127.0.0.1/32 16777343
[1518226046] IP ACL: 0.0.0.0/20 0
[1518226046] IP ACL: 45.55.251.117/32 1979397933
[1518226046] Added command[check_users]=/usr/local/nagios/libexec/check_users -w 5 -c 10
[1518226046] Added command[check_load]=/usr/local/nagios/libexec/check_load -r -w .15,.10,.05 -c .30,.25,.20
[1518226046] Added command[check_hda1]=/usr/local/nagios/libexec/check_disk -w 20% -c 10% -p /dev/hda1
[1518226046] Added command[check_zombie_procs]=/usr/local/nagios/libexec/check_procs -w 5 -c 10 -s Z
[1518226046] Added command[check_total_procs]=/usr/local/nagios/libexec/check_procs -w 150 -c 200
[1518226046] Error: could not use certificate file /etc/letsencrypt/live/monitor.theismailiusa.org/cert.pem : error:02001002:system library:fopen:No such file or directory
[1518226046] Error: could not use certificate file /etc/letsencrypt/live/monitor.theismailiusa.org/cert.pem : error:20074002:BIO routines:FILE_CTRL:system lib
[1518226046] Error: could not use certificate file /etc/letsencrypt/live/monitor.theismailiusa.org/cert.pem : error:140AD002:SSL routines:SSL_CTX_use_certificate_file:system lib
[1518226900] parse_allowed_hosts: parsing the allowed host string >127.0.0.1,::1,45.55.251.117< to add to ACL list
[1518226900] parse_allowed_hosts: ADDING this record (127.0.0.1) to ACL list!
[1518226900] add_ipv4_to_acl: checking ip-address >127.0.0.1<
[1518226900] add_ipv4_to_acl: ip-address >127.0.0.1< correct, adding.
[1518226900] parse_allowed_hosts: Record added to ACL list!
[1518226900] parse_allowed_hosts: ADDING this record (::1) to ACL list!
[1518226900] parse_allowed_hosts: Record added to ACL list!
[1518226900] parse_allowed_hosts: ADDING this record (45.55.251.117) to ACL list!
[1518226900] add_ipv4_to_acl: checking ip-address >45.55.251.117<
[1518226900] add_ipv4_to_acl: ip-address >45.55.251.117< correct, adding.
[1518226900] parse_allowed_hosts: Record added to ACL list!
[1518226900] Showing ACL lists for both IP and DOMAIN acl's:
[1518226900] IP ACL: 127.0.0.1/32 16777343
[1518226900] IP ACL: 0.0.0.0/20 0
[1518226900] IP ACL: 45.55.251.117/32 1979397933
[1518226900] Added command[check_users]=/usr/local/nagios/libexec/check_users -w 5 -c 10
[1518226900] Added command[check_load]=/usr/local/nagios/libexec/check_load -r -w .15,.10,.05 -c .30,.25,.20
[1518226900] Added command[check_hda1]=/usr/local/nagios/libexec/check_disk -w 20% -c 10% -p /dev/hda1
[1518226900] Added command[check_zombie_procs]=/usr/local/nagios/libexec/check_procs -w 5 -c 10 -s Z
[1518226900] Added command[check_total_procs]=/usr/local/nagios/libexec/check_procs -w 150 -c 200
[1518226900] Error: could not use certificate file /etc/letsencrypt/live/monitor.theismailiusa.org/cert.pem : error:02001002:system library:fopen:No such file or directory
[1518226900] Error: could not use certificate file /etc/letsencrypt/live/monitor.theismailiusa.org/cert.pem : error:20074002:BIO routines:FILE_CTRL:system lib
[1518226900] Error: could not use certificate file /etc/letsencrypt/live/monitor.theismailiusa.org/cert.pem : error:140AD002:SSL routines:SSL_CTX_use_certificate_file:system lib
ubuntuadmin@iusa-lin-db01:/usr/local/nagios/var$ ^C