Check_nrpe with Certificate Authentication

[ruffy01]

Hi all.
I've been busy trying to configure check_nrpe with authentication. I believe it's working but verification is not conclusive, I'd really appreciate your thoughts.

The set up: Nagios Core 4.1.1 and NRPE 3.0 on Centos 7 (192.168.0.9) monitoring (among others) Server 2008 (192.168.0.4) with NSClient++ Vers.0.4.4.19.
Note: NRPE is not installed on the servers being monitored, NSClient++ only.

nsclient.ini

Code: Select all

; Undocumented section
[/modules]

; CheckDisk - CheckDisk can check various file and disk related things. The current version has commands to check Size of hard drives and directories.
CheckDisk = 1

; Event log Checker. - Check for errors and warnings in the event log. This is only supported through NRPE so if you plan to use only NSClient this wont help you at all.
CheckEventLog = 0

; Check External Scripts - A simple wrapper to run external scripts and batch files.
CheckExternalScripts = 1

; Helper function - Various helper function to extend other checks. This is also only supported through NRPE.
CheckHelpers = 1

; Check NSCP - Checkes the state of the agent
CheckNSCP = 0

; CheckSystem - Various system related checks, such as CPU load, process state, service state memory usage and PDH counters.
CheckSystem = 1

; CheckWMI - CheckWMI can check various file and disk related things. The current version has commands to check Size of hard drives and directories.
CheckWMI = 0

; NRPE server - A simple server that listens for incoming NRPE connection and handles them.
NRPEServer = 1

; NSClient server - A simple server that listens for incoming NSClient (check_nt) connection and handles them. Although NRPE is the preferred method NSClient is fully supported and can be used for simplicity or for compatibility.
NSClientServer = 0


; Undocumented section
[/settings/default]

; ALLOWED HOSTS - A comaseparated list of allowed hosts. You can use netmasks (/ syntax) or * to create ranges.
allowed hosts = 127.0.0.1, 192.168.0.9

; PASSWORD - Password used to authenticate against server
;password = NSCPassword


[/settings/NSClient/server]

; PORT NUMBER - Port to use for check_nt.
;port=12489

; PERFORMANCE DATA
performance data=1

; ENABLE SSL ENCRYPTION
use ssl=1



[/settings/NRPE/server]

allowed ciphers = ALL:!MD5:@STRENGTH

ca = C:\Program Files\NSClient++\security\ca_cert.pem

certificate = C:\Program Files\NSClient++\security\client_cert.pem

certificate key = C:\Program Files\NSClient++\security\client_cert.key

verify mode = peer-cert

; COMMAND ALLOW NASTY META CHARS - This option determines whether or not the we will allow clients to specify nasty (as in |`&><'"\[]{}) characters in arguments.
allow nasty characters = 0

; COMMAND ARGUMENT PROCESSING - This option determines whether or not the we will allow clients to specify arguments to commands that are executed.
allow arguments = 0

; ENABLE SSL ENCRYPTION - This option controls if SSL should be enabled.
use ssl = 1

; EXTENDED RESPONSE - Send more then 1 return packet to allow response to go beyond payload size (requires modified client if legacy is true this defaults to false).
extended response = 1

; ALLOW INSECURE CHIPHERS and ENCRYPTION - Only enable this if you are using legacy check_nrpe client.
insecure = 0



; A list of aliases available. An alias is an internal command that has been "wrapped" (to add arguments). Be careful so you don't create loops (ie check_loop=check_a, check_a=check_loop)
[/settings/external scripts/alias]

; alias_cpu - Alias for alias_cpu. To configure this item add a section called: /settings/external scripts/alias/alias_cpu
alias_cpu = checkCPU warn=80 crit=90 time=5m time=1m time=30s

; alias_cpu_ex - Alias for alias_cpu_ex. To configure this item add a section called: /settings/external scripts/alias/alias_cpu_ex
alias_cpu_ex = checkCPU warn=$ARG1$ crit=$ARG2$ time=5m time=1m time=30s

; alias_disk - Alias for alias_disk. To configure this item add a section called: /settings/external scripts/alias/alias_disk
alias_disk = CheckDriveSize MinWarn=10% MinCrit=5% CheckAll FilterType=FIXED

; alias_disk_loose - Alias for alias_disk_loose. To configure this item add a section called: /settings/external scripts/alias/alias_disk_loose
alias_disk_loose = CheckDriveSize MinWarn=10% MinCrit=5% CheckAll FilterType=FIXED ignore-unreadable

; alias_event_log - Alias for alias_event_log. To configure this item add a section called: /settings/external scripts/alias/alias_event_log
alias_event_log = CheckEventLog file=application file=system MaxWarn=1 MaxCrit=1 "filter=generated gt -2d AND severity NOT IN ('success', 'informational') AND source != 'SideBySide'" truncate=800 unique descriptions "syntax=%severity%: %source%: %message% (%count%)"

; alias_file_age - Alias for alias_file_age. To configure this item add a section called: /settings/external scripts/alias/alias_file_age
alias_file_age = checkFile2 filter=out "file=$ARG1$" filter-written=>1d MaxWarn=1 MaxCrit=1 "syntax=%filename% %write%"

; alias_file_size - Alias for alias_file_size. To configure this item add a section called: /settings/external scripts/alias/alias_file_size
alias_file_size = CheckFiles "filter=size > $ARG2$" "path=$ARG1$" MaxWarn=1 MaxCrit=1 "syntax=%filename% %size%" max-dir-depth=10

; alias_mem - Alias for alias_mem. To configure this item add a section called: /settings/external scripts/alias/alias_mem
alias_mem = checkMem MaxWarn=80% MaxCrit=90% ShowAll=long type=physical type=virtual type=paged type=page

; alias_process - Alias for alias_process. To configure this item add a section called: /settings/external scripts/alias/alias_process
alias_process = checkProcState "$ARG1$=started"

; alias_process_count - Alias for alias_process_count. To configure this item add a section called: /settings/external scripts/alias/alias_process_count
alias_process_count = checkProcState MaxWarnCount=$ARG2$ MaxCritCount=$ARG3$ "$ARG1$=started"

; alias_process_hung - Alias for alias_process_hung. To configure this item add a section called: /settings/external scripts/alias/alias_process_hung
alias_process_hung = checkProcState MaxWarnCount=1 MaxCritCount=1 "$ARG1$=hung"

; alias_process_stopped - Alias for alias_process_stopped. To configure this item add a section called: /settings/external scripts/alias/alias_process_stopped
alias_process_stopped = checkProcState "$ARG1$=stopped"

; alias_sched_all - Alias for alias_sched_all. To configure this item add a section called: /settings/external scripts/alias/alias_sched_all
alias_sched_all = CheckTaskSched "filter=exit_code ne 0" "syntax=%title%: %exit_code%" warn=>0

; alias_sched_long - Alias for alias_sched_long. To configure this item add a section called: /settings/external scripts/alias/alias_sched_long
alias_sched_long = CheckTaskSched "filter=status = 'running' AND most_recent_run_time < -$ARG1$" "syntax=%title% (%most_recent_run_time%)" warn=>0

; alias_sched_task - Alias for alias_sched_task. To configure this item add a section called: /settings/external scripts/alias/alias_sched_task
alias_sched_task = CheckTaskSched "filter=title eq '$ARG1$' AND exit_code ne 0" "syntax=%title% (%most_recent_run_time%)" warn=>0

; alias_service - Alias for alias_service. To configure this item add a section called: /settings/external scripts/alias/alias_service
;alias_service = checkServiceState CheckAll
alias_service_MpsSvc = checkServiceState "MpsSvc=running"

; alias_service_ex - Alias for alias_service_ex. To configure this item add a section called: /settings/external scripts/alias/alias_service_ex
alias_service_ex = checkServiceState CheckAll "exclude=Net Driver HPZ12" "exclude=Pml Driver HPZ12" exclude=stisvc

; alias_up - Alias for alias_up. To configure this item add a section called: /settings/external scripts/alias/alias_up
alias_up = checkUpTime MinWarn=1d MinWarn=1h

; alias_updates - Alias for alias_updates. To configure this item add a section called: /settings/external scripts/alias/alias_updates
alias_updates = check_updates -warning 0 -critical 0

; alias_volumes - Alias for alias_volumes. To configure this item add a section called: /settings/external scripts/alias/alias_volumes
alias_volumes = CheckDriveSize MinWarn=10% MinCrit=5% CheckAll=volumes FilterType=FIXED

; alias_volumes_loose - Alias for alias_volumes_loose. To configure this item add a section called: /settings/external scripts/alias/alias_volumes_loose
alias_volumes_loose = CheckDriveSize MinWarn=10% MinCrit=5% CheckAll=volumes FilterType=FIXED ignore-unreadable 

; default - Alias for default. To configure this item add a section called: /settings/external scripts/alias/default
default = 

I essentially followed the guide as per here: https://support.nagios.com/kb/article.p ... ategory=99

When I run check_nrpe without referencing the certificates I get an error, as expected.
When I do reference the certificates it's all good. So, what am I worried about?

If I run check_nrpe against the Nagios server (192.168.0.9), the messages in /VAR/LOG indicate that certificates are in fact used.
If I run check_nrpe against Server 2008 (192.168.0.4) there is no indication of certificates being referenced.

check_nrpe without reference to certificates:

[root@COTESS-SYSMON ~]# /usr/local/nagios/libexec/check_nrpe -H 192.168.0.9
CHECK_NRPE: Error - Could not complete SSL handshake with 192.168.0.9: 1

[root@COTESS-SYSMON ~]# /usr/local/nagios/libexec/check_nrpe -H 192.168.0.4
CHECK_NRPE: Error - Could not connect to 192.168.0.4. Check system logs on 192.168.0.4

/VAR/LOG/Messages:
Aug 22 13:26:40 COTESS-SYSMON check_nrpe: Error: Could not complete SSL handshake with 192.168.0.4: rc=0 SSL-error=5

check_nrpe with reference to certificates:

[root@COTESS-SYSMON ~]# /usr/local/nagios/libexec/check_nrpe -f /usr/local/nagios/etc/check_nrpe.config -H 192.168.0.9
NRPE vnrpe-3.0
[root@COTESS-SYSMON ~]# /usr/local/nagios/libexec/check_nrpe -f /usr/local/nagios/etc/check_nrpe.config -H 192.168.0.4
I (0.4.4.19 2015-12-08) seem to be doing fine...

/VAR/LOG/Messages:
Aug 22 11:45:43 COTESS-SYSMON nrpe[8746]: Remote 192.168.0.9 - SSL Version: TLSv1.2
Aug 22 11:45:43 COTESS-SYSMON nrpe[8746]: Remote 192.168.0.9 - TLSv1/SSLv3, Cipher is DHE-RSA-AES256-GCM-SHA384
Aug 22 11:45:43 COTESS-SYSMON nrpe[8746]: SSL Client 192.168.0.9 has a valid certificate
Aug 22 11:45:43 COTESS-SYSMON nrpe[8746]: SSL Client 192.168.0.9 Cert Name: /C=AU/ST=Vic/O=CoTe/OU=IT/CN=SYSMON/emailAddress=xxxxxxxxx@xxxxxx.com.au
Aug 22 11:45:43 COTESS-SYSMON nrpe[8746]: SSL Client 192.168.0.9 Cert Issuer: /C=AU/ST=Vic/L=Hawthorn/O=CoTe/OU=IT/CN=SYSMON/emailAddress=xxxxxxxxx@xxxxxx.com.au
Aug 22 11:45:43 COTESS-SYSMON check_nrpe: Remote 192.168.0.9 accepted a Version 3 Packet
Aug 22 11:46:20 COTESS-SYSMON check_nrpe: Remote 192.168.0.4 does not support Version 3 Packets
Aug 22 11:46:20 COTESS-SYSMON check_nrpe: Remote 192.168.0.4 accepted a Version 2 Packet

As you can see, connection to 192.168.0.4 was successful but no mention of certificate validation.
Is something wrong with my config' or is it just not reporting correctly?

Thanks for looking,
Ruffy.

[Box293]

Thanks for the detailed post.

Can you try adding -s 1 to the check_nrpe command, this will log more to /var/log/messages

Code: Select all

/usr/local/nagios/libexec/check_nrpe -f /usr/local/nagios/etc/check_nrpe.config -H 192.168.0.4 -s 1

Can you also add the following to your nsclient.ini file:

Code: Select all

[/settings/log]
; LOG LEVEL - Log level to use. Available levels are error,warning,info,debug,trace
level = debug

Then restart the NSClient++ service. This should log a lot more into nsclient.log.

Let us know if this provides any additional information which proves certificates are being used.

I would like to add these steps to the existing KB article if everything is working as expected.

[ruffy01]

Thank you.

Did as you suggested and...

[root@COTESS-SYSMON ~]# /usr/local/nagios/libexec/check_nrpe -f /usr/local/nagios/etc/check_nrpe.config -H 192.168.0.4 -s 1
I (0.4.4.19 2015-12-08) seem to be doing fine...
/VAR/LOG/messages:
Aug 22 15:19:56 COTESS-SYSMON check_nrpe: SSL Certificate File: /usr/local/nagios/etc/ssl/nagios_server_certs/nagios_server.pem
Aug 22 15:19:56 COTESS-SYSMON check_nrpe: SSL Private Key File: /usr/local/nagios/etc/ssl/nagios_server_certs/nagios_server.key
Aug 22 15:19:56 COTESS-SYSMON check_nrpe: SSL CA Certificate File: /usr/local/nagios/etc/ssl/ca/ca_cert.pem
Aug 22 15:19:56 COTESS-SYSMON check_nrpe: SSL Cipher List: ALL:!MD5:@STRENGTH
Aug 22 15:19:56 COTESS-SYSMON check_nrpe: SSL Allow ADH: Allow
Aug 22 15:19:56 COTESS-SYSMON check_nrpe: SSL Log Options: 0x01
Aug 22 15:19:56 COTESS-SYSMON check_nrpe: SSL Version: TLSv1_plus And Above
Aug 22 15:19:56 COTESS-SYSMON check_nrpe: Remote 192.168.0.4 does not support Version 3 Packets
Aug 22 15:19:56 COTESS-SYSMON check_nrpe: SSL Certificate File: /usr/local/nagios/etc/ssl/nagios_server_certs/nagios_server.pem
Aug 22 15:19:56 COTESS-SYSMON check_nrpe: SSL Private Key File: /usr/local/nagios/etc/ssl/nagios_server_certs/nagios_server.key
Aug 22 15:19:56 COTESS-SYSMON check_nrpe: SSL CA Certificate File: /usr/local/nagios/etc/ssl/ca/ca_cert.pem
Aug 22 15:19:56 COTESS-SYSMON check_nrpe: SSL Cipher List: ALL:!MD5:@STRENGTH
Aug 22 15:19:56 COTESS-SYSMON check_nrpe: SSL Allow ADH: Allow
Aug 22 15:19:56 COTESS-SYSMON check_nrpe: SSL Log Options: 0x01
Aug 22 15:19:56 COTESS-SYSMON check_nrpe: SSL Version: TLSv1_plus And Above
Aug 22 15:19:56 COTESS-SYSMON check_nrpe: Remote 192.168.0.4 accepted a Version 2 Packet

and...

nsclient.log:
2016-08-22 15:19:56: debug:c:\source\nscp\include\nrpe/server/protocol.hpp:74: Accepting connection from: 192.168.0.9, count=1
2016-08-22 15:19:56: debug:c:\source\nscp\include\nrpe/server/protocol.hpp:74: Accepting connection from: 192.168.0.9, count=1

I think that looks better?
Still different and not as conclusive as the result from 192.168.0.9 where these lines were received:
Aug 22 15:15:20 COTESS-SYSMON nrpe[14825]: SSL Client 192.168.0.9 has a valid certificate
Aug 22 15:15:20 COTESS-SYSMON nrpe[14825]: SSL Client 192.168.0.9 Cert Name: /C=AU/ST=Vic/O=CoTe/OU=IT/CN=SYSMON/emailAddress=xxxx
Aug 22 15:15:20 COTESS-SYSMON nrpe[14825]: SSL Client 192.168.0.9 Cert Issuer: /C=AU/ST=Vic/L=Hawthorn/O=CoTe/OU=IT/CN=SYSMON/emailAddress=xxxxxx

Should I be happy with that, or not yet?


Cheers,
Ruffy.

[Box293]

I don't have a conclusive answer, I think that perhaps you need to investigate in the NSClient++ forums.
http://forums.nsclient.org/

A while ago I tried to understand how the certificates worked in NSClient++ and posted this thread:
http://forums.nsclient.org/t/trying-to- ... cates/4157

This was before NRPE v3 and I never received a response.

The only answer is for NSClient++ to produce more output in it's logging, but I'm not sure if there is a more detailed logging option for the certificates. This might need to be a GitHub Issue/Request https://github.com/mickem/nscp

I assume that because NSClient++ accepts your connection when supplying a certificate then it must be working.

There is some NSClient++ documentation for certificates however I could not understand how it all works:
https://www.medin.name/blog/2012/12/02/ ... ntication/

[ruffy01]

I don't have a conclusive answer, I think that perhaps you need to investigate in the NSClient++ forums.
....

I assume that because NSClient++ accepts your connection when supplying a certificate then it must be working.
...

Thanks again.
You've summed up my thoughts exactly. I feel that it is working, but...

I shall go to the NSClient++ forum and investigate a little further.
Naturally I'll report back here with any relevant (or not) findings.

Cheers,
Ruffy.

[bwallace]

Thanks - definitely let us know what you find out, we'll leave this thread open in the meantime.

[ruffy01]

Well, 30 views on the NSClient++ forum, but, alas, no replies as of yet:
https://forums.nsclient.org/t/check-nrp ... ation/4247

[Box293]

There's a lot of interest in it, it can be hard when there is only one developer for a project.

[ruffy01]

One reply on the NSClient++ forum:

"I dont know how the new Nagios check_nrpe works so you have to ask over at Nagios for that."

Oh well.

[tgriep]

I tried to get to get the SSL settings to work with NSCLient++ but I could never get it to work.
Using SSL with NSCLient seems to only work with NSClient talking to NSClient.