Replaced a firewall, host is now throwing errors

[chris1337c]

From reading into this I suspect it is possible that there was an SSH key that was used on the previous firewall, these are Pfsense firewalls. I had re-uploaded the pfsense config and got the network back up. I am just lacking the knowledge or know how to restore the functionality of these service checks. If someone has any pointers it would be appreciated.

Here is the host config

Code: Select all

define host{
	use		generic-firewall
	host_name	CHI_FW
	alias		CHI_FW
	address		172.16.9.1
	parents		CHC2MAIN
#	hostgroups	firewalls
	}

define service{
        use                             generic-service
        host_name                       CHI_FW
        service_description             Version
        check_command                   check_by_ssh!-H 172.16.9.1 -C "/usr/local/libexec/nagios/check_pf_version"
        notifications_enabled           1
#       hostgroups                   	firewalls
        }

define service{
        use                             generic-service
        host_name                       CHI_FW
        service_description             Uptime
        check_command                   check_by_ssh!-H 172.16.9.1 -C "/usr/local/libexec/nagios/check_pf_uptime"
        notifications_enabled           1
#       hostgroups   	                firewalls
        }

define service{
        use                             generic-service
        host_name                       CHI_FW
        service_description             CPU
        check_command                   check_by_ssh!-H 172.16.9.1 -C "/usr/local/libexec/nagios/check_pf_cpu -w 80 -c 95"
        notifications_enabled           1
#       hostgroups			firewalls
        }

define service{
        use                             generic-service
        host_name                       CHI_FW
        service_description             State Table
        check_command                   check_by_ssh!-H 172.16.9.1 -C "sudo /usr/local/libexec/nagios/check_pf_state_table -w 60 -c 90"
        notifications_enabled           1
#       hostgroups			firewalls
        }


define service{
        use                             generic-service
        host_name                       CHI_FW
        service_description             VPN To Genesis
        check_command                   check_by_ssh!-H 172.16.9.1 -C "sudo /usr/local/libexec/nagios/check_pf_ipsec_tunnel -e ***.**.108.179 -name zCOLO>Genesis"
        notifications_enabled           1
#       hostgroups                      firewalls
        }

define service{
        use                             generic-service
        host_name                       CHI_FW
        service_description             VPN To IND
        check_command                   check_by_ssh!-H 172.16.9.1 -C "sudo /usr/local/libexec/nagios/check_pf_ipsec_tunnel -e **.**.219.130 -name zCOLO>IND"
        notifications_enabled           1
#       hostgroups                      firewalls
        }

define service{
        use                             generic-service
        host_name                       CHI_FW
        service_description             VPN To STL
        check_command                   check_by_ssh!-H 172.16.9.1 -C "sudo /usr/local/libexec/nagios/check_pf_ipsec_tunnel -e **.***.119.74 -name zCOLO>STL"
        notifications_enabled           1
#       hostgroups                      firewalls
        }

define service{
        use                             generic-service
        host_name                       CHI_FW
        service_description             Snort Blocked IP
        check_command                   check_by_ssh!-H 172.16.9.1 -C "sudo pfctl -t snort2c -Ts"
        notifications_enabled           1
#       hostgroups                      firewalls
        }

	

These are the errors on the GUI:

I only had added a few **** in the host I copy dumped above to protect anonymity. These are not present on the actual host config.

CPU
Notifications for this service have been disabled
UNKNOWN 04-08-2019 15:53:29 13d 7h 59m 19s 3/3 Remote command execution failed: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Snort Blocked IP
Notifications for this service have been disabled
UNKNOWN 04-08-2019 15:53:45 13d 7h 57m 21s 3/3 Remote command execution failed: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
State Table
Notifications for this service have been disabled
UNKNOWN 04-08-2019 15:55:44 0d 0h 25m 1s 3/3 Remote command execution failed: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Uptime
Notifications for this service have been disabled
UNKNOWN 04-08-2019 15:57:42 0d 0h 23m 3s 3/3 Remote command execution failed: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
VPN To Genesis
Notifications for this service have been disabled
UNKNOWN 04-08-2019 15:59:41 0d 0h 21m 4s 3/3 Remote command execution failed: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
VPN To IND
Notifications for this service have been disabled
UNKNOWN 04-08-2019 15:51:29 0d 0h 29m 16s 3/3 Remote command execution failed: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
VPN To STL
Notifications for this service have been disabled
UNKNOWN 04-08-2019 15:53:48 13d 7h 57m 18s 3/3 Remote command execution failed: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Version
Notifications for this service have been disabled
UNKNOWN 04-08-2019 15:55:46 13d 7h 55m 20s 3/3 Remote command execution failed: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

[ssax]

It's essentially the same process as this, ignore that it's for XI, you need to get passwordless authentication working (and save the fingerprint when you connect so that it doesn't ask you to save the fingerprint every time the plugin runs in the backend):

https://assets.nagios.com/downloads/nag ... ng_SSH.pdf

[chris1337c]

So I need to create a nagios user for each "new" host I add to the system as a whole correct? Then register the user with the key on the host?

I found the previous IT guy's ssh user's and passwords. I was hesitant to create a new user if that meant breaking all of the other one's.

[chris1337c]

/usr/bin/ssh-copy-id: ERROR: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
ERROR: @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
ERROR: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
ERROR: IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
ERROR: Someone could be eavesdropping on you right now (man-in-the-middle attack)!
ERROR: It is also possible that a host key has just been changed.
ERROR: The fingerprint for the ED25519 key sent by the remote host is
ERROR: SHA256:BTyIyguVWjJcCRt9RMG2lctRkX8i/xBfRaByj9jgxpw.
ERROR: Please contact your system administrator.
ERROR: Add correct host key in /home/nagios/.ssh/known_hosts to get rid of this message.
ERROR: Offending ED25519 key in /home/nagios/.ssh/known_hosts:6
ERROR: ED25519 host key for 172.16.9.1 has changed and you have requested strict checking.
ERROR: Host key verification failed.


When trying to add the key I got this.

[chris1337c]

I was able to get the key added and test the passwordless login. The device went from red to the darker green and is throwing these warnings:

[04-09-2019 12:19:41] SERVICE ALERT: CHI_FW;VPN To Genesis;WARNING;HARD;3;Remote command execution failed: sudo: /usr/local/libexec/nagios/check_pf_ipsec_tunnel: command not found
Service Warning[04-09-2019 12:17:43] SERVICE ALERT: CHI_FW;Uptime;WARNING;HARD;3;Remote command execution failed: /usr/local/libexec/nagios/check_pf_uptime: Command not found.
Service Warning[04-09-2019 12:15:46] SERVICE ALERT: CHI_FW;Version;WARNING;HARD;3;Remote command execution failed: /usr/local/libexec/nagios/check_pf_version: Command not found.
Service Warning[04-09-2019 12:15:45] SERVICE ALERT: CHI_FW;State Table;WARNING;HARD;3;Remote command execution failed: sudo: /usr/local/libexec/nagios/check_pf_state_table: command not found

[chris1337c]

Also I verified against the other nagios boxes that all of the Authorized SSH keys listed on the boxes are all the same. I really believe I have the correct key on here now. I just don't know what to do from here.

[ssax]

Code: Select all

Remote command execution failed: sudo: /usr/local/libexec/nagios/check_pf_ipsec_tunnel: command not found

More than likely you need to put the plugins that it's unable to find back on the new system since you're running them through check_by_ssh.

If these plugins don't exist on the VPN, you will need to put them on their and make sure they work properly like your old ones did:

Code: Select all

/usr/local/libexec/nagios/check_pf_ipsec_tunnel
/usr/local/libexec/nagios/check_pf_uptime
/usr/local/libexec/nagios/check_pf_version
/usr/local/libexec/nagios/check_pf_state_table

Looks like you may have gotten them from here:

https://github.com/oneoffdallas/pfsense-nagios-checks

Additionally, given that it's prepending the command with sudo you will need to make sure you have the proper /etc/sudoers entries added as well.

[chris1337c]

That would make much more sense as I found that these files were not on the nagios box in these locations it is calling, but the other 5 pfsense hosts run them fine. I will have to dig to find this information. I suspect that the "backup" and "restore" may have missed these or some sort of settings associated. I will report back my findings, thank you for the help!

[chris1337c]

#sudo pkg install nagios-plugins

#curl -LO https://github.com/oneoffdallas/pfsense ... master.zip
# sudo unzip -j master.zip -d /usr/local/libexec/nagios/
# sudo chmod +x /usr/local/libexec/nagios/check_pf_*



These must be ran on the Shell of Pfsense in order to function with the CHECK SSH configuration's that are configured on the Nagios Hosts. After running these commands, everything is now working! Thank you for the pointers I appreciate you.

[ssax]

Correct, that's what I was saying, some plugins are meant to be run from the nagios server, some are meant to be run from the remote server.

Thanks for posting your solution! Are we okay to lock the topic and mark it as resolved?