Monitor Isilon with Nagios XI

[SilvesterDU]

I did before my fist try. Anyway I tried your suggestion:
1. Removed the line and re-run. No error and no warning. This is expected. As there is no reference to my isilon-services.cfg.
2. Add the line back and re-run. Same error.

I'm considering re-deploying an Nagios XI instance, if it is the last resort before I gave up...


Silvester

[jdalrymple]

As there is no reference to my isilon-services.cfg.

That is not correct. In your nagios.cfg:

Code: Select all

# STATIC OBJECT DEFINITIONS (THESE DON'T GET EXPORTED/IMPORTED BY NAGIOSQL)
cfg_dir=/usr/local/nagios/etc/static

When you restarted and got no error you should have looked in your interface. It's likely you didn't get an error because things simply were working. This is why I had you remove the line that effectively duplicated the host.

[SilvesterDU]

As there is no reference to my isilon-services.cfg.

That is not correct. In your nagios.cfg:

Code: Select all

# STATIC OBJECT DEFINITIONS (THESE DON'T GET EXPORTED/IMPORTED BY NAGIOSQL)
cfg_dir=/usr/local/nagios/etc/static

When you restarted and got no error you should have looked in your interface. It's likely you didn't get an error because things simply were working. This is why I had you remove the line that effectively duplicated the host.

OK, you're right. BTW, the cfg_dir line is born with nagios.cfg. So the lesson here is we don't need to add cfg_file in the nagios.cfg if the config file is located in the static folder.

At least we are making progress. Now the error is host not reachable.

Nagios_networkstatus.png

In my environment, the Nagios appliance is not located in the same network as Isilon, so I use another machine as gateway. The pic below shows I can ping Isilon or traceroute from within the Nagios appliance through another hop.

Nagios_networkstatus_ping.png

Nagios_tracerout.png

Any suggestions how to resolve this host unreachable issue?

Thanks a lot!
Silvester

[jolson]

One option is to add a static route to your Nagios box so that it's aware of the Isilon network at all times (as opposed to needing redirection). I assume the ping command can not handle the redirection taking place properly.

Code: Select all

route add -net 10.6.120.0 netmask 255.255.255.0 gw 192.168.102.101 dev eth0

Does the above work properly? You simply need to tell Nagios which gateway to use in the case of attempted access to the 10.6.120.0/24 network. I am almost certain that the gateway I have listed in the above command is incorrect, so be sure to use the proper gateway there. I am also assuming that your 10.6.120.163 network is using a /24 subnet mask, so make changes as necessary to the subnet mask as well. In addition to those assumptions, I am guessing that your primary interface is 'eth0' - be sure to verify that as well.

To set this route persistently (across reboots), you will need to edit your network interface script:

Code: Select all

vi /etc/sysconfig/network-scripts/route-eth0

Add:
GATEWAY0=192.168.102.101
NETMASK0=255.255.255.0
ADDRESS0=10.6.120.0

Once you are capable of reaching the Isilon device without requiring a redirection - let us know. Be sure to be careful with these settings, as I've made a lot of assumptions above (gateway, netmask, device, etc).

[SilvesterDU]

Thank you for the hint-- actually I takes a safer way: instead of modifying the network setting of Nagios appliance, I added an network interface on my Isilon nodes and ensure that interface is in the same subnet of Nagios.

Now the host is shown as "up".

Nagios_hostup2.png

Then I noticed all the services is showing "Nagios has not received an update from rtpsolisilon-1 for at least 25 hours".

Nagios_serviceDetails_issue.png

Also when I checked the "isilon-checks.log" (Isilon check log from /libexec/ce/isilon-checks.sh), it reports "Remote command execution failed: Host key verification failed."

Could these 2 issues be related? I attached the isilon-checks.sh script as well.

You have been very helpful, thanks again.

Silvester

[ssax]

Try running these commands to save the SSH key:

Code: Select all

su - nagios
ssh user@isilonaddress

Then accept the SSH key and exit and try again.

[SilvesterDU]

I've tried but seems the same result:(


Silvester

[jdalrymple]

Whose cron did you add this to?

Code: Select all

*/5 * * * * /opt/nagios/libexec/ce/isilon-checks.sh > /opt/nagios/var/isilon-checks.log

Whoever that is will need to be able to ssh to the isilon without any keyboard interaction I suspect. That's the debugging step I'd take next.

[SilvesterDU]

Whose cron did you add this to?

Code: Select all

*/5 * * * * /opt/nagios/libexec/ce/isilon-checks.sh > /opt/nagios/var/isilon-checks.log

Whoever that is will need to be able to ssh to the isilon without any keyboard interaction I suspect. That's the debugging step I'd take next.

I add to the cron of root user, and root can ssh to the Isilon without keyboard interaction.

So, we need to add the account "nagios" to Isilon as well so that nagios can ssh to Isilon without any password/pass-phrase?

I will have a try and let you know the result

Thanks
Silvester

[ssax]

Sounds good, let us know the results.