Remote Partition error on web console but not command line

[bwiseman]

Hi,
I have Nagios 4.0.8 installed on the server and am trying to monitor the free space on a remote rhel server. The Nagios plugins and NRPE client has been installed and configured. From the command line everything works. From the web interface only the sda1 check works. Any idea why the web interface on the server cannot see the remote partitions? I have tried referring to them by /dev/.. and by the mount point /local/.. but same problem. Why is sda1 working but sdb1 is not? Clues?

Space on amber /local/store
CRITICAL 10-13-2014 14:14:41 0d 0h 6m 52s 4/4 DISK CRITICAL - /local/store is not accessible: No such file or directory

Space on amber sda1
OK 10-13-2014 14:18:19 0d 0h 0m 14s 1/4 DISK OK - free space: / 98349 MB (99% inode=99%):

Remote Server (Amber)

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/vg_amber-lv_root 51606140 10390772 38593928 22% /
tmpfs 32964068 296 32963772 1% /dev/shm
/dev/sda1 495844 98539 371705 21% /boot
/dev/mapper/vg_amber-lv_home 21645884 176132 20370184 1% /home
/dev/sdb1 5767431768 2203903288 3270559776 41% /local/store
/dev/sdb2 6728670608 6024814268 362059476 95% /local/snapshots

[root@amber etc]# /usr/local/nagios/libexec/check_nrpe -H localhost
NRPE v2.15

[root@ objects]# /usr/local/nagios/libexec/check_nrpe -H amber
NRPE v2.15
[root@ objects]# /usr/local/nagios/libexec/check_nrpe -H amber -c check_disk_snaps
DISK CRITICAL - free space: /local/snapshots 353573 MB (5% inode=97%);| /local/snapshots=5883607MB;5256773;6242418;0;6570967

[root@ objects]# /usr/local/nagios/libexec/check_nrpe -H amber -c check_disk_store
DISK OK - free space: /local/store 3193906 MB (59% inode=99%);| /local/store=2152249MB;4505805;5069031;0;5632257

My config file on the server

define service{
use remote-service ; Name of service
host_name amber
service_description Space on amber /local/store
is_volatile 0
check_period 24x7
max_check_attempts 4
normal_check_interval 5
retry_check_interval 1
contact_groups admins
notification_options w,u,c,r
notification_interval 960
notification_period 24x7
check_command check_nrpe!check_disk_store
}
define service{
use remote-service ; Name of service
host_name amber
service_description Space on amber sda1
is_volatile 0
check_period 24x7
max_check_attempts 4
normal_check_interval 5
retry_check_interval 1
contact_groups admins
notification_options w,u,c,r
notification_interval 960
notification_period 24x7
check_command check_nrpe!check_sda1
}
define service{
use remote-service ; Name of service
host_name amber
service_description Space on /local/snapshots
is_volatile 0
check_period 24x7
max_check_attempts 4
normal_check_interval 5
retry_check_interval 1
contact_groups admins
notification_options w,u,c,r
notification_interval 960
notification_period 24x7
check_command check_nrpe!check_disk_snaps
}

The nrpe cfg file on the server and client

command[check_disk_store]=/usr/local/nagios/libexec/check_disk -w 20% -c 10% -p /local/store
command[check_disk_snaps]=/usr/local/nagios/libexec/check_disk -w 20% -c 5% -p /dev/sdb2

[abrist]

Can you try running the check again on the remote host as the nagios user?
This may be a permissions problem.

[bwiseman]

Yep, tried from the command line on the remote server using the nagios user and it works fine.
The web interface is working ok for all my other servers as well. Just a problem with the one server.

[lmiltchev]

What happens if you change your command on the client from this:

Code: Select all

command[check_disk_snaps]=/usr/local/nagios/libexec/check_disk -w 20% -c 5% -p /dev/sdb2

to this:

Code: Select all

command[check_disk_snaps]=/usr/local/nagios/libexec/check_disk -w 20% -c 5% -p /local/snapshots

and restart nrpe(or xinetd)?

[bwiseman]

using /local/snapshots or /dev/sdb2 fails.

I can successfully check the space on /dev/sda1 on the same machine though! Both sdb partitions fail using either the mount point or the dev mapping. Weird.

-barbara

[sreinhardt]

This may seem a bit odd, but just to be 100% certain, these are both local disks correct? The other thing I wanted to point out at the moment, is that -p is for partitions meaning /dev/sda1 and -M is for mountpoint meaning /local/store. Try the following, on the nrpe server and see if it checks any better:

Code: Select all

/usr/local/nagios/libexec/check_disk -w 20% -c 5% -M /local/store

Also, when you say it is not working from the web interface, is that via the test command button? If so, please save and apply your config and have nagios do a proper check itself instead of using the test command button and see how the interface responds.

[bwiseman]

These are both local disks yes.

Using the -M option still gives 'DISK CRITICAL - /local/store is not accessible: No such file or directory ' on the nagios console
'check_command check_nrpe!check_disk_store'

but from the command line manages to get info.

[root@brad objects]# /usr/local/nagios/libexec/check_nrpe -H amber -c check_disk_store
DISK OK - free space: /dev/sdb1 3183795 MB (59% inode=99%);| /dev/sdb1=2162360MB;4505805;5069031;0;5632257

[slansing]

Is the current service configuration the same as it was when we started this? If not, please show us the updated command_line, Also, the check_nrpe command in commands.cfg

[bwiseman]

Yes, the service cfg is unchanged. All other servers are talking fine. Just this one causing problems. I think it's something to do with the volume labels and the method used to create the volumes... but can't remember if there was anything done differently. The partitions certainly seem to be a mix of logical volumes and partitions. The check on sda1 works fine.

The redhat LVM shows the amber-lv partitions.
The redhat disk utility shows the sdb partitions

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_amber-lv_root 50G 9.5G 38G 21% /
tmpfs 32G 296K 32G 1% /dev/shm
/dev/sda1 485M 97M 363M 21% /boot
/dev/mapper/vg_amber-lv_home 21G 173M 20G 1% /home
/dev/sdb1 5.4T 2.1T 3.1T 41% /local/store
/dev/sdb2 6.3T 5.7T 319G 95% /local/snapshots

check_command check_nrpe!check_disk_store
check_command check_nrpe!check_sda1
check_command check_nrpe!check_sdb1
check_command check_nrpe!check_disk_snaps

command[check_sda1]=/usr/local/nagios/libexec/check_disk -w 20% -c 10% -p /dev/sda1
command[check_sdb1]=/usr/local/nagios/libexec/check_disk -w 20% -c 10% -p /dev/sdb1
command[check_disk_store]=/usr/local/nagios/libexec/check_disk -w 20% -c 10% -p /local/store
command[check_disk_snaps]=/usr/local/nagios/libexec/check_disk -w 20% -c 5% -p /dev/sdb2

[sreinhardt]

Are you having issues with both store and snap? I see /boot, /local/store, and /local/snaps are the only physical partitions. Do you have issues with a remote check of /boot?