Nested FS Monitoring troubles

[upsideofdown]

I'm still VERY new to Nagios and am having some issues with monitoring my filesystem.

The hierarchy of the FS in question looks something like this:
/opt/mount-a
/opt/mount-a/mount-a1
/opt/mount-a/mount-a2


The Disk Check command I'm using for the host1.cfg file looks like this:
# Check Disk Space
define service{
use local-service
host_name host1.corp.local
service_description /
check_command check_local_disk!10%!5%!/
}
define service{
use local-service
host_name host1.corp.local
service_description /var/log
check_command check_local_disk!10%!5%!/var/log
}
define service{
use local-service
host_name host1.corp.local
service_description /opt/mount-a
check_command check_local_disk!10%!5%!/opt/mount-a
}
define service{
use local-service
host_name host1.corp.local
service_description /opt/mount-a/mount-a1
check_command check_local_disk!10%!5%!/opt/mount-a/mount-a1
}
define service{
use local-service
host_name host1.corp.local
service_description /opt/mount-a/mount-a2
check_command check_local_disk!10%!5%!/opt/mount-a/mount-a2
}


And the Command Deffinition in the commands.cfg file looks like this:
# 'check_local_disk' command definition
define command{
command_name check_local_disk
command_line $USER1$/check_disk -w $ARG1$ -c $ARG2$ -p $ARG3$
}


All FS are successfully monitoring EXCEPT /opt/mount-a/mount-a1 and /opt/mount-a/mount-a2. When I execute the command manually on host1, I get exactly the output I am expecting, that the disk was located and reports utilization. However, when viewing from the Nagios console, I see that the FS is reporting as CRITICAL and there is "no such file or directory". Any ideas on what I'm missing?

Thanks in advance.

[sreinhardt]

Does the user(nagios) that is running your agent, have permission to access this mount\directory?

[upsideofdown]

Yes, I have 755 on all directories in question.

[upsideofdown]

One thing worth noting, but I'm not sure if it has any barring on the solution:

Each directory in the FS is mounted on local storage via LVM, as indicated in the FSTAB entries below:
/dev/vg00/lv_mount-a /opt/mount-a ext3 defaults 1 2
/dev/vg00/lv_mount-a1 /opt/mount-a/mount-a1 ext3 defaults 1 2
/dev/vg00/lv_mount-a2 /opt/mount-a/mount-a2 ext3 defaults 1 2

[lmiltchev]

Can you show up the actual command that you are running from the command line and the output of it?

[tmcdonald]

One thing worth noting, but I'm not sure if it has any barring on the solution:

Each directory in the FS is mounted on local storage via LVM, as indicated in the FSTAB entries below:
/dev/vg00/lv_mount-a /opt/mount-a ext3 defaults 1 2
/dev/vg00/lv_mount-a1 /opt/mount-a/mount-a1 ext3 defaults 1 2
/dev/vg00/lv_mount-a2 /opt/mount-a/mount-a2 ext3 defaults 1 2

LVM shouldn't™ make a difference in this case. From the Nagios standpoint, it shouldn't™ matter if you are on one root partition on a single HD or if you have a complicated RAID setup with networked drives. One of our techs is actually trying to replicate the issue on our end, so we might have a better answer for you soon.

[sreinhardt]

OK I'm working to emulate this now and see what happens. I would note that I got standard disk mounts without lvm in place, and it works as expected, providing separate statistics for both the mount points. I am creating some new disks in my vm and will test out lvm too!

[upsideofdown]

Can you show up the actual command that you are running from the command line and the output of it?

Here's what I see when I execute the check_disk command on the system in question:

[root@host1 ~]# /usr/lib64/nagios/plugins/check_disk -w 10% -c 5% -p /opt/mount-a/
DISK OK - free space: /opt/mount-a 382494 MB (99% inode=99%);| /opt/mount-a=198MB;362855;383014;0;403173

[root@host1 ~]# /usr/lib64/nagios/plugins/check_disk -w 10% -c 5% -p /opt/mount-a/mount-a1
DISK OK - free space: /opt/mount-a/mount-a1 143322 MB (99% inode=99%);| /opt/mount-a/mount-a1=187MB;136070;143629;0;151189

[root@host1 ~]# /usr/lib64/nagios/plugins/check_disk -w 10% -c 5% -p /opt/mount-a/mount-a2/
DISK OK - free space: /opt/mount-a/mount-a2 1910216 MB (99% inode=99%);| /opt/mount-a/mount-a2=198MB;1811385;1912018;0;2012651

NOTE:
I'm using the lib64 library as the 32-bit library (/usr/lib/nagios/plugins/check_disk) fails with the following failed module:
[root@host1 ~]# /usr/lib/nagios/plugins/check_disk -w 10% -c 5% -p /opt/mount-a/
-bash: /usr/lib/nagios/plugins/check_disk: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory

[lmiltchev]

All FS are successfully monitoring EXCEPT /opt/mount-a/mount-a1 and /opt/mount-a/mount-a2. When I execute the command manually on host1, I get exactly the output I am expecting, that the disk was located and reports utilization. However, when viewing from the Nagios console, I see that the FS is reporting as CRITICAL and there is "no such file or directory". Any ideas on what I'm missing?

I can see that you can successfully run the check locally:

[root@host1 ~]# /usr/lib64/nagios/plugins/check_disk -w 10% -c 5% -p /opt/mount-a/mount-a1
DISK OK - free space: /opt/mount-a/mount-a1 143322 MB (99% inode=99%);| /opt/mount-a/mount-a1=187MB;136070;143629;0;151189

[root@host1 ~]# /usr/lib64/nagios/plugins/check_disk -w 10% -c 5% -p /opt/mount-a/mount-a2/
DISK OK - free space: /opt/mount-a/mount-a2 1910216 MB (99% inode=99%);| /opt/mount-a/mount-a2=198MB;1811385;1912018;0;2012651

I actually wanted to see the actual command that you run from the command line and the output of it, ran on the Nagios server (not locally). You can also try running it locally as nagios user and show us the output:

Code: Select all

su -l nagios -c '/usr/lib64/nagios/plugins/check_disk -w 10% -c 5% -p /opt/mount-a/mount-a1'
su -l nagios -c '/usr/lib64/nagios/plugins/check_disk -w 10% -c 5% -p /opt/mount-a/mount-a2/'

[upsideofdown]

I actually wanted to see the actual command that you run from the command line and the output of it, ran on the Nagios server (not locally). You can also try running it locally as nagios user and show us the output:

Code: Select all

su -l nagios -c '/usr/lib64/nagios/plugins/check_disk -w 10% -c 5% -p /opt/mount-a/mount-a1'
su -l nagios -c '/usr/lib64/nagios/plugins/check_disk -w 10% -c 5% -p /opt/mount-a/mount-a2/'

That filesystem heiarchy does not exist on the nagios server itself, only on three client servers which are being monitored by the server. That said, I did create a temporary FS structure on the Nagios Monitoring Server as requested and executed the following commands.

[root@nagios-server01 ~]# /usr/lib64/nagios/plugins/check_disk -w 10% -c 5% -p /opt/mount-a/mount-a1
DISK OK - free space: / 3037 MB (40% inode=86%);| /=4556MB;7200;7600;0;8000
[root@nagios-server01 ~]# /usr/lib64/nagios/plugins/check_disk -w 10% -c 5% -p /opt/mount-a/mount-a2
DISK OK - free space: / 3037 MB (40% inode=86%);| /=4556MB;7200;7600;0;8000

Interestingly enough, this is the output I receive when executing as user: nagios:
[root@nagios-server01 ~]# su -l nagios -c '/usr/lib64/nagios/plugins/check_disk -w 10% -c 5% -p /opt/mount-a/mount-a1'
This account is currently not available.