Page 1 of 1
DISK UNKNOWN on AIX
Posted: Mon Oct 01, 2012 3:20 pm
by keith_buckley
Something odd. The check_disk binary is having problems with the shared filesystems on a global logical partition (LPAR). It is a virtual machine that hosts another virtual machine called a workload partition (WPAR).
Code: Select all
nagios@lpar:[/home/nagios] $ /opt/nagios/libexec/check_disk /usr
DISK UNKNOWN - free space:|
nagios@lpar:[/home/nagios] $ /opt/nagios/libexec/check_disk /opt
DISK UNKNOWN - free space:|
When you run the same command on the file systems on the WPAR (real-only copies of the filesystems from the global LPAR), it works okay:
Code: Select all
root@wpar:[/]
# su - nagios
$ /opt/nagios/libexec/check_disk /usr
DISK OK - free space: /usr 1137 MB (22% inode=81%);| /usr=3982MB;;;0;5120
$ /opt/nagios/libexec/check_disk /opt
DISK OK - free space: /opt 1058 MB (51% inode=92%);| /opt=989MB;;;0;2048
Any ideas?
Thank you for any assistance.
Keith A. Buckley
AIX System Administrator
Enterprise Tech Services / Power Systems Team
Cabela’s Inc.
Re: DISK UNKNOWN on AIX
Posted: Mon Oct 01, 2012 4:37 pm
by scottwilkerson
You might want to run something like this to see what exactly the plugin can see
Code: Select all
/opt/nagios/libexec/check_disk -w 80 -c 90 -A
Re: DISK UNKNOWN on AIX
Posted: Wed Oct 03, 2012 10:53 am
by keith_buckley
Scott:
Here is the response:
Code: Select all
nagios@hqjdfapp0:[/home/nagios] $ /opt/nagios/libexec/check_disk -w 80 -c 90 -A /usr
DISK CRITICAL - /syswpars/hqjdfapp1 is not accessible: The file access permissions do not allow the specified action.
hqjdfapp0 - is the global LPAR
hqjdfapp1 - is the WPAR hosted by the global LPAR hqjdfapp0
This is interesting as /syswpars/hqjdfapp1 has nothing to do with /usr. It is the root directory for the WPAR which is hosted by the global LPAR.
I changed the permissions for /syswpars/hqjdfapp1 from 755 to 777, but it had no effect.
Re: DISK UNKNOWN on AIX
Posted: Wed Oct 03, 2012 12:05 pm
by slansing
Did the solution you found to your other thread solve this issue as well?
Re: DISK UNKNOWN on AIX
Posted: Thu Oct 04, 2012 12:57 pm
by keith_buckley
No, but getting NRPE working on the WPAR allows us to monitor the shared filesystems (/usr and /opt) from a WPAR. However, that is not optimal.
Now, we have a similar but bigger problem. For a long time, we couldn't get net-snmp working. Today I figured out how to fix it, but now we have the DISK UNKNOWN problem across multiple hosts. I will quote an email chain describing the problem:
Well, this is what I have so far. We know that the snmpwalk query works on host3:
# /usr/linux/bin/snmpwalk -v 2c -c 7mm4elk host3 | more
SNMPv2-MIB::sysDescr.0 = STRING: AIX host3 1 6 00CE78944C00
SNMPv2-MIB::sysObjectID.0 = OID: SNMPv2-SMI::org
DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (104220531) 12 days, 1:30:05.31
…
Here are the snmp-related processes running on host3:
# ps -ef | grep snmp
root 8519774 1 0 Sep 21 - 2:25 /opt/freeware/sbin/snmptrapd -Lsd -p /var/run/snmptrapd.pid
root 29884562 1 0 Sep 21 - 1:37 /opt/freeware/net-snmp/sbin/snmpd
root 49938674 44368096 0 16:17:58 pts/0 0:00 grep snmp
Notice that the snmpd daemon is being executed from /opt/freeware/net-snmp/sbin. This is not the snmpd installed as part of the NRPE package, which is /opt/freeware/sbin/snmpd.
On host2, I had both of the snmp-related processes running:
# ps -ef | grep snmp
root 13369474 1 0 16:08:43 - 0:00 /opt/freeware/sbin/snmptrapd -Lsd -p /var/run/snmptrapd.pid
root 13631564 14549150 0 16:49:43 pts/0 0:00 grep snmp
root 15007866 1 0 16:36:35 - 0:00 /opt/freeware/sbin/snmpd -Lsd -Lf /dev/null -p /var/run/snmpd.pid -a
Note that the location of snmpd is different and is running with different options.
However the snmpwalk query did not work:
# /usr/linux/bin/snmpwalk -v 2c -c 7mm4elk host2
Timeout: No Response from host2
I tried completely un-installing NRPE from host2 and re-installing, but no change in the results. Net-snmp still ended up in /opt/freeware/sbin. The /opt/freeware/etc/snmp/snmpd.conf files were exactly the same as the ones on host3.
I then copied over the /opt/freeware/net-snmp/share/snmp/snmpd.conf file from host3 to /opt/freeware/etc/snmp/snmpd.conf on host2, recycled the snmpd daemons, and got this:
# /usr/linux/bin/snmpwalk -v 2c -c 7mm4elk host2 | more
SNMPv2-MIB::sysDescr.0 = STRING: AIX host2 1 6 00CE78944C00
SNMPv2-MIB::sysObjectID.0 = OID: NET-SNMP-MIB::netSnmpAgentOIDs.15
DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (16026) 0:02:40.26
SNMPv2-MIB::sysContact.0 = STRING: root@localhost
SNMPv2-MIB::sysName.0 = STRING: host2
…
So, whatever information is in the /opt/freeware/net-snmp/share/snmp/snmpd.conf file from host3 seems to work.
What do you all think?
Subject: RE: SNMP on AIX hosts
Negative:
Subject: SNMP on AIX hosts
I am trying to figure out what is the status of SNMP in our environment, and I would greatly appreciate your assistance. Currently, we have 11 our hosts running the native IBM SNMP daemon and 15 running a freeware version.
A good example is the following: two hosts, same OS level, same configuration, same purpose, one is Prod and one DevTest.
The production host is host1, and it runs the IBM SNMP:
root@host1:[/]
# ps -ef | grep snmp
+0 4915284 4653308 0 Aug 01 - 0:29 /usr/sbin/snmpd
root 14352494 3670180 0 Aug 31 - 0:15 /usr/sbin/snmpd
The DevTest host is host2, and it runs the freeware SNMP:
root@host2:[/]
# ps -ef | grep snmp
root 3276952 1 0 Sep 28 - 0:51 /opt/freeware/sbin/snmpd -Lsd -Lf /dev/null -p /var/run/snmpd.pid -a
root 5308584 1 0 Sep 28 - 1:07 /opt/freeware/sbin/snmptrapd -Lsd -p /var/run/snmptrapd.pid
Can you currently run SNMP queries against both hosts?
However, when I try to run your test command on one of the filesystems showing up as a problem (/home on host3 discussed above), I get an okay response:
Code: Select all
$ /opt/nagios/libexec/check_disk -w 80 -c 90 -A /home
DISK CRITICAL - free space: / 467 MB (45% inode=88%); /usr 3174 MB (39% inode=90%); /var 1269 MB (61% inode=96%); /tmp 1291 MB (63% inode=99%); /home 211 MB (20% inode=83%); /admin 127 MB (99% inode=99% ); /proc 0 MB (0% inode=-); /opt 1048 MB (51% inode=91%); /var/adm/ras/livedump 255 MB (99% inode=99% ); /app01 18740 MB (26% inode=94%); /orawork 42469 MB (69% inode=99%); /proj 93141 MB (90% inode=99%) ; /cabsatmp 1149 MB (28% inode=99%); /u02 29125953 MB (26% inode=99%); /cabsa 1221609 MB (20% inode=9 9%);| /=556MB;944;934;0;1024 /usr=4889MB;7984;7974;0;8064 /var=778MB;1968;1958;0;2048 /tmp=756MB;1968 ;1958;0;2048 /home=812MB;944;934;0;1024 /admin=0MB;48;38;0;128 /proc=2147483647MB;2147483647;21474836 47;0;2147483647 /opt=999MB;1968;1958;0;2048 /var/adm/ras/livedump=0MB;176;166;0;256 /app01=52939MB;71 600;71590;0;71680 /orawork=18970MB;61360;61350;0;61440 /proj=9258MB;102320;102310;0;102400 /cabsatmp= 2946MB;4016;4006;0;4096 /u02=81770332MB;110896205;110896195;0;110896285 /cabsa=4653328MB;6189229;6189 219;0;6189309
Weird.
Keith A. Buckley
AIX System Administrator
Enterprise Tech Services / Power Systems Team
Cabela’s Inc.
Re: DISK UNKNOWN on AIX
Posted: Thu Oct 04, 2012 1:36 pm
by scottwilkerson
keith_buckley wrote:Scott:
Here is the response:
Code: Select all
nagios@hqjdfapp0:[/home/nagios] $ /opt/nagios/libexec/check_disk -w 80 -c 90 -A /usr
DISK CRITICAL - /syswpars/hqjdfapp1 is not accessible: The file access permissions do not allow the specified action.
hqjdfapp0 - is the global LPAR
hqjdfapp1 - is the WPAR hosted by the global LPAR hqjdfapp0
This is interesting as /syswpars/hqjdfapp1 has nothing to do with /usr. It is the root directory for the WPAR which is hosted by the global LPAR.
I changed the permissions for /syswpars/hqjdfapp1 from 755 to 777, but it had no effect.
The -A flag I has you set was to try to see all available partitions, it was not intended to be run with a specific directory.
keith_buckley wrote:Notice that the snmpd daemon is being executed from /opt/freeware/net-snmp/sbin. This is not the snmpd installed as part of the NRPE package, which is /opt/freeware/sbin/snmpd.
Now I'm confused because as far as I know, the NRPE package doesn't install snmpd...
Re: DISK UNKNOWN on AIX
Posted: Thu Oct 04, 2012 3:48 pm
by keith_buckley
We install it as part of our standard install NRPE package as it is a pre-requisite (along with a bunch of other stuff). I created a script to install our NRPE agents:
Code: Select all
#!/bin/ksh
#set -x
# Title: run-npre-install.sh
# Created By: Keith Buckley
# Purpose: Script to install the Nagios NPRE AIX agent and all prerequisites
# It is designed to be run locally on the install host as root
# Date: August 17, 2012
#Install all prequisite software packages
rpm -Uvh ./prereqs/fping-3.2-1.aix6.1.ppc.rpm
rpm -Uvh ./prereqs/gettext-0.10.40-8.aix5.2.ppc.rpm
rpm -Uvh ./prereqs/libsigsegv-2.10-1.aix5.2.ppc.rpm
rpm -Uvh ./prereqs/info-4.13a-2.aix5.1.ppc.rpm
rpm -Uvh ./prereqs/readline-6.2-4.aix5.1.ppc.rpm
rpm -Uvh ./prereqs/libsigsegv-2.10-1.aix5.2.ppc.rpm
rpm -Uvh ./prereqs/gawk-4.0.1-1.aix6.1.ppc.rpm
rpm -Uvh ./prereqs/openssl-1.0.1c-1.aix6.1.ppc.rpm
rpm -Uvh ./prereqs/net-snmp-5.7.1-1.aix6.1.ppc.rpm
rpm -Uvh ./prereqs/libgcc-4.7.1-2.aix6.1.ppc.rpm
rpm -Uvh ./prereqs/gdbm-1.10-1.aix5.1.ppc.rpm
rpm -Uvh --force ./prereqs/perl-5.8.8-2.aix5.1.ppc.rpm
rpm -Uvh ./prereqs/net-snmp-perl-5.7.1-1.aix6.1.ppc.rpm
rpm -Uvh ./prereqs/net-snmp-utils-5.7.1-1.aix6.1.ppc.rpm
rpm -Uvh ./prereqs/openldap-2.4.23-0.3.aix6.1.ppc.rpm
rpm -Uvh ./prereqs/openssl-1.0.1c-1.aix6.1.ppc.rpm
rpm -Uvh ./prereqs/gmp-5.0.5-1.aix5.1.ppc.rpm
rpm -Uvh ./prereqs/zlib-1.2.7-1.aix5.1.ppc.rpm
rpm -Uvh ./prereqs/expat-2.1.0-1.aix5.1.ppc.rpm ./prereqs/expat-devel-2.1.0-1.aix5.1.ppc.rpm
rpm -Uvh ./prereqs/sqlite-3.7.13-1.aix5.1.ppc.rpm
rpm -Uvh ./prereqs/db4-4.7.25-2.aix5.1.ppc.rpm
rpm -Uvh ./prereqs/python-2.6.7-1.aix6.1.ppc.rpm ./prereqs/python-devel-2.6.7-1.aix5.1.ppc.rpm
rpm -Uvh ./prereqs/python-libs-2.6.7-1.aix5.1.ppc.rpm
#Run the NPRE install package
./fullinstall <<EOF
y
xxx.xxx.xxx.xxx
EOF
#Make entry in /etc/inittab to start agent automatically
mkitab nrpe:2:once:"startsrc -s nrpe >/dev/null 2>&1"
#Update permissions on /opt/nagios
chmod 755 /opt/nagios
#end