plugins to monitor wifi Access points and Controllers

[kimjaggi]

Hi thanks to both of you for explaining the OIDs but none of these let me monitor Access Points of controller which is the most important thing to monitor in wifi.
I ran snmpwalk once again without using IFtable at the end, and it provided me more OIDs. but the output is pretty huge so I sent it in your messages.

Also, Do you think I can use check_cisco_ap.sh by changing the OID? If yes, then what OID would be needed in the script because in the script only one OID has been used.

[kimjaggi]

Somehow it worked by changing some OIDs in check_aruba.pl script. Now I am able to see the output on command line. But now I have a new problem.
I have shut down the server on weekend and started today morning but Now nagiso is not updating web interface. It shows the last check was on 30 april. I restart nagios service and apache service but its still not updating.

/var/log/messages ::
May 5 15:40:05 DEISMNETMON02 nagios: Warning: Return code of 127 for check of service 'PING' on host 'localhost' was out of bounds. Make sure the plugin you're trying to run actually exists.
May 5 15:40:45 DEISMNETMON02 nagios: Warning: Return code of 127 for check of service 'Root Partition' on host 'localhost' was out of bounds. Make sure the plugin you're trying to run actually exists.
May 5 15:41:25 DEISMNETMON02 nagios: Warning: Return code of 127 for check of service 'SSH' on host 'localhost' was out of bounds. Make sure the plugin you're trying to run actually exists.
May 5 15:41:55 DEISMNETMON02 nagios: Warning: Return code of 127 for check of service 'Swap Usage' on host 'localhost' was out of bounds. Make sure the plugin you're trying to run actually exists.
May 5 15:41:59 DEISMNETMON02 nagios: Caught SIGTERM, shutting down...
May 5 15:41:59 DEISMNETMON02 nagios: Successfully shutdown... (PID=5217)
May 5 15:42:00 DEISMNETMON02 nagios: Nagios 3.5.1 starting... (PID=5384)
May 5 15:42:00 DEISMNETMON02 nagios: Local time is Mon May 05 15:42:00 CEST 2014
May 5 15:42:00 DEISMNETMON02 nagios: LOG VERSION: 2.0
May 5 15:42:00 DEISMNETMON02 nagios: Finished daemonizing... (New PID=5385)



Any idea?

[slansing]

Do you have the nagios plugins package installed? (I would think you do) What else did you change on the nagios server? How did you shut down the server? If you did not shut it down safely i.e:

Code: Select all

shutdown -h now

Then you could have caused some damage to the software. What does the web interface look like right now? What is the output of the following:

Code: Select all

tail -30 /usr/local/nagios/var/nagios.log

[kimjaggi]

yes it is installed. I have shut it down properly from idrac (Power Off). here is the output of tail -30 /usr/local/nagios/var/nagios.log

Code: Select all

[1398856681] wproc:   stderr line 01: /bin/sh: /bin/mail: No such file or directory
[1398856681] wproc:   stderr line 02: /usr/bin/printf: write error: Broken pipe
[1398858149] SERVICE ALERT: DEISMSW03;fans;OK;HARD;3;OK: fan1 unit1 normal.  OK: fan2 unit1 normal.  OK: fan1 unit2 normal.  OK: fan2 unit2 normal.  OK: fan1 unit3 normal.  OK: fan2 unit3 normal.  OK: fan1 unit4 normal.  OK: fan2 unit4 normal.
[1398858529] SERVICE NOTIFICATION: nagiosadmin;DEISMSW02;Power Supplies;UNKNOWN;notify-service-by-email;OK: System (ac power) normal.  OK: Main (ac power) normal.  UNKNOWN: Secondary (dc power) not present - please ensure you are monitoring the right device and IP address.  OK: System (ac power) normal.  OK: Main (ac power) normal.  UNKNOWN: Secondary (dc power) not present - please ensure you are monitoring the right device and IP address.
[1398859269] Auto-save of retention data completed successfully.
[1398859350] SERVICE ALERT: DEISMSW03;fans;CRITICAL;SOFT;1;OK: fan1 unit1 normal.  OK: fan2 unit1 normal.  OK: fan1 unit2 normal.  OK: fan2 unit2 normal.  OK: fan1 unit3 normal.  OK: fan2 unit3 normal.  CRITICAL: fan1 unit4 critical! Please shutdown the unit!  CRITICAL: fan2 unit4 critical! Please shutdown the unit!
[1398859469] SERVICE ALERT: DEISMSW03;fans;CRITICAL;SOFT;2;OK: fan1 unit1 normal.  OK: fan2 unit1 normal.  OK: fan1 unit2 normal.  OK: fan2 unit2 normal.  OK: fan1 unit3 normal.  OK: fan2 unit3 normal.  CRITICAL: fan1 unit4 critical! Please shutdown the unit!  CRITICAL: fan2 unit4 critical! Please shutdown the unit!
[1398859589] SERVICE ALERT: DEISMSW03;fans;CRITICAL;HARD;3;OK: fan1 unit1 normal.  OK: fan2 unit1 normal.  OK: fan1 unit2 normal.  OK: fan2 unit2 normal.  OK: fan1 unit3 normal.  OK: fan2 unit3 normal.  CRITICAL: fan1 unit4 critical! Please shutdown the unit!  CRITICAL: fan2 unit4 critical! Please shutdown the unit!
[1398860880] SERVICE NOTIFICATION: nagiosadmin;DEISMSW03;Power Supplies;UNKNOWN;notify-service-by-email;OK: ps1 unit1 (internal power) normal.  UNKNOWN: ps2 unit1 (external power) not present - please ensure you are monitoring the right device and IP address.  OK: ps1 unit2 (internal power) normal.  UNKNOWN: ps2 unit2 (external power) not present - please ensure you are monitoring the right device and IP address.  OK: ps1 unit3 (internal power) normal.  UNKNOWN: ps2 unit3 (external power) not present - please ensure you are monitoring the right device and IP address.  OK: ps1 unit4 (internal power) normal.  UNKNOWN: ps2 unit4 (external power) not present - please ensure you are monitoring the right device and IP address.
[1398862129] SERVICE NOTIFICATION: nagiosadmin;DEISMSW02;Power Supplies;UNKNOWN;notify-service-by-email;OK: System (ac power) normal.  OK: Main (ac power) normal.  UNKNOWN: Secondary (dc power) not present - please ensure you are monitoring the right device and IP address.  OK: System (ac power) normal.  OK: Main (ac power) normal.  UNKNOWN: Secondary (dc power) not present - please ensure you are monitoring the right device and IP address.
[1398862869] Auto-save of retention data completed successfully.
[1398865080] SERVICE NOTIFICATION: nagiosadmin;DEISMSW03;Power Supplies;UNKNOWN;notify-service-by-email;OK: ps1 unit1 (internal power) normal.  UNKNOWN: ps2 unit1 (external power) not present - please ensure you are monitoring the right device and IP address.  OK: ps1 unit2 (internal power) normal.  UNKNOWN: ps2 unit2 (external power) not present - please ensure you are monitoring the right device and IP address.  OK: ps1 unit3 (internal power) normal.  UNKNOWN: ps2 unit3 (external power) not present - please ensure you are monitoring the right device and IP address.  OK: ps1 unit4 (internal power) normal.  UNKNOWN: ps2 unit4 (external power) not present - please ensure you are monitoring the right device and IP address.
[1398865729] SERVICE NOTIFICATION: nagiosadmin;DEISMSW02;Power Supplies;UNKNOWN;notify-service-by-email;OK: System (ac power) normal.  OK: Main (ac power) normal.  UNKNOWN: Secondary (dc power) not present - please ensure you are monitoring the right device and IP address.  OK: System (ac power) normal.  OK: Main (ac power) normal.  UNKNOWN: Secondary (dc power) not present - please ensure you are monitoring the right device and IP address.
[1398866469] Auto-save of retention data completed successfully.
[1398868584] SERVICE ALERT: DEISMCMTEST01;CPU Queue Length;UNKNOWN;SOFT;1;UNKNOWN - Plugin Timed out (15 sec). There are multiple possible reasons for this, some of them include - The host 172.18.2.99 might just be really busy, it might not even be running Windows.
[1398868639] SERVICE ALERT: DEISMCMTEST01;CPU Queue Length;OK;SOFT;2;OK - Average CPU Queue Length 0.1 (20 points with 0 sec delay gives values: 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1)
[1398869280] SERVICE NOTIFICATION: nagiosadmin;DEISMSW03;Power Supplies;UNKNOWN;notify-service-by-email;OK: ps1 unit1 (internal power) normal.  UNKNOWN: ps2 unit1 (external power) not present - please ensure you are monitoring the right device and IP address.  OK: ps1 unit2 (internal power) normal.  UNKNOWN: ps2 unit2 (external power) not present - please ensure you are monitoring the right device and IP address.  OK: ps1 unit3 (internal power) normal.  UNKNOWN: ps2 unit3 (external power) not present - please ensure you are monitoring the right device and IP address.  OK: ps1 unit4 (internal power) normal.  UNKNOWN: ps2 unit4 (external power) not present - please ensure you are monitoring the right device and IP address.
[1398869330] SERVICE NOTIFICATION: nagiosadmin;DEISMSW02;Power Supplies;UNKNOWN;notify-service-by-email;OK: System (ac power) normal.  OK: Main (ac power) normal.  UNKNOWN: Secondary (dc power) not present - please ensure you are monitoring the right device and IP address.  OK: System (ac power) normal.  OK: Main (ac power) normal.  UNKNOWN: Secondary (dc power) not present - please ensure you are monitoring the right device and IP address.
[1398870069] Auto-save of retention data completed successfully.
[1398870389] SERVICE ALERT: DEISMSW03;fans;OK;HARD;3;OK: fan1 unit1 normal.  OK: fan2 unit1 normal.  OK: fan1 unit2 normal.  OK: fan2 unit2 normal.  OK: fan1 unit3 normal.  OK: fan2 unit3 normal.  OK: fan1 unit4 normal.  OK: fan2 unit4 normal.
[1398870989] SERVICE ALERT: DEISMSW03;fans;CRITICAL;SOFT;1;OK: fan1 unit1 normal.  OK: fan2 unit1 normal.  OK: fan1 unit2 normal.  OK: fan2 unit2 normal.  OK: fan1 unit3 normal.  OK: fan2 unit3 normal.  CRITICAL: fan1 unit4 critical! Please shutdown the unit!  CRITICAL: fan2 unit4 critical! Please shutdown the unit!
[1398871109] SERVICE ALERT: DEISMSW03;fans;CRITICAL;SOFT;2;OK: fan1 unit1 normal.  OK: fan2 unit1 normal.  OK: fan1 unit2 normal.  OK: fan2 unit2 normal.  OK: fan1 unit3 normal.  OK: fan2 unit3 normal.  CRITICAL: fan1 unit4 critical! Please shutdown the unit!  CRITICAL: fan2 unit4 critical! Please shutdown the unit!
[1398871229] SERVICE ALERT: DEISMSW03;fans;CRITICAL;HARD;3;OK: fan1 unit1 normal.  OK: fan2 unit1 normal.  OK: fan1 unit2 normal.  OK: fan2 unit2 normal.  OK: fan1 unit3 normal.  OK: fan2 unit3 normal.  CRITICAL: fan1 unit4 critical! Please shutdown the unit!  CRITICAL: fan2 unit4 critical! Please shutdown the unit!
[1398872844] SERVICE ALERT: DEISMCMTEST01;CPU Queue Length;UNKNOWN;SOFT;1;UNKNOWN - Plugin Timed out (15 sec). There are multiple possible reasons for this, some of them include - The host 172.18.2.99 might just be really busy, it might not even be running Windows.
[1398872903] SERVICE ALERT: DEISMCMTEST01;CPU Queue Length;OK;SOFT;2;OK - Average CPU Queue Length 0.1 (20 points with 0 sec delay gives values: 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0)
[1398873480] SERVICE NOTIFICATION: nagiosadmin;DEISMSW03;Power Supplies;UNKNOWN;notify-service-by-email;OK: ps1 unit1 (internal power) normal.  UNKNOWN: ps2 unit1 (external power) not present - please ensure you are monitoring the right device and IP address.  OK: ps1 unit2 (internal power) normal.  UNKNOWN: ps2 unit2 (external power) not present - please ensure you are monitoring the right device and IP address.  OK: ps1 unit3 (internal power) normal.  UNKNOWN: ps2 unit3 (external power) not present - please ensure you are monitoring the right device and IP address.  OK: ps1 unit4 (internal power) normal.  UNKNOWN: ps2 unit4 (external power) not present - please ensure you are monitoring the right device and IP address.
[1398873529] SERVICE NOTIFICATION: nagiosadmin;DEISMSW02;Power Supplies;UNKNOWN;notify-service-by-email;OK: System (ac power) normal.  OK: Main (ac power) normal.  UNKNOWN: Secondary (dc power) not present - please ensure you are monitoring the right device and IP address.  OK: System (ac power) normal.  OK: Main (ac power) normal.  UNKNOWN: Secondary (dc power) not present - please ensure you are monitoring the right device and IP address.
[1398873669] Auto-save of retention data completed successfully.
[1398874704] SERVICE ALERT: DEISMCMTEST01;CPU Queue Length;UNKNOWN;SOFT;1;UNKNOWN - Plugin Timed out (15 sec). There are multiple possible reasons for this, some of them include - The host 172.18.2.99 might just be really busy, it might not even be running Windows.
[1398874761] SERVICE ALERT: DEISMCMTEST01;CPU Queue Length;OK;SOFT;2;OK - Average CPU Queue Length 0.1 (20 points with 0 sec delay gives values: 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0)

[tmcdonald]

DRAC might not properly shut things down. There is a specific set of steps that the server should take in order to ensure the databases get closed safely, files are written, etc. Running "shutdown -r now" is the preferred way to restart a server safely. In addition, slansing had asked what the web interface looked like. Can you please post a screenshot?

[kimjaggi]

I have attached the screenshot of the web interface. It refreshes but nothing updates. The last update is still 30 April. Not sure what is broken.

I have updated/installed some plugins so I don't see "Return Code 127" error anymore in /var/log/messages.
I figured out that nagios.log file is not updating the status is same from yesterday.

Also it doesnt matter if the nagios service is stopped or started- the output on web remains same.

also output of service ndo2db status --> ndo2db is stopped "somehow I cannot restart it and I am not sure how much it is important for nagios core"
Does it gets installed with nagios core? because I only see two databases (mysql and information_schema ). Have I deleted something accidentally from database? Should I install NDoutils?

I have upgraded Nagios from 4.0.4 to 4.0.6 and upgraded plugins to 2.0.1. but that didnt help as well

Additional info:

Code: Select all

 tail -30 /var/log/httpd/error_log
[Tue May 06 14:42:23 2014] [notice] Digest: done
[Tue May 06 14:42:23 2014] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.4.27 configured -- resuming normal operations
[Tue May 06 14:46:37 2014] [notice] caught SIGTERM, shutting down
[Tue May 06 14:47:07 2014] [notice] SELinux policy enabled; httpd running as context unconfined_u:system_r:httpd_t:s0
[Tue May 06 14:47:07 2014] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Tue May 06 14:47:07 2014] [notice] Digest: generating secret for digest authentication ...
[Tue May 06 14:47:07 2014] [notice] Digest: done
[Tue May 06 14:47:07 2014] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.4.27 configured -- resuming normal operations
[Tue May 06 15:46:09 2014] [notice] caught SIGTERM, shutting down
[Tue May 06 15:46:14 2014] [notice] SELinux policy enabled; httpd running as context unconfined_u:system_r:httpd_t:s0
[Tue May 06 15:46:14 2014] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Tue May 06 15:46:14 2014] [notice] Digest: generating secret for digest authentication ...
[Tue May 06 15:46:14 2014] [notice] Digest: done
[Tue May 06 15:46:14 2014] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.4.27 configured -- resuming normal operations
[Tue May 06 15:49:18 2014] [error] [client 172.17.3.131] File does not exist: /var/www/html/pnp4nagios, referer: http://172.18.2.89/nagios/cgi-bin/status.cgi?host=DEISMCMTEST01
[Tue May 06 15:50:09 2014] [notice] caught SIGTERM, shutting down
[Tue May 06 15:50:26 2014] [notice] SELinux policy enabled; httpd running as context unconfined_u:system_r:httpd_t:s0
[Tue May 06 15:50:26 2014] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Tue May 06 15:50:26 2014] [notice] Digest: generating secret for digest authentication ...
[Tue May 06 15:50:26 2014] [notice] Digest: done
[Tue May 06 15:50:26 2014] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.4.27 configured -- resuming normal operations
[Tue May 06 15:51:52 2014] [warn] [client 172.17.3.131] Timeout waiting for output from CGI script /usr/local/nagios/sbin/cmd.cgi, referer: http://172.18.2.89/nagios/cgi-bin/cmd.cgi?cmd_typ=96&host=DEISMSW01&force_check
[Tue May 06 15:51:52 2014] [error] [client 172.17.3.131] Script timed out before returning headers: cmd.cgi, referer: http://172.18.2.89/nagios/cgi-bin/cmd.cgi?cmd_typ=96&host=DEISMSW01&force_check
[Tue May 06 15:51:53 2014] [warn] [client 172.17.3.131] Timeout waiting for output from CGI script /usr/local/nagios/sbin/cmd.cgi, referer: http://172.18.2.89/nagios/cgi-bin/cmd.cgi?cmd_typ=96&host=DEISMSW01&force_check
[Tue May 06 15:51:53 2014] [error] [client 172.17.3.131] Script timed out before returning headers: cmd.cgi, referer: http://172.18.2.89/nagios/cgi-bin/cmd.cgi?cmd_typ=96&host=DEISMSW01&force_check
[Tue May 06 15:51:53 2014] [warn] [client 172.17.3.131] Timeout waiting for output from CGI script /usr/local/nagios/sbin/cmd.cgi, referer: http://172.18.2.89/nagios/cgi-bin/cmd.cgi?cmd_typ=96&host=DEISMSW01&force_check
[Tue May 06 15:51:53 2014] [error] [client 172.17.3.131] Script timed out before returning headers: cmd.cgi, referer: http://172.18.2.89/nagios/cgi-bin/cmd.cgi?cmd_typ=96&host=DEISMSW01&force_check
[Tue May 06 15:52:52 2014] [warn] [client 172.17.3.131] Timeout waiting for output from CGI script /usr/local/nagios/sbin/cmd.cgi, referer: http://172.18.2.89/nagios/cgi-bin/cmd.cgi?cmd_typ=96&host=DEISMSW01&force_check
[Tue May 06 15:52:53 2014] [warn] [client 172.17.3.131] Timeout waiting for output from CGI script /usr/local/nagios/sbin/cmd.cgi, referer: http://172.18.2.89/nagios/cgi-bin/cmd.cgi?cmd_typ=96&host=DEISMSW01&force_check
[Tue May 06 15:52:53 2014] [warn] [client 172.17.3.131] Timeout waiting for output from CGI script /usr/local/nagios/sbin/cmd.cgi, referer: http://172.18.2.89/nagios/cgi-bin/cmd.cgi?cmd_typ=96&host=DEISMSW01&force_chec

[lmiltchev]

Can you disable selinux temporarily just to rule this out as being an issue? What happens if you try to submit a service command to reschedule the next check on this service? Does the "Last Check" time change?

[kimjaggi]

I am sure SElinux has nothing to do with this. It is set to permissive but I can still try if you ask.
No nothing changes on Web Interface. Its frozen. Moreover nagios logs are are also frozen. Nothing updates there as well. So something is really broken,

[abrist]

What are the permissions on the cmd pipe?

Code: Select all

ls -la /usr/local/nagios/var/rw/nagios.cmd

[kimjaggi]

Code: Select all

prw-rw----. 1 nagios nagiocmd 0 Apr 29 14:01 /usr/local/nagios/var/rw/nagios.cmd