No Update on Nagios core UI.

[sattanathan]

Hi,

I was monitoring a esx using check_esx3.pl plugin for few days. it was working fine. some 7 days back some user permission was changed in ESX. so i was getting unknown error in both command line and in UI. But yesterday, the user permission was provided and i am able to fetch the output in command line. But the output is not getting updated in nagios core UI. (Still displaying the old output in status information). I have attached the screenshots of the same. Please help me to make the UI update.

Thanks ,
Sattanathan.S

[sattanathan]

Hi,

I have tried to fix the issue. Meanwhile i found the checks for this host and service are scheduled but not really being executed . I could not find the logs related to this host and service of that host. Only for this esx server i am facing this issue.. Other monitoring are working fine. Only for one host the checks are not happening. Please help me out in fixing this..

I have restarted nagios service as well.. but still the checks are not happening only for this host.

Thanks,
Sattanathan.S

[abrist]

Could you post the output of:

Code: Select all

cat /usr/local/nagios/var/nagios.log | grep  esx5

[sattanathan]

Hi Abrist,

I found these logs before 10 days and there is no log being written now. Attached the same log for your reference.

I have done tail -f nagios.log | grep mmkndhaesx5 for 3 hours but i could not see even a single log.

This means the checks for this service is not really being happen i think. But other host and service checks are happening properly. Checks are not being executed only for this host. Please help me in finding a solution for this.

Thanks,
Sattanathan.S

[abrist]

Verify the Nagios configuration and report back:

Code: Select all

/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

[sattanathan]

Hi Abrist,

I am verifying the configuration everytime before i restart the nagois service. There is no error or warning message.

Today i found some logs for this esx at exactly midnight(Jan 15 00:00:00). Everyday at the midnight there is a log for this esx with the incorrect output and same is being displayed in the UI. After the check at midnight there is no further checks happening.(No logs for the ESX).

I have left a session for the whole day with the command(tail -f nagios.log | grep mmkndhaesx5) being executed from which i found the check has happened at midnight.

Please suggest a solution to overcome this ghost.

Thanks,
Sattanathan.S

[abrist]

Could you post those logs concerning ESX that happen at midnight?

[sattanathan]

Hi ,

This the log i can found for the whole day.

[root@mmkndnagxi servers]# tail -1000000 /usr/local/nagios/var/nagios.log | grep mmkndhaesx5 | perl -pe 's/(\d+)/localtime($1)/e'
[Wed Jan 16 00:00:00 2013] CURRENT HOST STATE: mmkndhaesx5;UP;HARD;1;OK - 10.2.203.68: rta 0.178ms, lost 0%
[Wed Jan 16 00:00:00 2013] CURRENT SERVICE STATE: MMKNDHAESX5;Datastore usage for VMHost;CRITICAL;HARD;3;CHECK_ESX3.PL CRITICAL - Storages : 'Storage1 (6)' is not accessible, 'mmkndsrmdep' is not accessible, 'Web media' is not accessible, 'mmkndhaesx5' is not accessible
[Wed Jan 16 00:00:00 2013] CURRENT SERVICE STATE: mmkndesx01;Datastore usage for VMHost;OK;HARD;1;CHECK_ESX3.PL OK - Storages : 'mmkndhaesx5'(free)=46150.00 MB (6.44%), 'datastore1 (7)'(free)=30899.00 MB (22.90%)
[Wed Jan 16 00:00:00 2013] CURRENT SERVICE STATE: mmkndhaesx5;CPU Usage for VMHost;UNKNOWN;HARD;3;CHECK_ESX3.PL UNKNOWN - HOST CPU Unknown error
[Wed Jan 16 00:00:00 2013] CURRENT SERVICE STATE: mmkndhaesx5;Datastore usage for VMHost;CRITICAL;HARD;3;CHECK_ESX3.PL CRITICAL - Storages : 'Storage1 (6)' is not accessible, 'mmkndsrmdep' is not accessible, 'Web media' is not accessible, 'mmkndhaesx5' is not accessible
[Wed Jan 16 00:00:00 2013] CURRENT SERVICE STATE: mmkndhaesx5;Memory for VMHost;UNKNOWN;HARD;3;CHECK_ESX3.PL UNKNOWN - HOST MEM Unknown error
[Wed Jan 16 00:00:00 2013] CURRENT SERVICE STATE: mmkndhaesx5;Networking for VMHost;CRITICAL;HARD;3;ESX31 CRITICAL - all 3 NICs are connected
[Wed Jan 16 00:00:00 2013] CURRENT SERVICE STATE: mmkndhaesx5;Services for VMHost;OK;HARD;1;CHECK_ESX3.PL OK - services : ntpd (up), sshd (up), vmware-vpxa (up), vmware-webAccess (up)
[Wed Jan 16 00:00:00 2013] CURRENT SERVICE STATE: mmkndhaesx5;UPTIME for VMHost;UNKNOWN;HARD;3;CHECK_ESX3.PL UNKNOWN - HOST UPTIME Unknown error
[Wed Jan 16 00:00:00 2013] CURRENT SERVICE STATE: mmkndhaesx5;VM Status for VMHost;OK;HARD;1;CHECK_ESX3.PL OK - 5/5 VMs up, overall status=red, connection state=notResponding, maintenance=no, All 46 health checks are Green, no config issues

After this time till now there were no checks happened for this and no logs also.. But in UI only Last Check field and Duration field getting updated.( Nagios i exactly scheduling the checks for this host but the checks are not really done and i could not find the log).

Output returned when executed in command line: (I am getting the correct output when executing through command line)

[root@mmkndnagxi servers]# cd ../../../libexec/
[root@mmkndnagxi libexec]# ./check_esx3.pl -H 10.2.203.68 -f /usr/local/nagiosxi/etc/components/vmware/mmkndesx4_auth.txt -l CPU -s usage -w 75 -c 90
CHECK_ESX3.PL OK - cpu usage=10.25 % | cpu_usage=10.25%;75;90
[root@mmkndnagxi libexec]# ./check_esx3.pl -H 10.2.203.68 -f /usr/local/nagiosxi/etc/components/vmware/mmkndesx4_auth.txt -l MEM -s usage -w 75 -c 90
CHECK_ESX3.PL OK - mem usage=64.45 % | mem_usage=64.45%;75;90
[root@mmkndnagxi libexec]# ./check_esx3.pl -H 10.2.203.68 -f /usr/local/nagiosxi/etc/components/vmware/mmkndesx4_auth.txt -l VMFS -w 75 -c 90
CHECK_ESX3.PL CRITICAL - Storages : 'Storage1 (10)'(free)=36934.00 MB (26.67%), 'Web media'(free)=1123.00 MB (0.52%), 'mmkndhaesx5'(free)=58442.00 MB (8.16%), 'mmkndsrmdep'(free)=1192.00 MB (0.69%) | 'Storage1 (10)'=36934.00MB;75;90 'Web media'=1123.00MB;75;90 mmkndhaesx5=58442.00MB;75;90 mmkndsrmdep=1192.00MB;75;90

Thanks,
Sattanathan.S

[abrist]

This is odd behavior. Lets make sure you don't have more than one nagios process running.

Code: Select all

service nagios stop
killall -9 nagios
service nagios start

[sattanathan]

This was the first step i did when i looked at this error. Restarted nagios service several times and also verified only one instance of Nagios is running. But still facing the issue. Please let me know if there are any other thing to be checked.

Thanks..