Nagios Error: Could not read program status information

[vtthanh9999]

Hi Experts,

I don't know why my Nagios server is down automatically. It worked perfectly before.
The error message of service status page is now:

Error: Could not read host and service status information!

With the detail is:
"The most common cause of this error message (especially for new users), is the fact that Nagios is not actually running. If Nagios is indeed not running, this is a normal error message. It simply indicates that the CGIs could not obtain the current status of hosts and services that are being monitored. If you've just installed things, make sure you read the documentation on starting Nagios.

Some other things you should check in order to resolve this error include:

Check the Nagios log file for messages relating to startup or status data errors.
Always verify configuration options using the -v command-line option before starting or restarting Nagios!
Make sure you read the documentation on installing, configuring and running Nagios thoroughly before continuing. If all else fails, try sending a message to one of the mailing lists. More information can be found at http://www.nagios.org.

Here is the content of log message file (/var/log/nagios/nagios.log):
[1344832901] Warning: Return code of 127 for check of host 'localhost' was out of bounds. Make sure the plugin you're trying to run actually exists.
[1344832901] Warning: Return code of 127 for check of host 'localhost' was out of bounds. Make sure the plugin you're trying to run actually exists.
[1344832981] Warning: Return code of 127 for check of service 'SSH' on host 'localhost' was out of bounds. Make sure the plugin you're trying to run actually exists.
[1344832981] Warning: Return code of 127 for check of service 'Total Processes' on host 'localhost' was out of bounds. Make sure the plugin you're trying to run actually exists.
[1344832981] Warning: Return code of 127 for check of service 'Swap Usage' on host 'localhost' was out of bounds. Make sure the plugin you're trying to run actually exists.
[1344833061] Warning: Return code of 127 for check of service 'Current Users' on host 'localhost' was out of bounds. Make sure the plugin you're trying to run actually exists.
[1344833061] Warning: Return code of 127 for check of service 'Current Load' on host 'localhost' was out of bounds. Make sure the plugin you're trying to run actually exists.
[1344833131] Warning: Return code of 127 for check of service 'PING' on host 'localhost' was out of bounds. Make sure the plugin you're trying to run actually exists.
[1344833131] Warning: Return code of 127 for check of service 'HTTP' on host 'localhost' was out of bounds. Make sure the plugin you're trying to run actually exists.
[1344833171] Warning: Return code of 127 for check of service 'Root Partition' on host 'localhost' was out of bounds. Make sure the plugin you're trying to run actually exists.
[1344833211] Warning: Return code of 127 for check of host 'localhost' was out of bounds. Make sure the plugin you're trying to run actually exists.
[1344833281] Warning: Return code of 127 for check of service 'SSH' on host 'localhost' was out of bounds. Make sure the plugin you're trying to run actually exists.
[1344833281] Warning: Return code of 127 for check of service 'Total Processes' on host 'localhost' was out of bounds. Make sure the plugin you're trying to run actually exists.
[1344833281] Warning: Return code of 127 for check of service 'Swap Usage' on host 'localhost' was out of bounds. Make sure the plugin you're trying to run actually exists.
[1344833361] Warning: Return code of 127 for check of service 'Current Users' on host 'localhost' was out of bounds. Make sure the plugin you're trying to run actually exists.
[1344833361] Warning: Return code of 127 for check of service 'Current Load' on host 'localhost' was out of bounds. Make sure the plugin you're trying to run actually exists.
[1344833431] Warning: Return code of 127 for check of service 'PING' on host 'localhost' was out of bounds. Make sure the plugin you're trying to run actually exists.
[1344833431] Warning: Return code of 127 for check of service 'HTTP' on host 'localhost' was out of bounds. Make sure the plugin you're trying to run actually exists.
[1344833471] Warning: Return code of 127 for check of service 'Root Partition' on host 'localhost' was out of bounds. Make sure the plugin you're trying to run actually exists.
[1344833511] Warning: Return code of 127 for check of host 'localhost' was out of bounds. Make sure the plugin you're trying to run actually exists.
[1344833521] Warning: Return code of 127 for check of host 'localhost' was out of bounds. Make sure the plugin you're trying to run actually exists.
[1344833581] Warning: Return code of 127 for check of service 'SSH' on host 'localhost' was out of bounds. Make sure the plugin you're trying to run actually exists.
[1344833581] Warning: Return code of 127 for check of service 'Total Processes' on host 'localhost' was out of bounds. Make sure the plugin you're trying to run actually exists.
[1344833581] Warning: Return code of 127 for check of service 'Swap Usage' on host 'localhost' was out of bounds. Make sure the plugin you're trying to run actually exists.
[1344833661] Warning: Return code of 127 for check of service 'Current Users' on host 'localhost' was out of bounds. Make sure the plugin you're trying to run actually exists.
[1344833661] Warning: Return code of 127 for check of service 'Current Load' on host 'localhost' was out of bounds. Make sure the plugin you're trying to run actually exists.
[1344833731] Warning: Return code of 127 for check of service 'PING' on host 'localhost' was out of bounds. Make sure the plugin you're trying to run actually exists.
[1344833731] Warning: Return code of 127 for check of service 'HTTP' on host 'localhost' was out of bounds. Make sure the plugin you're trying to run actually exists.
[1344833771] Warning: Return code of 127 for check of service 'Root Partition' on host 'localhost' was out of bounds. Make sure the plugin you're trying to run actually exists

I checked the status_file field from nagios.cfg, it's /usr/lcal/nagios/var/status.dat but I don't see this status.dat file from the path, as I know this file is generated automatically but I alo tried to create this file then restart Nagios service. And the message now is:
Logged in as nagiosadmin
- Notifications are disabled
- Service checks are disabled

Please give me a recommendation to fix this issue. Let me know if you need more information.

Any advice is appreciated.

Thanh Vo

[agriffin]

You should delete the status.dat file you created manually because what we really need is for Nagios to generate it itself. Have you run out of hard disk space? What do you see when you run the 'df' command and the 'df -i' command?

[vtthanh9999]

Thanks Agriffin for your very quick response. I removed the status.dat file as your recommendation. And here is the result of df and df -i command
$ sudo df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda2 39015880 2167776 34866152 6% /
tmpfs 957516 0 957516 0% /dev/shm
/dev/sda1 198337 117988 70109 63% /boot

$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda2 2482176 50264 2431912 3% /
tmpfs 239379 1 239378 1% /dev/shm
/dev/sda1 51200 63 51137 1% /boot

Additional information: I compared with Nagios Server for production environment (the one that is working perfectly now, the failure envionment is Beta) and see that Beta environment is missing 2 plugins are check_ldap and check_ldaps. But my configuration does not use these two plugins, so can it be a problem?

I also re-installed plugins follow a help from http://www.samir.ca/2011/01/16/nagios-r ... xists.html
But it still does not include these two plugins and Nagios server still down with the same error message. How can I get these 2 missing plugins if it's a cause?

Please let me know if you need other information.

Regards,

Thanh Vo

[agriffin]

What do you see when you execute the following command?

Code: Select all

# ls -l /usr/local/nagios/var/retention.dat

[vtthanh9999]

Hi Agriffin,

Here is the result:

$ ls -l /usr/local/nagios/var/retention.dat
-rw------- 1 nagios nagios 208203 Aug 8 16:43 /usr/local/nagios/var/retention.dat

What should I do now?

Thanks & Best regards,

Thanh Vo

[mguthrie]

What do you get in the nagios log when you run:

Code: Select all

/etc/init.d/nagios stop
/etc/init.d/nagios start

[vtthanh9999]

Thanks to Agriffin and Mguthrie,

This is very urgent issue and impeding many tasks of team. So we re-installed all Nagios server, it works well now.
I will let you know if it happens again.

Regards

Thanh Vo