Incorrect Date Issue

[Box293]

After upgrading to 1.2 I had a timezone issue http://support.nagios.com/forum/viewtop ... t=10#p2121.

In the process of fixing that problem I accidently set the date forward to 2010-08-07.

Even though the problem was correctly in a manner of minutes, it appears all service checks were performed in the future.
If I "Schedule an immediate check" then the next check date changes to the real date/time. This service continues to get back to a regular schedule.

Service Status Last Check.png

Now it appears that no checks are being performed, the server sees that the next check is in August.

The seems likely as I can see the Nagios XI server isn't consuming nearly as much CPU since the accident.

Nagios XI CPU Usage past 2 days.png

How do I force all hosts/services to perform a check immediately?

If there is no quick/easy solution I can revert back to the VMware snapshot I made before I installed the upgrade (and not make the same mistake again)

[tonyyarusso]

Usually this type of problem should correct itself if you leave it overnight or so - let us know if that works in your case.

[mmestnik]

Stop and start nagios core, you might also need to remove or edit the retention data.

• /usr/local/nagios/var/retention.dat

You could also send massive commands using a script(awk being my language of choice for this task).

Code: Select all

< /usr/local/nagios/var/objects.cache awk -F '\t' '
BEGIN {dat=systime(); l=1};
/^define service/ {l=0}
/^\thost_name/ {hostname=$3; if (l) { print dat " SCHEDULE_HOST_CHECK;" $3 ";" systime(); } }
/^\tservice_description/ {print dat " SCHEDULE_SVC_CHECK;" hostname ";" $3 ";" systime(); }
' > /usr/local/nagios/var/rw/nagios.cmd

[Box293]

Usually this type of problem should correct itself if you leave it overnight or so - let us know if that works in your case.

10 more hours have past and the services all still think their next check is in the future. By now it's been about 34 hours since the accident.

I will try mmestnik's suggestions.

I did this
/etc/rc.d/init.d/nagios stop
/etc/rc.d/init.d/nagios start
and waited 10 minutes. No change.

I created the script and ran it. It seemed to run in about 5 seconds with no output.
I waited 10 minutes. No change.
I did a stop / start again. No change.

I did this
/etc/rc.d/init.d/nagios stop
mv /usr/local/nagios/var/retention.dat /usr/local/nagios/var/retention.dat.old
/etc/rc.d/init.d/nagios start

And this fixed the problem. All of the services had a last check time of 1999-11-30 00:00:00.
Nagios then went about performing all of it's checks again and all is working OK.

Thanks for helping fix my problem.

[mmestnik]

I have no objection to adding output.

Code: Select all

< /usr/local/nagios/var/objects.cache awk -F '\t' '
BEGIN {dat="[" systime() "]"; l=1};
/^define service/ {l=0}
/^\thost_name/ {hostname=$3; if (l) { print dat " SCHEDULE_HOST_CHECK;" $3 ";" systime(); } }
/^\tservice_description/ {print dat " SCHEDULE_SVC_CHECK;" hostname ";" $3 ";" systime(); }
' tee -a /usr/local/nagios/var/rw/nagios.cmd

The only change is the last line where we input the commands into Nagios. You know now that I think about it, this is likely the wrong format. I changed that as well, this is a new script.

This is documented here http://www.nagios.org/development/apis/externalcommands.

[aaldrich]

I have a similar problem which is as yet unresolved.

After installing the virtual appliance in vmware vsphere environment I noticed that the server time was in the Central Time Zone
so changed it to be Pacific

Now, everything I do in terms of defining check commands, services, adding hosts seems to the system as if it happens 2 hours in the future,
so it always takes at least two hours for events to catch up with no changes before things will sync,
and my configs are constantly warning me that things are out of sync.

Anybody had this issue and been able to resolve it?

Just some more on this
all of the service checks are always 2 hours ahead of the current time
If I look at the "last checked" for instance, it is 2 hours ahead of the "service status summary" which is
the current system time

[mmestnik]

Did you stop/start all of the services as part of changing the time zone? I bet Nagios Core is still running in CST and the php is converting GMT to PST. This explains why all the time stamps are adjusted, CST -> GMT -> PST. If instead it's PST -> GMT -> PST then the time dose not shift after the conversion.

Please describe how(the steps) you changed the time zone as if you did reboot afterwards(as many guided tutorials would instruct) my above theory holds no water.

[lccouncil]

Mate,

I wrote a script that checks all the hosts on our box..
It goes like this..

Code: Select all

# This is a sample shell script showing how you can submit the SCHEDULE_FORCED_HOST_CHECK command
# to Nagios.  Adjust variables to fit your environment as necessary.

now=`date +%s`
commandfile='/usr/local/nagios/var/rw/nagios.cmd'


printf "[%lu] SCHEDULE_FORCED_HOST_CHECK;HOST1;1110741500\n" $now > $commandfile
printf "[%lu] SCHEDULE_FORCED_HOST_CHECK;HOST2;1110741500\n" $now > $commandfile

Where HOST1 and HOST2 are sequential names of all the hosts in your installation.
You could prbably get really fancy and start using arrays and stuff.. But i needed a quick fix.

Dave

[lccouncil]

Have re-written the script..

Code: Select all

# This is a sample shell script showing how you can submit the SCHEDULE_FORCED_HOST_CHECK command
# to Nagios.  Adjust variables to fit your environment as necessary.

now=`date +%s`
commandfile='/usr/local/nagios/var/rw/nagios.cmd'
hosts=("host1" "host2")

for host in ${hosts[@]}
do
#echo $host
printf "[%lu] SCHEDULE_FORCED_HOST_CHECK;$host;1110741500\n" $now > $commandfile
done

Where host1 and host 2 is an array of all of your hosts.
You can get this list by doing a

Code: Select all

SELECT Alias FROM nagios_hosts ORDER BY Alias;

On the nagios MySQL table.