NTP service only see the Puppet server

[tezarin]

Hi all,

I have a puppetized environment and had Puppet install Nagios and NRPE on one of my virtual machines. Nagios is able to monitor all the hosts in this environment and all the services are working fine except for the NTP. I'm getting a CRITICAL issue for all of the NTP services, i.e. NTP CRITICAL: Offset -558.895350 sec > +/- 120 sec, jitter 0.005 msec

I checked and looks like all the nodes can see the Puppet server's NTP but the NTP service fails because the nodes cannot access Nagios's NTP. This is what I have inside the Puppet's ntp.conf file:

Code: Select all

tinker panic 0

restrict default kod nomodify notrap

restrict 127.0.0.1
server  127.127.1.0     # local clock
fudge   127.127.1.0 stratum 2
driftfile /var/lib/ntp/drift
broadcastdelay  0.008
server 192.125.121.5 minpoll 4 maxpoll 4 iburst prefer
restrict 192.125.121.0 mask 255.255.255.0 nomodify notrap

And this is what I have inside the Nagios ntp.conf:

Code: Select all

tinker panic 0
restrict default kod nomodify notrap
restrict 127.0.0.1
driftfile /var/lib/ntp/drift
broadcastdelay  0.008
server 192.125.121.5 minpoll 4 maxpoll 4 iburst prefer

Now here are couple other commands I ran to test

Code: Select all

[root@nagios conf.d]# /usr/lib64/nagios/plugins/check_ntp_time -H 192.125.121.5 -v
sending request to peer 0
response from peer 0: offset 1.931190491e-05
sending request to peer 0
response from peer 0: offset 2.193450928e-05
sending request to peer 0
response from peer 0: offset -2.384185791e-05
sending request to peer 0
response from peer 0: offset -1.263618469e-05
overall average offset: 1.931190491e-05
NTP OK: Offset 1.931190491e-05 secs|offset=0.000019s;60.000000;120.000000;

And I ran this from another node, which shows it sees the Puppet's NTP:

Code: Select all

[root@testserver ~]# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*puppet.machine LOCAL(0)         3 u   13   16  377    0.385    0.038   0.044

I thought by adding the two lines (below) to my Nagios's ntp.conf I can fix the issue but the NTP service is still CRITICAL.

Code: Select all

server 192.125.121.12 minpoll 4 maxpoll 4 iburst prefer
restrict 192.125.121.0 mask 255.255.255.0 nomodify notrap nopeer

Can someone please shed some light on this? Not sure what I'm doing wrong that NTP service doesn't work.

Thanks in advance

[jdalrymple]

What does your check_ntp or check_ntp_time service/command look like?

Is it possible that all of your internal hosts clocks are about 560 seconds out of whack and that your Nagios check is reaching out to a public time source?

[tezarin]

What does your check_ntp or check_ntp_time service/command look like?

Is it possible that all of your internal hosts clocks are about 560 seconds out of whack and that your Nagios check is reaching out to a public time source?

This is the service definition I found inside the /etc/nagios/conf.d/system.default.cfg file:

Code: Select all

define service{
        use                     nrpe-service    ; Name of service template to use
        hostgroup_name          linux-servers
        service_description     NTP
        check_command           check_nrpe!check_ntp
}

I don't see a command definition inside the commands.cfg file. Here's an example of what's currectly inside the commands.cfg file:

Code: Select all

define command{
        command_name    check_local_procs
        command_line    $USER1$/check_procs -w $ARG1$ -c $ARG2$ -s $ARG3$
        }

I ran the date command on Nagios, Puppet master and a random host and they all returned the same date, time and timezone.
Tried to stop the NTP on puppet master and update it but it didn't let me:

Code: Select all

[root@puppet ~]# service ntpd stop
Shutting down ntpd:                                        [  OK  ]
[root@puppet ~]# ntpdate
 7 Apr 15:59:36 ntpdate[16181]: no servers can be used, exiting

Here are some more tests to see if the port is open:

Code: Select all

Ran this: 
[root@puppet ~]# netstat -an | grep 123 | grep -i listen
unix  2      [ ACC ]     STREAM     LISTENING     11232  private/proxywrite
unix  2      [ ACC ]     STREAM     LISTENING     11236  private/smtp

[root@puppet ~]# netstat –tuplen
udp        0      0 :::123                      :::*                                    0          18465679   13186/ntpd

I tried to update the NTP on Puppet but nothing happens and I have to ctrl+C to exit:

Code: Select all

[root@puppet ~]# ntpd -gq

I'm definitely missing something here, just not sure what it is...

[tezarin]

What does your check_ntp or check_ntp_time service/command look like?

Is it possible that all of your internal hosts clocks are about 560 seconds out of whack and that your Nagios check is reaching out to a public time source?

I forgot to add, the timestamps on all these VMs are the same (GMT time), if I run the date command on the nodes I get 4:09 PM right now and if I Google the GMT file I get 4:00 PM. Could it be why?

[tezarin]

Also, when I run the command below on the puppet master I get a different result than the one I get when I run it on the nagios machine:

Code: Select all

[root@puppet ~]# ntpdate 207.32.191.59
 7 Apr 16:29:20 ntpdate[19913]: adjust time server 207.32.191.59 offset -0.002584 sec

Code: Select all

[root@nagios ~]# ntpdate 207.32.191.59
 7 Apr 16:37:37 ntpdate[18284]: the NTP socket is in use, exiting

The date command returns the same exact time on all the nodes which this time is off by 9 minutes from the real GMT time (I Googled "current GMT time")

[jdalrymple]

What does the line that starts with "check_ntp" look like in one of your servers' nrpe.cfg file?

[tezarin]

What does the line that starts with "check_ntp" look like in one of your servers' nrpe.cfg file?

This is the line I found in the nrpe.cfg file:

command[check_ntp]=/usr/lib64/nagios/plugins/check_ntp.pl -H [a 65. IP address here]

[jdalrymple]

Where did that 65 address come from?

Can you configure your puppet server to have a good external time source (pool.ntp.org)? If you do I think it will solve all your problems.

[tezarin]

Where did that 65 address come from?

Can you configure your puppet server to have a good external time source (pool.ntp.org)? If you do I think it will solve all your problems.

I have no idea what that 65. address is.

Just to test, I manually change the date on the Puppet master machine using the date command so the time on this server would be the same as the internet source GMT time and it fixed the issue. Nagios's NTP service no longer reports CRITICAL.

Now I tried to do a ntpdate 207.32.191.59 on the puppet master and it works. But if I do a ntpdate pool.ntp.org it fails as the DNS stuff is missing there.

Next I'll replace the .65 in the nrpe.cfg file and put Puppet master's IP there, right? Thanks in advance

[jdalrymple]

That should work...

I would suggest fixing DNS on the Puppet server so that it has a good time source. That should be trivial.