XI ntpd service stops

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
Locked
dlukinski
Posts: 1130
Joined: Tue Oct 06, 2015 9:42 am

XI ntpd service stops

Post by dlukinski »

Hello XI Support

We are having problem with our production XI (failtback,tests are ok)
ntpd service stops from time to time, which creates problems with systems time and logs (immedialtely 20-30 off)
We routinely start it back and chkconfig ntpd on, which does not resolve the issue

Please help
Top
rkennedy
Posts: 6579
Joined: Mon Oct 05, 2015 11:45 am

Re: XI ntpd service stops

Post by rkennedy »

A few questions that may affect this:
1. What hypervisor are you running on the host machine?
2. What ntp server are you syncing with?
3. Does your syslog show anything about it? grep 'ntpd' /var/log/messages
Former Nagios Employee
Top
dlukinski
Posts: 1130
Joined: Tue Oct 06, 2015 9:42 am

Re: XI ntpd service stops

Post by dlukinski »

EDIT: Please use code tag, not inline code tag for long output.
rkennedy wrote:A few questions that may affect this:
1. What hypervisor are you running on the host machine?
2. What ntp server are you syncing with?
3. Does your syslog show anything about it? grep 'ntpd' /var/log/messages
-----------------------
1. VMWARE (ntpd works in other cases)
2. internal (works in other cases)

Code: Select all

root@fikc-nagxiprod01 etc]# grep 'ntpd' /var/log/messages
Nov 15 19:15:08 fikc-nagxiprod01 ntpd[1415]: ntpd [email protected] Tue May 31 10:09:21 UTC 2016 (1)
Nov 15 19:15:08 fikc-nagxiprod01 ntpd[1416]: proto: precision = 0.053 usec
Nov 15 19:15:08 fikc-nagxiprod01 ntpd[1416]: 0.0.0.0 c01d 0d kern kernel time sync enabled
Nov 15 19:15:08 fikc-nagxiprod01 ntpd[1416]: Listen and drop on 0 v4wildcard 0.0.0.0 UDP 123
Nov 15 19:15:08 fikc-nagxiprod01 ntpd[1416]: Listen and drop on 1 v6wildcard :: UDP 123
Nov 15 19:15:08 fikc-nagxiprod01 ntpd[1416]: Listen normally on 2 lo 127.0.0.1 UDP 123
Nov 15 19:15:08 fikc-nagxiprod01 ntpd[1416]: Listen normally on 3 eth0 10.102.36.162 UDP 123
Nov 15 19:15:08 fikc-nagxiprod01 ntpd[1416]: Listen normally on 4 lo ::1 UDP 123
Nov 15 19:15:08 fikc-nagxiprod01 ntpd[1416]: Listen normally on 5 eth0 fe80::250:56ff:fea5:15e3 UDP 123
Nov 15 19:15:08 fikc-nagxiprod01 ntpd[1416]: Listening on routing socket on fd #22 for interface updates
Nov 15 19:15:08 fikc-nagxiprod01 ntpd[1416]: 0.0.0.0 c016 06 restart
Nov 15 19:15:08 fikc-nagxiprod01 ntpd[1416]: 0.0.0.0 c012 02 freq_set kernel -21.328 PPM
Nov 15 18:53:26 fikc-nagxiprod01 ntpd[1416]: 0.0.0.0 c61c 0c clock_step -1302.849784 s
Nov 15 18:53:26 fikc-nagxiprod01 ntpd[1416]: 0.0.0.0 c614 04 freq_mode
Nov 15 18:53:27 fikc-nagxiprod01 ntpd[1416]: 0.0.0.0 c618 08 no_sys_peer
Nov 15 19:09:05 fikc-nagxiprod01 ntpd[1416]: 0.0.0.0 c612 02 freq_set kernel -2.031 PPM
Nov 15 19:09:05 fikc-nagxiprod01 ntpd[1416]: 0.0.0.0 c615 05 clock_sync
Nov 16 15:02:55 fikc-nagxiprod01 ntpd[1416]: 0.0.0.0 0618 08 no_sys_peer
Nov 16 17:20:13 fikc-nagxiprod01 ntpd[1416]: 0.0.0.0 0617 07 panic_stop -1332 s; set clock manually within 1000 s.
Nov 18 15:42:27 fikc-nagxiprod01 ntpd[10932]: ntpd [email protected] Tue May 31 10:09:21 UTC 2016 (1)
Nov 18 15:42:27 fikc-nagxiprod01 ntpd[10955]: proto: precision = 0.054 usec
Nov 18 15:42:27 fikc-nagxiprod01 ntpd[10955]: 0.0.0.0 c01d 0d kern kernel time sync enabled
Nov 18 15:42:28 fikc-nagxiprod01 ntpd[10955]: Listen and drop on 0 v4wildcard 0.0.0.0 UDP 123
Nov 18 15:42:28 fikc-nagxiprod01 ntpd[10955]: Listen and drop on 1 v6wildcard :: UDP 123
Nov 18 15:42:28 fikc-nagxiprod01 ntpd[10955]: Listen normally on 2 lo 127.0.0.1 UDP 123
Nov 18 15:42:28 fikc-nagxiprod01 ntpd[10955]: Listen normally on 3 eth0 10.102.36.162 UDP 123
Nov 18 15:42:28 fikc-nagxiprod01 ntpd[10955]: Listen normally on 4 lo ::1 UDP 123
Nov 18 15:42:28 fikc-nagxiprod01 ntpd[10955]: Listen normally on 5 eth0 fe80::250:56ff:fea5:15e3 UDP 123
Nov 18 15:42:28 fikc-nagxiprod01 ntpd[10955]: Listening on routing socket on fd #22 for interface updates
Nov 18 15:42:28 fikc-nagxiprod01 ntpd[10955]: 0.0.0.0 c016 06 restart
Nov 18 15:42:28 fikc-nagxiprod01 ntpd[10955]: 0.0.0.0 c012 02 freq_set kernel -3.718 PPM
Nov 18 15:20:16 fikc-nagxiprod01 ntpd[10955]: 0.0.0.0 c61c 0c clock_step -1332.317001 s
Nov 18 15:20:16 fikc-nagxiprod01 ntpd[10955]: 0.0.0.0 c614 04 freq_mode
Nov 18 15:20:17 fikc-nagxiprod01 ntpd[10955]: 0.0.0.0 c618 08 no_sys_peer
Nov 18 15:22:12 fikc-nagxiprod01 ntpd[10955]: ntpd exiting on signal 15
Nov 18 15:23:07 fikc-nagxiprod01 ntpd[1418]: ntpd [email protected] Tue May 31 10:09:21 UTC 2016 (1)
Nov 18 15:23:07 fikc-nagxiprod01 ntpd[1419]: proto: precision = 0.097 usec
Nov 18 15:23:07 fikc-nagxiprod01 ntpd[1419]: 0.0.0.0 c01d 0d kern kernel time sync enabled
Nov 18 15:23:07 fikc-nagxiprod01 ntpd[1419]: Listen and drop on 0 v4wildcard 0.0.0.0 UDP 123
Nov 18 15:23:07 fikc-nagxiprod01 ntpd[1419]: Listen and drop on 1 v6wildcard :: UDP 123
Nov 18 15:23:07 fikc-nagxiprod01 ntpd[1419]: Listen normally on 2 lo 127.0.0.1 UDP 123
Nov 18 15:23:07 fikc-nagxiprod01 ntpd[1419]: Listen normally on 3 eth0 10.102.36.162 UDP 123
Nov 18 15:23:07 fikc-nagxiprod01 ntpd[1419]: Listen normally on 4 lo ::1 UDP 123
Nov 18 15:23:07 fikc-nagxiprod01 ntpd[1419]: Listen normally on 5 eth0 fe80::250:56ff:fea5:15e3 UDP 123
Nov 18 15:23:07 fikc-nagxiprod01 ntpd[1419]: Listening on routing socket on fd #22 for interface updates
Nov 18 15:23:07 fikc-nagxiprod01 ntpd[1419]: 0.0.0.0 c016 06 restart
Nov 18 15:23:07 fikc-nagxiprod01 ntpd[1419]: 0.0.0.0 c012 02 freq_set kernel -3.718 PPM
Nov 18 15:23:08 fikc-nagxiprod01 ntpd[1419]: 0.0.0.0 c615 05 clock_sync
Nov 18 16:12:20 fikc-nagxiprod01 ntpd[1419]: 0.0.0.0 0613 03 spike_detect +0.575086 s
Nov 18 16:36:33 fikc-nagxiprod01 ntpd[1419]: 0.0.0.0 061c 0c clock_step +0.572736 s
Nov 18 16:36:33 fikc-nagxiprod01 ntpd[1419]: 0.0.0.0 0615 05 clock_sync
Nov 18 16:36:34 fikc-nagxiprod01 ntpd[1419]: 0.0.0.0 c618 08 no_sys_peer
[root@fikc-nagxiprod01 etc]#
Top
rkennedy
Posts: 6579
Joined: Mon Oct 05, 2015 11:45 am

Re: XI ntpd service stops

Post by rkennedy »

Are you using VMware tools on the VM? The stepping usually coincides with virtual machines on linux / vmware tools disagreeing with NTPD.
See this link - http://serverfault.com/questions/703702 ... ux-machine which points to https://kb.vmware.com/selfservice/micro ... Id=1006427

My guess is they are both arguing with each other.

The error is pretty evident right here -
Nov 16 17:20:13 fikc-nagxiprod01 ntpd[1416]: 0.0.0.0 0617 07 panic_stop -1332 s; set clock manually within 1000 s.
Former Nagios Employee
Top
dlukinski
Posts: 1130
Joined: Tue Oct 06, 2015 9:42 am

Re: XI ntpd service stops

Post by dlukinski »

rkennedy wrote:Are you using VMware tools on the VM? The stepping usually coincides with virtual machines on linux / vmware tools disagreeing with NTPD.
See this link - http://serverfault.com/questions/703702 ... ux-machine which points to https://kb.vmware.com/selfservice/micro ... Id=1006427

My guess is they are both arguing with each other.

The error is pretty evident right here -
Nov 16 17:20:13 fikc-nagxiprod01 ntpd[1416]: 0.0.0.0 0617 07 panic_stop -1332 s; set clock manually within 1000 s.
Yes we are using open-vm-tools as well. Curious why in this case only (other VM machines also run open-vm-tools, but have no troubles)
Top
ssax
Dreams In Code
Posts: 7682
Joined: Wed Feb 11, 2015 12:54 pm

Re: XI ntpd service stops

Post by ssax »

Code: Select all

Nov 18 16:36:33 fikc-nagxiprod01 ntpd[1419]: 0.0.0.0 0615 05 clock_sync
Nov 18 16:36:34 fikc-nagxiprod01 ntpd[1419]: 0.0.0.0 c618 08 no_sys_peer
Those indicate that ntpd is not able to find a peer that meets its criteria.

Please run these commands:
- Change time.nist.gov to whatever your NTP server IP/DNS is

Code: Select all

service ntpd stop
ntpdate time.nist.gov
service ntpd start
What do you have defined in your /etc/ntp.conf? Please attach it.


Thank you
Top
dlukinski
Posts: 1130
Joined: Tue Oct 06, 2015 9:42 am

Re: XI ntpd service stops

Post by dlukinski »

ssax wrote:

Code: Select all

Nov 18 16:36:33 fikc-nagxiprod01 ntpd[1419]: 0.0.0.0 0615 05 clock_sync
Nov 18 16:36:34 fikc-nagxiprod01 ntpd[1419]: 0.0.0.0 c618 08 no_sys_peer
Those indicate that ntpd is not able to find a peer that meets its criteria.

Please run these commands:
- Change time.nist.gov to whatever your NTP server IP/DNS is

Code: Select all

service ntpd stop
ntpdate time.nist.gov
service ntpd start
What do you have defined in your /etc/ntp.conf? Please attach it.

Attached ntp.conf

it is timeserver.konecranes.com that we are using
Thank you
You do not have the required permissions to view the files attached to this post.
Top
ssax
Dreams In Code
Posts: 7682
Joined: Wed Feb 11, 2015 12:54 pm

Re: XI ntpd service stops

Post by ssax »

Please post the output of this command:

Code: Select all

ntpq -d -c peers -c as -c rv
Top
dlukinski
Posts: 1130
Joined: Tue Oct 06, 2015 9:42 am

Re: XI ntpd service stops

Post by dlukinski »

ssax wrote:Please post the output of this command:

Code: Select all

ntpq -d -c peers -c as -c rv
Here you go

Code: Select all

[root@fikc-nagxiprod01 ~]# ntpq -d -c peers -c as -c rv
1 packets reassembled into response
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
2 packets reassembled into response
*fihp-idm01.kone 194.100.49.152   3 u   32  128  377    0.180    0.030   0.028
2 packets reassembled into response
 ns2.customer-re .INIT.          16 u    - 1024    0    0.000    0.000   0.000
2 packets reassembled into response
 ntp.ifj.edu.pl  .INIT.          16 u    - 1024    0    0.000    0.000   0.000
2 packets reassembled into response
 monkey.spod.net .INIT.          16 u    - 1024    0    0.000    0.000   0.000
2 packets reassembled into response
 magnat.ip-conne .INIT.          16 u    - 1024    0    0.000    0.000   0.000
1 packets reassembled into response

ind assid status  conf reach auth condition  last_event cnt
===========================================================
  1 36740  963a   yes   yes  none  sys.peer    sys_peer  3
  2 36741  8011   yes    no  none    reject    mobilize  1
  3 36742  8011   yes    no  none    reject    mobilize  1
  4 36743  8011   yes    no  none    reject    mobilize  1
  5 36744  8011   yes    no  none    reject    mobilize  1
1 packets reassembled into response
associd=0 status=0615 leap_none, sync_ntp, 1 event, clock_sync,
version="ntpd [email protected] Tue May 31 10:09:21 UTC 2016 (1)",
processor="x86_64", system="Linux/2.6.32-642.4.2.el6.x86_64", leap=00,
stratum=4, precision=-23, rootdelay=3.217, rootdisp=26.482,
refid=10.102.34.25,
reftime=dbdddeba.c3ef37dd  Mon, Nov 21 2016 20:59:38.765,
clock=dbdde16f.fba57f74  Mon, Nov 21 2016 21:11:11.982, peer=36740, tc=7,
mintc=3, offset=0.030, frequency=-4.955, sys_jitter=0.000,
clk_jitter=0.060, clk_wander=0.008
Top
ssax
Dreams In Code
Posts: 7682
Joined: Wed Feb 11, 2015 12:54 pm

Re: XI ntpd service stops

Post by ssax »

It only looks like you have one server defined and it can't connect to the others, do you have any additional server's that you can add that are also reachable? You may want to comment out the defaults:

Code: Select all

#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst
Then restart NTP:

Code: Select all

service ntpd restart
That means that if NTP syncing fails for your defined server then it has no backup servers to try, maybe that is why you are receiving that message.
Top
Locked

Return to “Nagios XI”