Page 1 of 2
XI ntpd service stops
Posted: Fri Nov 18, 2016 10:24 am
by dlukinski
Hello XI Support
We are having problem with our production XI (failtback,tests are ok)
ntpd service stops from time to time, which creates problems with systems time and logs (immedialtely 20-30 off)
We routinely start it back and chkconfig ntpd on, which does not resolve the issue
Please help
Re: XI ntpd service stops
Posted: Fri Nov 18, 2016 11:21 am
by rkennedy
A few questions that may affect this:
1. What hypervisor are you running on the host machine?
2. What ntp server are you syncing with?
3. Does your syslog show anything about it? grep 'ntpd' /var/log/messages
Re: XI ntpd service stops
Posted: Fri Nov 18, 2016 2:07 pm
by dlukinski
EDIT: Please use code tag, not inline code tag for long output.
rkennedy wrote:A few questions that may affect this:
1. What hypervisor are you running on the host machine?
2. What ntp server are you syncing with?
3. Does your syslog show anything about it? grep 'ntpd' /var/log/messages
-----------------------
1. VMWARE (ntpd works in other cases)
2. internal (works in other cases)
Code: Select all
root@fikc-nagxiprod01 etc]# grep 'ntpd' /var/log/messages
Nov 15 19:15:08 fikc-nagxiprod01 ntpd[1415]: ntpd [email protected] Tue May 31 10:09:21 UTC 2016 (1)
Nov 15 19:15:08 fikc-nagxiprod01 ntpd[1416]: proto: precision = 0.053 usec
Nov 15 19:15:08 fikc-nagxiprod01 ntpd[1416]: 0.0.0.0 c01d 0d kern kernel time sync enabled
Nov 15 19:15:08 fikc-nagxiprod01 ntpd[1416]: Listen and drop on 0 v4wildcard 0.0.0.0 UDP 123
Nov 15 19:15:08 fikc-nagxiprod01 ntpd[1416]: Listen and drop on 1 v6wildcard :: UDP 123
Nov 15 19:15:08 fikc-nagxiprod01 ntpd[1416]: Listen normally on 2 lo 127.0.0.1 UDP 123
Nov 15 19:15:08 fikc-nagxiprod01 ntpd[1416]: Listen normally on 3 eth0 10.102.36.162 UDP 123
Nov 15 19:15:08 fikc-nagxiprod01 ntpd[1416]: Listen normally on 4 lo ::1 UDP 123
Nov 15 19:15:08 fikc-nagxiprod01 ntpd[1416]: Listen normally on 5 eth0 fe80::250:56ff:fea5:15e3 UDP 123
Nov 15 19:15:08 fikc-nagxiprod01 ntpd[1416]: Listening on routing socket on fd #22 for interface updates
Nov 15 19:15:08 fikc-nagxiprod01 ntpd[1416]: 0.0.0.0 c016 06 restart
Nov 15 19:15:08 fikc-nagxiprod01 ntpd[1416]: 0.0.0.0 c012 02 freq_set kernel -21.328 PPM
Nov 15 18:53:26 fikc-nagxiprod01 ntpd[1416]: 0.0.0.0 c61c 0c clock_step -1302.849784 s
Nov 15 18:53:26 fikc-nagxiprod01 ntpd[1416]: 0.0.0.0 c614 04 freq_mode
Nov 15 18:53:27 fikc-nagxiprod01 ntpd[1416]: 0.0.0.0 c618 08 no_sys_peer
Nov 15 19:09:05 fikc-nagxiprod01 ntpd[1416]: 0.0.0.0 c612 02 freq_set kernel -2.031 PPM
Nov 15 19:09:05 fikc-nagxiprod01 ntpd[1416]: 0.0.0.0 c615 05 clock_sync
Nov 16 15:02:55 fikc-nagxiprod01 ntpd[1416]: 0.0.0.0 0618 08 no_sys_peer
Nov 16 17:20:13 fikc-nagxiprod01 ntpd[1416]: 0.0.0.0 0617 07 panic_stop -1332 s; set clock manually within 1000 s.
Nov 18 15:42:27 fikc-nagxiprod01 ntpd[10932]: ntpd [email protected] Tue May 31 10:09:21 UTC 2016 (1)
Nov 18 15:42:27 fikc-nagxiprod01 ntpd[10955]: proto: precision = 0.054 usec
Nov 18 15:42:27 fikc-nagxiprod01 ntpd[10955]: 0.0.0.0 c01d 0d kern kernel time sync enabled
Nov 18 15:42:28 fikc-nagxiprod01 ntpd[10955]: Listen and drop on 0 v4wildcard 0.0.0.0 UDP 123
Nov 18 15:42:28 fikc-nagxiprod01 ntpd[10955]: Listen and drop on 1 v6wildcard :: UDP 123
Nov 18 15:42:28 fikc-nagxiprod01 ntpd[10955]: Listen normally on 2 lo 127.0.0.1 UDP 123
Nov 18 15:42:28 fikc-nagxiprod01 ntpd[10955]: Listen normally on 3 eth0 10.102.36.162 UDP 123
Nov 18 15:42:28 fikc-nagxiprod01 ntpd[10955]: Listen normally on 4 lo ::1 UDP 123
Nov 18 15:42:28 fikc-nagxiprod01 ntpd[10955]: Listen normally on 5 eth0 fe80::250:56ff:fea5:15e3 UDP 123
Nov 18 15:42:28 fikc-nagxiprod01 ntpd[10955]: Listening on routing socket on fd #22 for interface updates
Nov 18 15:42:28 fikc-nagxiprod01 ntpd[10955]: 0.0.0.0 c016 06 restart
Nov 18 15:42:28 fikc-nagxiprod01 ntpd[10955]: 0.0.0.0 c012 02 freq_set kernel -3.718 PPM
Nov 18 15:20:16 fikc-nagxiprod01 ntpd[10955]: 0.0.0.0 c61c 0c clock_step -1332.317001 s
Nov 18 15:20:16 fikc-nagxiprod01 ntpd[10955]: 0.0.0.0 c614 04 freq_mode
Nov 18 15:20:17 fikc-nagxiprod01 ntpd[10955]: 0.0.0.0 c618 08 no_sys_peer
Nov 18 15:22:12 fikc-nagxiprod01 ntpd[10955]: ntpd exiting on signal 15
Nov 18 15:23:07 fikc-nagxiprod01 ntpd[1418]: ntpd [email protected] Tue May 31 10:09:21 UTC 2016 (1)
Nov 18 15:23:07 fikc-nagxiprod01 ntpd[1419]: proto: precision = 0.097 usec
Nov 18 15:23:07 fikc-nagxiprod01 ntpd[1419]: 0.0.0.0 c01d 0d kern kernel time sync enabled
Nov 18 15:23:07 fikc-nagxiprod01 ntpd[1419]: Listen and drop on 0 v4wildcard 0.0.0.0 UDP 123
Nov 18 15:23:07 fikc-nagxiprod01 ntpd[1419]: Listen and drop on 1 v6wildcard :: UDP 123
Nov 18 15:23:07 fikc-nagxiprod01 ntpd[1419]: Listen normally on 2 lo 127.0.0.1 UDP 123
Nov 18 15:23:07 fikc-nagxiprod01 ntpd[1419]: Listen normally on 3 eth0 10.102.36.162 UDP 123
Nov 18 15:23:07 fikc-nagxiprod01 ntpd[1419]: Listen normally on 4 lo ::1 UDP 123
Nov 18 15:23:07 fikc-nagxiprod01 ntpd[1419]: Listen normally on 5 eth0 fe80::250:56ff:fea5:15e3 UDP 123
Nov 18 15:23:07 fikc-nagxiprod01 ntpd[1419]: Listening on routing socket on fd #22 for interface updates
Nov 18 15:23:07 fikc-nagxiprod01 ntpd[1419]: 0.0.0.0 c016 06 restart
Nov 18 15:23:07 fikc-nagxiprod01 ntpd[1419]: 0.0.0.0 c012 02 freq_set kernel -3.718 PPM
Nov 18 15:23:08 fikc-nagxiprod01 ntpd[1419]: 0.0.0.0 c615 05 clock_sync
Nov 18 16:12:20 fikc-nagxiprod01 ntpd[1419]: 0.0.0.0 0613 03 spike_detect +0.575086 s
Nov 18 16:36:33 fikc-nagxiprod01 ntpd[1419]: 0.0.0.0 061c 0c clock_step +0.572736 s
Nov 18 16:36:33 fikc-nagxiprod01 ntpd[1419]: 0.0.0.0 0615 05 clock_sync
Nov 18 16:36:34 fikc-nagxiprod01 ntpd[1419]: 0.0.0.0 c618 08 no_sys_peer
[root@fikc-nagxiprod01 etc]#
Re: XI ntpd service stops
Posted: Fri Nov 18, 2016 2:21 pm
by rkennedy
Are you using VMware tools on the VM? The stepping usually coincides with virtual machines on linux / vmware tools disagreeing with NTPD.
See this link -
http://serverfault.com/questions/703702 ... ux-machine which points to
https://kb.vmware.com/selfservice/micro ... Id=1006427
My guess is they are both arguing with each other.
The error is pretty evident right here -
Nov 16 17:20:13 fikc-nagxiprod01 ntpd[1416]: 0.0.0.0 0617 07 panic_stop -1332 s; set clock manually within 1000 s.
Re: XI ntpd service stops
Posted: Fri Nov 18, 2016 3:02 pm
by dlukinski
Yes we are using open-vm-tools as well. Curious why in this case only (other VM machines also run open-vm-tools, but have no troubles)
Re: XI ntpd service stops
Posted: Mon Nov 21, 2016 1:22 pm
by ssax
Code: Select all
Nov 18 16:36:33 fikc-nagxiprod01 ntpd[1419]: 0.0.0.0 0615 05 clock_sync
Nov 18 16:36:34 fikc-nagxiprod01 ntpd[1419]: 0.0.0.0 c618 08 no_sys_peer
Those indicate that ntpd is not able to find a peer that meets its criteria.
Please run these commands:
- Change time.nist.gov to whatever your NTP server IP/DNS is
Code: Select all
service ntpd stop
ntpdate time.nist.gov
service ntpd start
What do you have defined in your
/etc/ntp.conf? Please attach it.
Thank you
Re: XI ntpd service stops
Posted: Mon Nov 21, 2016 3:17 pm
by dlukinski
ssax wrote:Code: Select all
Nov 18 16:36:33 fikc-nagxiprod01 ntpd[1419]: 0.0.0.0 0615 05 clock_sync
Nov 18 16:36:34 fikc-nagxiprod01 ntpd[1419]: 0.0.0.0 c618 08 no_sys_peer
Those indicate that ntpd is not able to find a peer that meets its criteria.
Please run these commands:
- Change time.nist.gov to whatever your NTP server IP/DNS is
Code: Select all
service ntpd stop
ntpdate time.nist.gov
service ntpd start
What do you have defined in your
/etc/ntp.conf? Please attach it.
Attached ntp.conf
it is timeserver.konecranes.com that we are using
Thank you
Re: XI ntpd service stops
Posted: Mon Nov 21, 2016 3:50 pm
by ssax
Please post the output of this command:
Re: XI ntpd service stops
Posted: Mon Nov 21, 2016 4:11 pm
by dlukinski
ssax wrote:Please post the output of this command:
Here you go
Code: Select all
[root@fikc-nagxiprod01 ~]# ntpq -d -c peers -c as -c rv
1 packets reassembled into response
remote refid st t when poll reach delay offset jitter
==============================================================================
2 packets reassembled into response
*fihp-idm01.kone 194.100.49.152 3 u 32 128 377 0.180 0.030 0.028
2 packets reassembled into response
ns2.customer-re .INIT. 16 u - 1024 0 0.000 0.000 0.000
2 packets reassembled into response
ntp.ifj.edu.pl .INIT. 16 u - 1024 0 0.000 0.000 0.000
2 packets reassembled into response
monkey.spod.net .INIT. 16 u - 1024 0 0.000 0.000 0.000
2 packets reassembled into response
magnat.ip-conne .INIT. 16 u - 1024 0 0.000 0.000 0.000
1 packets reassembled into response
ind assid status conf reach auth condition last_event cnt
===========================================================
1 36740 963a yes yes none sys.peer sys_peer 3
2 36741 8011 yes no none reject mobilize 1
3 36742 8011 yes no none reject mobilize 1
4 36743 8011 yes no none reject mobilize 1
5 36744 8011 yes no none reject mobilize 1
1 packets reassembled into response
associd=0 status=0615 leap_none, sync_ntp, 1 event, clock_sync,
version="ntpd [email protected] Tue May 31 10:09:21 UTC 2016 (1)",
processor="x86_64", system="Linux/2.6.32-642.4.2.el6.x86_64", leap=00,
stratum=4, precision=-23, rootdelay=3.217, rootdisp=26.482,
refid=10.102.34.25,
reftime=dbdddeba.c3ef37dd Mon, Nov 21 2016 20:59:38.765,
clock=dbdde16f.fba57f74 Mon, Nov 21 2016 21:11:11.982, peer=36740, tc=7,
mintc=3, offset=0.030, frequency=-4.955, sys_jitter=0.000,
clk_jitter=0.060, clk_wander=0.008
Re: XI ntpd service stops
Posted: Mon Nov 21, 2016 4:54 pm
by ssax
It only looks like you have one server defined and it can't connect to the others, do you have any additional server's that you can add that are also reachable? You may want to comment out the defaults:
Code: Select all
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst
Then restart NTP:
That means that if NTP syncing fails for your defined server then it has no backup servers to try, maybe that is why you are receiving that message.