rrdcached errors: found extra data

[rajasegar]

I am getting this errors in /var/log/messages since I installed the rrdcached.
Even if I delete the xml and rrd file, it comes back again.

Please advice how to solve this issue.

Code: Select all

Oct 13 07:02:36 nagiosprodxi1 rrdcached[9828]: queue_thread_main: rrd_update_r (/usr/local/nagios/share/perfdata/MY1PUDRRSTTSM/Disk__All_Partitions.rrd) failed with status -1. (/usr/local/nagios/share/perfdata/MY1PUDRRSTTSM/Disk__All_Partitions.rrd: expected 12 data source readings (got 1) from 1444690041)
Oct 13 07:02:36 nagiosprodxi1 rrdcached[9828]: queue_thread_main: rrd_update_r (/usr/local/nagios/share/perfdata/MYUCBPESBAPP14/CPU_Load.rrd) failed with status -1. (/usr/local/nagios/share/perfdata/MYUCBPESBAPP14/CPU_Load.rrd: expected 6 data source readings (got 3) from 1444690241)
Oct 13 07:02:37 nagiosprodxi1 rrdcached[9828]: queue_thread_main: rrd_update_r (/usr/local/nagios/share/perfdata/MYCLKSUPRDDB01/Disk__Non_Exempt_db2.rrd) failed with status -1. (/usr/local/nagios/share/perfdata/MYCLKSUPRDDB01/Disk__Non_Exempt_db2.rrd: expected 23 data source readings (got 9) from 1444690041)
Oct 13 07:02:37 nagiosprodxi1 rrdcached[9828]: queue_thread_main: rrd_update_r (/usr/local/nagios/share/perfdata/MYRPPUPRDDB02/Disk__All_Partitions.rrd) failed with status -1. (/usr/local/nagios/share/perfdata/MYRPPUPRDDB02/Disk__All_Partitions.rrd: found extra data on update argument: 0)
Oct 13 07:02:52 nagiosprodxi1 rrdcached[9828]: queue_thread_main: rrd_update_r (/usr/local/nagios/share/perfdata/CPMS-PROD-APP/Disk__All_Partitions.rrd) failed with status -1. (/usr/local/nagios/share/perfdata/CPMS-PROD-APP/Disk__All_Partitions.rrd: expected 16 data source readings (got 11) from 1444690051)
Oct 13 07:02:52 nagiosprodxi1 rrdcached[9828]: queue_thread_main: rrd_update_r (/usr/local/nagios/share/perfdata/MYCLKSUDRDB01/Disk__All_Partitions.rrd) failed with status -1. (/usr/local/nagios/share/perfdata/MYCLKSUDRDB01/Disk__All_Partitions.rrd: expected 18 data source readings (got 17) from 1444690051)
Oct 13 07:03:08 nagiosprodxi1 rrdcached[9828]: queue_thread_main: rrd_update_r (/usr/local/nagios/share/perfdata/MYCLKSUDRWEB03/Disk__All_Partitions.rrd) failed with status -1. (/usr/local/nagios/share/perfdata/MYCLKSUDRWEB03/Disk__All_Partitions.rrd: expected 14 data source readings (got 13) from 1444690071)
Oct 13 07:03:24 nagiosprodxi1 rrdcached[9828]: queue_thread_main: rrd_update_r (/usr/local/nagios/share/perfdata/MY1PUPRDZAFDB01/Disk__All_Partitions.rrd) failed with status -1. (/usr/local/nagios/share/perfdata/MY1PUPRDZAFDB01/Disk__All_Partitions.rrd: found extra data on update argument: 18243)
Oct 13 07:03:28 nagiosprodxi1 rrdcached[9828]: queue_thread_main: rrd_update_r (/usr/local/nagios/share/perfdata/CPMS-PROD-DB/Disk__All_Partitions.rrd) failed with status -1. (/usr/local/nagios/share/perfdata/CPMS-PROD-DB/Disk__All_Partitions.rrd: found extra data on update argument: 334:1584:21053:55253:16373)
Oct 13 07:03:28 nagiosprodxi1 rrdcached[9828]: queue_thread_main: rrd_update_r (/usr/local/nagios/share/perfdata/MXiBKP/Disk__MXiDB1_Clone_u01_-__u03.rrd) failed with status -1. (/usr/local/nagios/share/perfdata/MXiBKP/Disk__MXiDB1_Clone_u01_-__u03.rrd: expected 3 data source readings (got 2) from 1444690161)
Oct 13 07:03:28 nagiosprodxi1 rrdcached[9828]: queue_thread_main: rrd_update_r (/usr/local/nagios/share/perfdata/MY1PWDRETPDB02/Disk__Drive_All.rrd) failed with status -1. (/usr/local/nagios/share/perfdata/MY1PWDRETPDB02/Disk__Drive_All.rrd: expected 12 data source readings (got 6) from 1444689901)
Oct 13 07:03:28 nagiosprodxi1 rrdcached[9828]: queue_thread_main: rrd_update_r (/usr/local/nagios/share/perfdata/MYRPPUPRDCCH02/Disk__All_Partitions.rrd) failed with status -1. (/usr/local/nagios/share/perfdata/MYRPPUPRDCCH02/Disk__All_Partitions.rrd: found extra data on update argument: 0)
Oct 13 07:03:28 nagiosprodxi1 rrdcached[9828]: queue_thread_main: rrd_update_r (/usr/local/nagios/share/perfdata/MYRPPUPRDMSG01/Disk__All_Partitions.rrd) failed with status -1. (/usr/local/nagios/share/perfdata/MYRPPUPRDMSG01/Disk__All_Partitions.rrd: found extra data on update argument: 0)
Oct 13 07:03:28 nagiosprodxi1 rrdcached[9828]: queue_thread_main: rrd_update_r (/usr/local/nagios/share/perfdata/MYUCBPESBAPP08/CPU_Load.rrd) failed with status -1. (/usr/local/nagios/share/perfdata/MYUCBPESBAPP08/CPU_Load.rrd: expected 6 data source readings (got 3) from 1444690401)
Oct 13 07:03:28 nagiosprodxi1 rrdcached[9828]: queue_thread_main: rrd_update_r (/usr/local/nagios/share/perfdata/SIGNTELLER_APP2/Disk__Drive_All.rrd) failed with status -1. (/usr/local/nagios/share/perfdata/SIGNTELLER_APP2/Disk__Drive_All.rrd: found extra data on update argument: 65:63.904:0:13.035)
Oct 13 07:03:28 nagiosprodxi1 rrdcached[9828]: queue_thread_main: rrd_update_r (/usr/local/nagios/share/perfdata/ccsappsthp01/Disk__All_Partitions.rrd) failed with status -1. (/usr/local/nagios/share/perfdata/ccsappsthp01/Disk__All_Partitions.rrd: expected 8 data source readings (got 1) from 1444689881)
Oct 13 07:03:28 nagiosprodxi1 rrdcached[9828]: queue_thread_main: rrd_update_r (/usr/local/nagios/share/perfdata/mysmsuprdapp02/Disk__All_Partitions.rrd) failed with status -1. (/usr/local/nagios/share/perfdata/mysmsuprdapp02/Disk__All_Partitions.rrd: found extra data on update argument: 0:0)
Oct 13 07:03:28 nagiosprodxi1 rrdcached[9828]: queue_thread_main: rrd_update_r (/usr/local/nagios/share/perfdata/sascrmspweb01/Disk__All_Partitions.rrd) failed with status -1. (/usr/local/nagios/share/perfdata/sascrmspweb01/Disk__All_Partitions.rrd: found extra data on update argument: 236:0:0:0:0:0)
~

[Box293]

I beleive this article will fix your problem (even though it is for a previous version of XI):

ICMP and Ping Checks Stopped Graphing After XI 2014 Upgrade
http://support.nagios.com/wiki/index.ph ... 14_Upgrade

You should be able to target the directories like /usr/local/nagios/share/perfdata/MY1PUDRRSTTSM/

Does this fix your problem?

[rajasegar]

I beleive this article will fix your problem (even though it is for a previous version of XI):

ICMP and Ping Checks Stopped Graphing After XI 2014 Upgrade
http://support.nagios.com/wiki/index.ph ... 14_Upgrade

You should be able to target the directories like /usr/local/nagios/share/perfdata/MY1PUDRRSTTSM/

Does this fix your problem?

It is running now. Will update once it is done and if the errors still show up again.

Code: Select all

Fixing permissions for file: /usr/local/nagios/share/perfdata/SIGNTELLER_APP2/Disk__Drive_All.rrd
/usr/local/nagios/share/perfdata/SIGNTELLER_APP2/Disk__Drive_All.rrd updated with 4 additional datasource(s)
Fixing permissions for file: /usr/local/nagios/share/perfdata/HQ_BR_BangsarSouth/Hardware_Health.rrd
/usr/local/nagios/share/perfdata/HQ_BR_BangsarSouth/Hardware_Health.rrd updated with 7 additional datasource(s)
Fixing permissions for file: /usr/local/nagios/share/perfdata/smswebglb01/Disk__All_Partitions.rrd
/usr/local/nagios/share/perfdata/smswebglb01/Disk__All_Partitions.rrd updated with 1 additional datasource(s)
Fixing permissions for file: /usr/local/nagios/share/perfdata/SGR_BR_EDSCyberjaya/Hardware_Health.rrd
/usr/local/nagios/share/perfdata/SGR_BR_EDSCyberjaya/Hardware_Health.rrd updated with 13 additional datasource(s)
Fixing permissions for file: /usr/local/nagios/share/perfdata/MMSASAPP01/Disk__Exemption_All_98_98_Others.rrd
/usr/local/nagios/share/perfdata/MMSASAPP01/Disk__Exemption_All_98_98_Others.rrd updated with 5 additional datasource(s)
Fixing permissions for file: /usr/local/nagios/share/perfdata/RCSWEB03/Disk__All_Partitions.rrd
/usr/local/nagios/share/perfdata/RCSWEB03/Disk__All_Partitions.rrd updated with 5 additional datasource(s)
Fixing permissions for file: /usr/local/nagios/share/perfdata/BCWEB04/Disk__All_Partitions.rrd
/usr/local/nagios/share/perfdata/BCWEB04/Disk__All_Partitions.rrd updated with 2 additional datasource(s)
Fixing permissions for file: /usr/local/nagios/share/perfdata/RCSWEB04/Disk__All_Partitions.rrd
/usr/local/nagios/share/perfdata/RCSWEB04/Disk__All_Partitions.rrd updated with 2 additional datasource(s)
Fixing permissions for file: /usr/local/nagios/share/perfdata/MYRPPUPRDREP01/Disk__All_Partitions.rrd
/usr/local/nagios/share/perfdata/MYRPPUPRDREP01/Disk__All_Partitions.rrd updated with 1 additional datasource(s)
Fixing permissions for file: /usr/local/nagios/share/perfdata/RCSDB01/Disk__Non_Exempted.rrd
/usr/local/nagios/share/perfdata/RCSDB01/Disk__Non_Exempted.rrd updated with 5 additional datasource(s)
Fixing permissions for file: /usr/local/nagios/share/perfdata/mysmsuprdappdbgbl02/Disk__All_Partitions.rrd
/usr/local/nagios/share/perfdata/mysmsuprdappdbgbl02/Disk__All_Partitions.rrd updated with 6 additional datasource(s)

[rajasegar]

All graphs stopped updating since I ran the script.
Please assist.

rrdcached journals are not getting created.
rrdcached is running, npcd is running .

Rebooted, still same problem.

[Box293]

Lets increase the logging verbosity and then take a deeper look into the logs. Follow the FAQ entry below to increase the log level of process_perfdata and npcd:
http://support.nagios.com/wiki/index.ph ... leshooting

Wait 15 - 20 minutes and then get a tail of the logs:

Code: Select all

tail -250 /usr/local/nagios/var/perfdata.log > /tmp/perfdata.txt
tail -250 /usr/local/nagios/var/npcd.log > /tmp/npcd.txt

Send us a copy of /tmp/perfdata.txt and /tmp/npcd.txt

Don't forget to turn down the log level as per the FAQ when you are done!

[rajasegar]

Lets increase the logging verbosity and then take a deeper look into the logs. Follow the FAQ entry below to increase the log level of process_perfdata and npcd:
http://support.nagios.com/wiki/index.ph ... leshooting

Wait 15 - 20 minutes and then get a tail of the logs:

Code: Select all

tail -250 /usr/local/nagios/var/perfdata.log > /tmp/perfdata.txt
tail -250 /usr/local/nagios/var/npcd.log > /tmp/npcd.txt

Send us a copy of /tmp/perfdata.txt and /tmp/npcd.txt

Don't forget to turn down the log level as per the FAQ when you are done!

I have remarked the RRD_DAEMON_OPTS in the /usr/local/nagios/etc/pnp/process_perfdata.cfg and the graphs started updating again.

Code: Select all

#RRD_DAEMON_OPTS = unix:/var/rrdtool/rrdcached/rrdcached.sock

So this is definitely rrdcached issue. Do we still need to do the above?

[Box293]

Can you please post the following files:

Code: Select all

/usr/local/nagios/etc/nagios.cfg
/usr/local/nagios/etc/pnp/npcd.cfg
/usr/local/nagios/etc/pnp/process_perfdata.cfg
/etc/sysconfig/rrdcached
/etc/init.d/rrdcached

Is rrdcached running?

Code: Select all

service rrdcached status

And also the output of:

Code: Select all

ls -alR /var/rrdtool/rrdcached/

[rajasegar]

Can you please post the following files:

Code: Select all

/usr/local/nagios/etc/nagios.cfg
/usr/local/nagios/etc/pnp/npcd.cfg
/usr/local/nagios/etc/pnp/process_perfdata.cfg
/etc/sysconfig/rrdcached
/etc/init.d/rrdcached

Is rrdcached running?

Code: Select all

service rrdcached status

And also the output of:

Code: Select all

ls -alR /var/rrdtool/rrdcached/

Downloads.zip

Code: Select all

[nagios@nagiosprodxi1 var]$ ls -alR /var/rrdtool/rrdcached
/var/rrdtool/rrdcached:
total 12
drwxr-xr-x 2 nagios nagios 4096 Oct 13 09:42 .
drwxr-xr-x 3 nagios nagios 4096 Apr  9  2015 ..
-rw-r--r-- 1 nagios nagios    6 Oct 13 09:42 rrdcached.pid
srw-rw---- 1 nagios nagios    0 Oct 13 09:42 rrdcached.sock
[nagios@nagiosprodxi1 var]$

I noticed this in /var/log/messages, segfault for rrdcached but it started normally when I executed service rrdcached start. It is running

Code: Select all

Oct 13 08:56:21 nagiosprodxi1 kernel: set_rtc_mmss: can't update from 8 to 56
Oct 13 08:56:22 nagiosprodxi1 kernel: rrdcached[2677]: segfault at 0 ip 00000034c1681451 sp 00007fff9f1594d8 error 4 in libc-2.12.so[34c1600000+18b000]
Oct 13 08:56:22 nagiosprodxi1 abrtd: Directory 'ccpp-2015-10-13-08:56:22-2677' creation detected
Oct 13 08:56:22 nagiosprodxi1 abrt[2678]: Saved core dump of pid 2677 (/usr/bin/rrdcached) to /var/spool/abrt/ccpp-2015-10-13-08:56:22-2677 (770048 bytes)
Oct 13 08:56:22 nagiosprodxi1 kernel: set_rtc_mmss: can't update from 8 to 56
Oct 13 08:56:22 nagiosprodxi1 rhnsd[2737]: Red Hat Network Services Daemon starting up, check in interval 240 minutes.
Oct 13 08:56:23 nagiosprodxi1 abrtd: Executable '/usr/bin/rrdcached' doesn't belong to any package and ProcessUnpackaged is set to 'no'
Oct 13 08:56:23 nagiosprodxi1 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2015-10-13-08:56:22-2677' exited with 1
Oct 13 08:56:23 nagiosprodxi1 abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2015-10-13-08:56:22-2677'
Oct 13 08:56:23 nagiosprodxi1 kernel: set_rtc_mmss: can't update from 8 to 56
Oct 13 08:56:24 nagiosprodxi1 kernel: set_rtc_mmss: can't update from 8 to 56
Oct 13 09:08:43 nagiosprodxi1 ntpd[2367]: 0.0.0.0 c61c 0c clock_step +738.981031 s
Oct 13 09:08:43 nagiosprodxi1 ntpd[2367]: 0.0.0.0 c614 04 freq_mode
Oct 13 09:08:44 nagiosprodxi1 nrpe[2830]: Unknown option specified in config file '/usr/local/nagios/etc/nrpe.cfg' - Line 80
Oct 13 09:08:44 nagiosprodxi1 nrpe[2830]: INFO: SSL/TLS initialized. All network traffic will be encrypted.
Oct 13 09:08:44 nagiosprodxi1 nrpe[2831]: Starting up daemon
Oct 13 09:08:44 nagiosprodxi1 nrpe[2831]: Server listening on 0.0.0.0 port 5666.
Oct 13 09:08:44 nagiosprodxi1 nrpe[2831]: Server listening on :: port 5666.
Oct 13 09:08:44 nagiosprodxi1 nrpe[2831]: Warning: Daemon is configured to accept command arguments from clients!
Oct 13 09:08:44 nagiosprodxi1 nrpe[2831]: Listening for connections on port 0
Oct 13 09:08:44 nagiosprodxi1 ntpd[2367]: 0.0.0.0 c618 08 no_sys_peer
Oct 13 09:17:25 nagiosprodxi1 rrdcached[31069]: starting up
Oct 13 09:17:25 nagiosprodxi1 rrdcached[31069]: checking for journal files
Oct 13 09:17:25 nagiosprodxi1 rrdcached[31069]: journal processing complete
Oct 13 09:17:25 nagiosprodxi1 rrdcached[31069]: listening for connections
Oct 13 09:23:56 nagiosprodxi1 ntpd[2367]: 0.0.0.0 c612 02 freq_set kernel 35.423 PPM
Oct 13 09:23:56 nagiosprodxi1 ntpd[2367]: 0.0.0.0 c615 05 clock_sync
Oct 13 09:42:51 nagiosprodxi1 rrdcached[31069]: caught SIGTERM
Oct 13 09:42:51 nagiosprodxi1 rrdcached[31069]: starting shutdown
Oct 13 09:42:51 nagiosprodxi1 rrdcached[31069]: clean shutdown; all RRDs flushed
Oct 13 09:42:51 nagiosprodxi1 rrdcached[31069]: removing journals
Oct 13 09:42:51 nagiosprodxi1 rrdcached[31069]: goodbye
Oct 13 09:42:52 nagiosprodxi1 rrdcached[31708]: starting up
Oct 13 09:42:52 nagiosprodxi1 rrdcached[31708]: checking for journal files
Oct 13 09:42:52 nagiosprodxi1 rrdcached[31708]: journal processing complete
Oct 13 09:42:52 nagiosprodxi1 rrdcached[31708]: listening for connections

[rajasegar]

Left out /etc/sysconfig/rrdcached

Code: Select all

[nagios@nagiosprodxi1 log]$ cat /etc/sysconfig/rrdcached
    OPTIONS="-s nagios -m 0660 -l unix:/var/rrdtool/rrdcached/rrdcached.sock -F -w 900 -z 300 -j /var/nagiosramdisk/tmp/ -b /var/rrdtool/rrdcached -P FLUSH,PENDING"
    RRDC_USER=nagios
[nagios@nagiosprodxi1 log]$

Here is the logs for perdata.log and npcd.log in case you still need them

logs.zip

[Box293]

So everything looks OK.

Let's check the version of rrdtool:

Code: Select all

rrdtool

I noticed this in /var/log/messages, segfault for rrdcached but it started normally when I executed service rrdcached start. It is running

Just to double check, if you enable the RRD_DAEMON_OPTS in the /usr/local/nagios/etc/pnp/process_perfdata.cfg and restart the npcd service does the same problem occur?

I'm interested in seeing the logs output with RRD_DAEMON_OPTS enabled.