NagiosXI MySQL offloading

[Mitchell]

I recently made following changes using latest manuals from asset.nagios
1) Use of Ramdisk
2) MySQL offload
3) RRDcached use for npcd/rrdtool

everything is working fine except few things I noticed...

A) few add-ons like BPA are not working as not able to connect to MySQL. here are the logs when I try to use bpa... There may be other add-ons facing this issue after offloading mysql.

[Sun Jul 01 12:56:29 2012] [error] [client 172.24.41.233] [Sun Jul 1 12:56:29 2012] nagios-bp.cgi: Error: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2), referer: http://pnagios02lxv/nagiosxi/
[Sun Jul 01 12:56:29 2012] [error] [client 172.24.41.233] Premature end of script headers: nagios-bp.cgi, referer: http://pnagios02lxv/nagiosxi/
[Sun Jul 01 12:56:31 2012] [error] [client 172.24.41.233] [Sun Jul 1 12:56:31 2012] nagios-bp.cgi: DBI connect('nagios:localhost:3306','ndoutils',...) failed: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) at /usr/local/nagiosbp/lib/ndodb.pm line 68, referer: http://pnagios02lxv/nagiosxi/

B) rrdtool is using rrdcached but constantly trying perl module first. is there a way to stop rrdtool checking for it.

2012-07-01 10:43:46 [15844] [2] RRDs Perl Modules are not installed. Falling back to rrdtool system call.
2012-07-01 10:43:46 [15844] [2] /usr/bin/rrdtool update --daemon=unix:/var/rrdtool/rrdcached/rrdcached.sock /usr/local/nagios/share/perfdata/UWEB248NTV/Disk_Usage.rrd 1341164613:76:50
2012-07-01 10:43:46 [15844] [1] rrdtool update returns 0
2012-07-01 10:43:46 [15844] [1] 6 lines processed
2012-07-01 10:43:46 [15844] [1] /var/nagiosramdisk/spool/perfdata//1341164616.perfdata.service-PID-15844 deleted
2012-07-01 10:43:46 [15844] [1] PNP exiting (runtime 0.03363s) ...

C) Seeing some errors in syslog from rrdtool update

Jul 1 13:21:07 pnagios02lxv rrdcached[26019]: queue_thread_main: rrd_update_r (/usr/local/nagios/share/perfdata/UAPP520NTV/Disk_Usage.rrd) failed with status -1. (/usr/local/nagios/share/perfdata/UAPP520NTV/Disk_Usage.rrd: found extra data on update argument: 21)
Jul 1 13:21:37 pnagios02lxv rrdcached[26019]: queue_thread_main: rrd_update_r (/usr/local/nagios/share/perfdata/USQL03NTV/CPU_Load.rrd) failed with status -1. (/usr/local/nagios/share/perfdata/USQL03NTV/CPU_Load.rrd: found extra data on update argument: 0)
Jul 1 13:21:37 pnagios02lxv rrdcached[26019]: queue_thread_main: rrd_update_r (/usr/local/nagios/share/perfdata/OPSPCPMSIS01/_HOST_.rrd) failed with status -1. (/usr/local/nagios/share/perfdata/OPSPCPMSIS01/_HOST_.rrd: expected 2 data source readings (got 1) from 1341173207)
Jul 1 13:22:23 pnagios02lxv rrdcached[26019]: queue_thread_main: rrd_update_r (/usr/local/nagios/share/perfdata/USACTX12NTV/CPU_Load.rrd) failed with status -1. (/usr/local/nagios/share/perfdata/USACTX12NTV/CPU_Load.rrd: found extra data on update argument: 0:0)
Jul 1 13:23:24 pnagios02lxv rrdcached[26019]: queue_thread_main: rrd_update_r (/usr/local/nagios/share/perfdata/USAWEB19NTV/CPU_Load.rrd) failed with status -1. (/usr/local/nagios/share/perfdata/USAWEB19NTV/CPU_Load.rrd: found extra data on update argument: 0)
Jul 1 13:23:24 pnagios02lxv rrdcached[26019]: queue_thread_main: rrd_update_r (/usr/local/nagios/share/perfdata/USQL07NTV/Disk_Usage.rrd) failed with status -1. (/usr/local/nagios/share/perfdata/USQL07NTV/Disk_Usage.rrd: found extra data on update argument: 18)
Jul 1 13:27:28 pnagios02lxv rrdcached[26019]: queue_thread_main: rrd_update_r (/usr/local/nagios/share/perfdata/USACTXW14NTV/CPU_Load.rrd) failed with status -1. (/usr/local/nagios/share/perfdata/USACTXW14NTV/CPU_Load.rrd: found extra data on update argument: 0)
Jul 1 13:31:17 pnagios02lxv nagios: Auto-save of retention data completed successfully.
Jul 1 13:34:06 pnagios02lxv rrdcached[26019]: queue_thread_main: rrd_update_r (/usr/local/nagios/share/perfdata/UIMAGE5NTV/Disk_Usage.rrd) failed with status -1. (/usr/local/nagios/share/perfdata/UIMAGE5NTV/Disk_Usage.rrd: expected 9 data source readings (got 3) from 1341173621)
Jul 1 13:36:55 pnagios02lxv rrdcached[26019]: queue_thread_main: rrd_update_r (/usr/local/nagios/share/perfdata/UAPP10NTV/Disk_Usage.rrd) failed with status -1. (/usr/local/nagios/share/perfdata/UAPP10NTV/Disk_Usage.rrd: found extra data on update argument: 25)

D) I reduced the npcd and process_perfdata logging to minimum as those were other top disk users. By default the logging is set to log level 1.

E) I used iotop to see the top IO users and can see postgres continuously writing to disk. Will it be helpful to offload postgres as well?

[scottwilkerson]

A) Yes, if you are using addon components that also need to read from the database using their own config file, they will need to be updated also.

B) This is a default debug output from from the /usr/local/nagios/libexec/process_perfdata.pl script

C) I believe this may be related
http://forum.proxmox.com/archive/index.php/t-9489.html

D) This is almost always a good idea

E) We don't have any statistics good/bad on the ramifications of offloading postgres at this time.

[Mitchell]

Thank you. please see my queries.

A) Yes, if you are using addon components that also need to read from the database using their own config file, they will need to be updated also.

B) This is a default debug output from from the /usr/local/nagios/libexec/process_perfdata.pl script

C) I believe this may be related
http://forum.proxmox.com/archive/index.php/t-9489.html

D) This is almost always a good idea

E) We don't have any statistics good/bad on the ramifications of offloading postgres at this time.

A) Where do I make the change? I mean which php or ini file?

C) I could not figure it out. does this mean deleting existing .rrd files holding perfdata?


Thanks
Ashish

[scottwilkerson]

A) setting for DB connection in BPA I believe are in /usr/local/nagiosbp/etc/ndo.cfg

C) Yes, most likely the RRD was created with perfdata that contained a different number of values than the updates contains. Try deleting the RRD file then let rrdtool create a new one. As long as the plugin perfdata does not randomly change from reporting only a value to reporting value, warn, crit, max, and min, everything should work.