Page 1 of 1
Check Results Truncated - Corrupting MRTG data
Posted: Wed Dec 17, 2014 3:42 pm
by mberkley
We have a problem.
Nagios is truncating result data down to 255 charcters, causing complications in our reporting and graphing. Because it's truncating result data, is in creating improper syntax for the MRTG to interpret.
Below is the actual command output.... below that is what Nagios is being sent, and what it is consuming.
This is 646 Characters
DISK OK - free space: / 994 MB (55% inode=94%): /usr 1619 MB (28% inode=84%): /var 1481 MB (32% inode=96%): /tmp 8332 MB (77% inode=99%): /home 3187 MB (31% inode=93%): /admin 255 MB (99% inode=99%): /opt 3369 MB (82% inode=97%): /var/adm/ras/livedump 255 MB (99% inode=99%): /usr/local 1166 MB (75% inode=97%): /usr/infmx 2524 MB (30% inode=93%): /usr/infmx/dump 484589 MB (99% inode=99%): /logbkups 483520 MB (59% inode=99%): /app 7385 MB (36% inode=95%): /mksys_image 14743 MB (71% inode=99%): /var/perf_data 165257 MB (80% inode=99%): /var/tmp/hds 10451 MB (99% inode=99%): /ifmx 5098 MB (99% inode=99%): /ifmx/hyvp 163297 MB (82% inode=99%):
This is 255 Characters
/=797MB;1433;1612;0;1792 /usr=4012MB;5350;5519;0;5632 /var=3126MB;3686;4147;0;4608 /tmp=2419MB;8601;9676;0;10752 /home=7053MB;8192;9216;0;10240 /admin=0MB;204;230;0;256 /opt=726MB;3276;3686;0;4096 /var/adm/ras/livedump=0MB;204;230;0;256 /usr/local=369MB;1228;1382;0;1536 /usr/infmx=5667MB;6553;7372;0;8192 /usr/infmx/dump=1810MB;389120;437760;0;486400 /logbkups=333939MB;6543
Here's what it looks like when extra volumes were added... the graphic stopped due to incorrect input resulting from truncation.
Re: Check Results Truncated - Corrupting MRTG data
Posted: Wed Dec 17, 2014 4:11 pm
by scottwilkerson
To expand the database tables you can run the following
Code: Select all
echo "
alter table nagios_servicestatus modify output varchar(65535) not null,modify long_output varchar(65535) not null,modify perfdata varchar(65535) not null;
alter table nagios_hoststatus modify output varchar(65535) not null, modify long_output varchar(65535) not null,modify perfdata varchar(65535) not null;
alter table nagios_servicechecks modify output varchar(65535) not null,modify long_output varchar(65535) not null,modify perfdata varchar(65535) not null;
alter table nagios_hostchecks modify output varchar(65535) not null,modify long_output varchar(65535) not null,modify perfdata varchar(65535) not null;
" | mysql -pnagiosxi nagios
[/quote]
Re: Check Results Truncated - Corrupting MRTG data
Posted: Wed Dec 17, 2014 4:23 pm
by sreinhardt
In this case, it is actually neither nagios nor mrtg technically doing anything wrong. The issue here, is one of several potentially. Biggest one is that rrds are created with a finite number of tracks(per mount point in this case) when the service is initially created. Adding data to rrds does not take into account adding new tracks, or reordering based on labels. Due to this, rrdtool is likely unable to populate your rrd with any meaningful data if any at all. If it is actually populating data, it is very likely that the track to mountpoint alignment originally setup is quite off. I see two possible resolutions, if this is the case.
1) resize the rrd with the script found at the faq post for correcting track changes with check_ping and check_icmp. This will actually fix this for any rrds with less data than tracks. However, this will not correct your labeling to rrd track differences, and there is not really a way to do that. The issue with this approach, is that it is likely to happen again, and cause further issues.
http://support.nagios.com/wiki/index.ph ... 14_Upgrade
http://assets.nagios.com/downloads/nagi ... ds_fix.zip
2) The alternative that I would highly suggest, is to create a base service with mount points that should never change such as / & /boot, etc. Then for system specific mounts and mounts that may be added or removed, you can create another single grouping if the are unlikely to change, or several individual services for additional mount points. This avoids ever having to resize rrds, as the base mount points never change, and dynamic or very likely to change mount points can be properly managed with a single service. You will likely still want to reorder and possibly resize the current rrd issue, but I would suggest these sorts of changes going forward.
If however, you are having issues with checks that have not altered performance data input, then we have another issue, let us know if that is the case.
Re: Check Results Truncated - Corrupting MRTG data
Posted: Tue Dec 01, 2015 4:11 pm
by mberkley
Hey Guys,
We are back again with this exact same issue. Are there any solutions for the newest Nagios revision that we can implement? I'm terribly scared to implement anything from the former revision, especially with the challenges that we had with this current upgrade.
sreinhardt wrote:In this case, it is actually neither nagios nor mrtg technically doing anything wrong. The issue here, is one of several potentially. Biggest one is that rrds are created with a finite number of tracks(per mount point in this case) when the service is initially created. Adding data to rrds does not take into account adding new tracks, or reordering based on labels. Due to this, rrdtool is likely unable to populate your rrd with any meaningful data if any at all. If it is actually populating data, it is very likely that the track to mountpoint alignment originally setup is quite off. I see two possible resolutions, if this is the case.
1) resize the rrd with the script found at the faq post for correcting track changes with check_ping and check_icmp. This will actually fix this for any rrds with less data than tracks. However, this will not correct your labeling to rrd track differences, and there is not really a way to do that. The issue with this approach, is that it is likely to happen again, and cause further issues.
http://support.nagios.com/wiki/index.ph ... 14_Upgrade
http://assets.nagios.com/downloads/nagi ... ds_fix.zip
2) The alternative that I would highly suggest, is to create a base service with mount points that should never change such as / & /boot, etc. Then for system specific mounts and mounts that may be added or removed, you can create another single grouping if the are unlikely to change, or several individual services for additional mount points. This avoids ever having to resize rrds, as the base mount points never change, and dynamic or very likely to change mount points can be properly managed with a single service. You will likely still want to reorder and possibly resize the current rrd issue, but I would suggest these sorts of changes going forward.
If however, you are having issues with checks that have not altered performance data input, then we have another issue, let us know if that is the case.
Re: Check Results Truncated - Corrupting MRTG data
Posted: Wed Dec 02, 2015 10:13 am
by scottwilkerson
mberkley wrote:Hey Guys,
We are back again with this exact same issue. Are there any solutions for the newest Nagios revision that we can implement? I'm terribly scared to implement anything from the former revision, especially with the challenges that we had with this current upgrade.
sreinhardt wrote:In this case, it is actually neither nagios nor mrtg technically doing anything wrong. The issue here, is one of several potentially. Biggest one is that rrds are created with a finite number of tracks(per mount point in this case) when the service is initially created. Adding data to rrds does not take into account adding new tracks, or reordering based on labels. Due to this, rrdtool is likely unable to populate your rrd with any meaningful data if any at all. If it is actually populating data, it is very likely that the track to mountpoint alignment originally setup is quite off. I see two possible resolutions, if this is the case.
1) resize the rrd with the script found at the faq post for correcting track changes with check_ping and check_icmp. This will actually fix this for any rrds with less data than tracks. However, this will not correct your labeling to rrd track differences, and there is not really a way to do that. The issue with this approach, is that it is likely to happen again, and cause further issues.
http://support.nagios.com/wiki/index.ph ... 14_Upgrade
http://assets.nagios.com/downloads/nagi ... ds_fix.zip
2) The alternative that I would highly suggest, is to create a base service with mount points that should never change such as / & /boot, etc. Then for system specific mounts and mounts that may be added or removed, you can create another single grouping if the are unlikely to change, or several individual services for additional mount points. This avoids ever having to resize rrds, as the base mount points never change, and dynamic or very likely to change mount points can be properly managed with a single service. You will likely still want to reorder and possibly resize the current rrd issue, but I would suggest these sorts of changes going forward.
If however, you are having issues with checks that have not altered performance data input, then we have another issue, let us know if that is the case.
This is still the best solution if you are dynamically changing the quantity of datasources being sent in the perfdata of your checks.
Re: Check Results Truncated - Corrupting MRTG data
Posted: Thu Dec 03, 2015 1:14 pm
by mberkley
So in short...
1. This will allow the full amount of characters, which in hope will allow MRTG to create the graph?
2. This will be compatible with the new version of Nagios....
Re: Check Results Truncated - Corrupting MRTG data
Posted: Thu Dec 03, 2015 5:04 pm
by rkennedy
1 & 2. The script above should work again as it's not version specific.