Page 1 of 3
Post Upgrade Issues on 2014R1.3
Posted: Mon Jul 28, 2014 4:19 am
by chriscamm
Hi Since I upgraded to 2014R1.3 I have getting
/var/lib/mrtg/xxx.xxx.xxx.xxx_17.rrd does not exist.\n
This is on nearly all of my port of all switches (Currently checking over 100 switches with lots of ports)
If I run the command from the command line it works
Code: Select all
/usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/xxx.xxx.xxx.xxx_17.rrd -w 50,50 -c 80,80 -l M
OK - Current BW in: 0Mbps Out: 0Mbps|in=.001633Mb/s;50;80 out=.001357Mb/s;50;80
When I run the command from the test option in CCM it also returns the correct data back.
With regards to the
at the end of all data collected this in not on every check and I have put in place the mod_gearman and performance data tweaks outlined the FAQ but this still appears in random data returns.
Thanks in advance.
Chris
Re: Post Upgrade Issues on 2014R1.3
Posted: Mon Jul 28, 2014 1:17 pm
by tmcdonald
chriscamm wrote:[...]
This is on nearly all of my port of all switches (Currently checking over 100 switches with lots of ports)
[...] this in not on every check and I have put in place the mod_gearman and performance data tweaks outlined the FAQ
Is there any pattern as to which hosts/services are displaying this behavior? Only certain ports or devices? Does it always appear or only occasionally for a given service?
Re: Post Upgrade Issues on 2014R1.3
Posted: Mon Jul 28, 2014 3:59 pm
by chriscamm
Hi,
Sent you a PM with the last 100 errors
Thanks
Chris
Re: Post Upgrade Issues on 2014R1.3
Posted: Mon Jul 28, 2014 4:03 pm
by tmcdonald
I saw the last 100 errors, but that only tells me what is *wrong* and not what is *right*. I am trying to determine a pattern, for example if every other check for a service is coming back as UNKNOWN, or maybe every 5 minutes it is OK but is otherwise UNKNOWN, if it is for a certain host only, etc. Since the log is only showing what is wrong, I can't really get a feel for what might be connecting the errors.
Re: Post Upgrade Issues on 2014R1.3
Posted: Mon Jul 28, 2014 4:55 pm
by chriscamm
Hi,
Here is a snapshot
Code: Select all
n
Avaya IP 500 BandwidthThis service has commentsThis service is flapping Unknown 8m 20s 2/5 28/07/2014 22:46:48 /var/lib/mrtg/172.20.50.250_2.rrd does not exist.\n
ESX P1 Bandwidth Unknown 40m 2s 5/5 28/07/2014 22:41:46 /var/lib/mrtg/172.20.50.250_21.rrd does not exist.\n
ESX P2 BandwidthThis service has commentsThis service is flapping Unknown 9m 30s 3/5 28/07/2014 22:46:48 /var/lib/mrtg/172.20.50.250_22.rrd does not exist.\n
ESX P3 BandwidthThis service has commentsThis service is flapping Unknown 13m 22s 5/5 28/07/2014 22:46:09 /var/lib/mrtg/172.20.50.250_23.rrd does not exist.\n
ESX P4 BandwidthThis service has commentsThis service is flapping Unknown 8m 13s 2/5 28/07/2014 22:46:48 /var/lib/mrtg/172.20.50.250_24.rrd does not exist.\n
Firewall BandwidthThis service has commentsThis service is flapping Unknown 6m 55s 1/5 28/07/2014 22:46:44 /var/lib/mrtg/172.20.50.250_20.rrd does not exist.\n
Interlink P2 Bandwidth Unknown 8m 57s 2/5 28/07/2014 22:46:09 /var/lib/mrtg/172.20.50.250_4.rrd does not exist.\n
AG Interlink port1 Bandwidth Unknown 7m 33s 1/5 28/07/2014 22:46:06 /var/lib/mrtg/172.20.50.251_1.rrd does not exist.\n
Port 1 Bandwidth Unknown 6m 55s 1/5 28/07/2014 22:46:44 /var/lib/mrtg/172.20.50.251_1.rrd does not exist.\n
Port 12 BandwidthThis service has commentsThis service is flapping Unknown 7m 36s 1/5 28/07/2014 22:46:03 /var/lib/mrtg/172.20.50.251_12.rrd does not exist.\n
Port 13 Bandwidth Unknown 9m 29s 3/5 28/07/2014 22:46:48 /var/lib/mrtg/172.20.50.251_13.rrd does not exist.\n
Port 14 BandwidthThis service has commentsThis service is flapping Unknown 9m 5s 2/5 28/07/2014 22:46:09 /var/lib/mrtg/172.20.50.251_14.rrd does not exist.\n
Port 18 BandwidthThis service has commentsThis service is flapping Unknown 41m 34s 5/5 28/07/2014 22:45:29 /var/lib/mrtg/172.20.50.251_18.rrd does not exist.\n
Port 19 BandwidthThis service has commentsThis service is flapping Unknown 7m 33s 1/5 28/07/2014 22:46:06 /var/lib/mrtg/172.20.50.251_19.rrd does not exist.\n
Port 2 Bandwidth Unknown 6m 55s 1/5 28/07/2014 22:46:44 /var/lib/mrtg/172.20.50.251_2.rrd does not exist.\n
Port 23 BandwidthThis service has commentsThis service is flapping Unknown 8m 58s 2/5 28/07/2014 22:46:09 /var/lib/mrtg/172.20.50.251_23.rrd does not exist.\n
Port 4 BandwidthThis service has commentsThis service is flapping Unknown 6m 55s 1/5 28/07/2014 22:46:44 /var/lib/mrtg/172.20.50.251_4.rrd does not exist.\n
Port 7 BandwidthThis service has commentsThis service is flapping Unknown 15m 49s 5/5 28/07/2014 22:43:15 /var/lib/mrtg/172.20.50.251_7.rrd does not exist.\n
Port 9 BandwidthThis service has commentsThis service is flapping Unknown 19m 41s 5/5 28/07/2014 22:46:06 /var/lib/mrtg/172.20.50.251_9.rrd does not exist.\n
I hope this is showing the info you need. 172.20.50.250 is a 24 port switch as is 172.20.50.251
Thanks
Chris
Re: Post Upgrade Issues on 2014R1.3
Posted: Tue Jul 29, 2014 10:14 am
by lmiltchev
Can you also run the following commands and show us the output?
Re: Post Upgrade Issues on 2014R1.3
Posted: Tue Jul 29, 2014 5:05 pm
by chriscamm
Code: Select all
[root@qualngs ~]# uname -a
Linux nagios.local 2.6.32-431.11.2.el6.x86_64 #1 SMP Tue Mar 25 19:59:55 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
[root@qualngs ~]# cat /etc/*release
CentOS release 6.5 (Final)
CentOS release 6.5 (Final)
CentOS release 6.5 (Final)
Re: Post Upgrade Issues on 2014R1.3
Posted: Wed Jul 30, 2014 12:51 pm
by lmiltchev
I haven't been able to reproduce the issue so far. So you never used to get "/var/lib/mrtg/xxx.xxx.xxx.xxx_17.rrd does not exist.\n" messages prior to the upgrade to 1.3?
Can you also run the following commands and show the output?
Code: Select all
/usr/bin/check_gearman -V
LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg
Re: Post Upgrade Issues on 2014R1.3
Posted: Wed Jul 30, 2014 5:15 pm
by chriscamm
Hi,
I don't remember ever seeing these errors before. However, I have had to install mod_gearman as the Nagios Server could not cope with the amount of checks post upgrade from 2012 to 2014
Thanks
Chris
Code: Select all
/usr/bin/check_gearman -V
check_gearman: version 1.4_nagios4 running on libgearman 0.25
[root@qualngs ~]# LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg
SNMP Error:
no response received
SNMPv2c_Session (remote host: "172.20.10.254" [172.20.10.254].161)
community: "public"
request ID: 1366168290
PDU bufsize: 8000 bytes
timeout: 2s
retries: 5
backoff: 1)
at /usr/bin/../lib/mrtg2/SNMP_util.pm line 497
SNMPGET Problem for ifHCInOctets.1 ifHCOutOctets.1 on [email protected]:::::2:v4only
at /usr/bin/mrtg line 2330
2014-07-30 23:12:25: WARNING: skipping because at least the query for ifHCInOctets.1 on 172.20.10.254 did not succeed
2014-07-30 23:12:25: WARNING: no data for ifHCInOctets&ifHCOutOctets:[email protected]. Skipping further queries for Host 172.20.10.254 in this round.
SNMP Error:
Received SNMP response with error code
error status: noSuchName
index 1 (OID: 1.3.6.1.2.1.2.2.1.10.56)
SNMPv1_Session (remote host: "172.20.10.250" [172.20.10.250].161)
community: "qls-priv"
request ID: 876173011
PDU bufsize: 8000 bytes
timeout: 2s
retries: 5
backoff: 1)
at /usr/bin/../lib/mrtg2/SNMP_util.pm line 497
SNMPGET Problem for ifInOctets.56 ifOutOctets.56 on [email protected]:::::1:v4only
at /usr/bin/mrtg line 2330
2014-07-30 23:12:51: ERROR: Target[qualsw1_56][_IN_] ' $target->[49]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[qualsw1_56][_OUT_] ' $target->[49]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[217.111.235.2_2][_IN_] ' $target->[57]{$mode} $snmpversion' (warn): (Missing operator before $snmpversion?)
2014-07-30 23:12:51: ERROR: Target[217.111.235.2_2][_OUT_] ' $target->[57]{$mode} $snmpversion' (warn): (Missing operator before $snmpversion?)
2014-07-30 23:12:51: ERROR: Target[217.111.235.2_3][_IN_] ' $target->[58]{$mode} $snmpversion' (warn): (Missing operator before $snmpversion?)
2014-07-30 23:12:51: ERROR: Target[217.111.235.2_3][_OUT_] ' $target->[58]{$mode} $snmpversion' (warn): (Missing operator before $snmpversion?)
2014-07-30 23:12:51: ERROR: Target[217.111.235.2_5][_IN_] ' $target->[59]{$mode} $snmpversion' (warn): (Missing operator before $snmpversion?)
2014-07-30 23:12:51: ERROR: Target[217.111.235.2_5][_OUT_] ' $target->[59]{$mode} $snmpversion' (warn): (Missing operator before $snmpversion?)
2014-07-30 23:12:51: ERROR: Target[217.111.235.2_10][_IN_] ' $target->[60]{$mode} $snmpversion' (warn): (Missing operator before $snmpversion?)
2014-07-30 23:12:51: ERROR: Target[217.111.235.2_10][_OUT_] ' $target->[60]{$mode} $snmpversion' (warn): (Missing operator before $snmpversion?)
2014-07-30 23:12:51: ERROR: Target[217.111.235.2_11][_IN_] ' $target->[61]{$mode} $snmpversion' (warn): (Missing operator before $snmpversion?)
2014-07-30 23:12:51: ERROR: Target[217.111.235.2_11][_OUT_] ' $target->[61]{$mode} $snmpversion' (warn): (Missing operator before $snmpversion?)
2014-07-30 23:12:51: ERROR: Target[217.111.235.2_12][_IN_] ' $target->[62]{$mode} $snmpversion' (warn): (Missing operator before $snmpversion?)
2014-07-30 23:12:51: ERROR: Target[217.111.235.2_12][_OUT_] ' $target->[62]{$mode} $snmpversion' (warn): (Missing operator before $snmpversion?)
2014-07-30 23:12:51: ERROR: Target[217.111.235.2_13][_IN_] ' $target->[63]{$mode} $snmpversion' (warn): (Missing operator before $snmpversion?)
2014-07-30 23:12:51: ERROR: Target[217.111.235.2_13][_OUT_] ' $target->[63]{$mode} $snmpversion' (warn): (Missing operator before $snmpversion?)
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_1][_IN_] ' $target->[187]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_1][_OUT_] ' $target->[187]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_2][_IN_] ' $target->[188]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_2][_OUT_] ' $target->[188]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_4][_IN_] ' $target->[189]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_4][_OUT_] ' $target->[189]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_5][_IN_] ' $target->[190]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_5][_OUT_] ' $target->[190]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_6][_IN_] ' $target->[191]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_6][_OUT_] ' $target->[191]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_7][_IN_] ' $target->[192]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_7][_OUT_] ' $target->[192]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_8][_IN_] ' $target->[193]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_8][_OUT_] ' $target->[193]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_9][_IN_] ' $target->[194]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_9][_OUT_] ' $target->[194]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_10][_IN_] ' $target->[195]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_10][_OUT_] ' $target->[195]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_11][_IN_] ' $target->[196]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_11][_OUT_] ' $target->[196]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_12][_IN_] ' $target->[197]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_12][_OUT_] ' $target->[197]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_13][_IN_] ' $target->[198]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_13][_OUT_] ' $target->[198]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_14][_IN_] ' $target->[199]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_14][_OUT_] ' $target->[199]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.250_56][_IN_] ' $target->[251]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.250_56][_OUT_] ' $target->[251]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.17.0.3_117][_IN_] ' $target->[503]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.17.0.3_117][_OUT_] ' $target->[503]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.17.0.22_11149][_IN_] ' $target->[822]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.17.0.22_11149][_OUT_] ' $target->[822]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.17.0.22_11150][_IN_] ' $target->[823]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.17.0.22_11150][_OUT_] ' $target->[823]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.17.0.11_10625][_IN_] ' $target->[1233]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.17.0.11_10625][_OUT_] ' $target->[1233]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.17.0.11_10626][_IN_] ' $target->[1234]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.17.0.11_10626][_OUT_] ' $target->[1234]{$mode} ' did not eval into defined data
Re: Post Upgrade Issues on 2014R1.3
Posted: Thu Jul 31, 2014 9:29 am
by tmcdonald
Did these issues show up right after the upgrade, or only after you set up mod_gearman? The randomness of the messages makes me think maybe a core or gearman worker is messing up.