Post Upgrade Issues on 2014R1.3

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
Locked
chriscamm
Posts: 72
Joined: Thu Aug 22, 2013 6:12 am

Post Upgrade Issues on 2014R1.3

Post by chriscamm »

Hi Since I upgraded to 2014R1.3 I have getting

/var/lib/mrtg/xxx.xxx.xxx.xxx_17.rrd does not exist.\n

This is on nearly all of my port of all switches (Currently checking over 100 switches with lots of ports)

If I run the command from the command line it works

Code: Select all

 /usr/local/nagios/libexec/check_rrdtraf -f /var/lib/mrtg/xxx.xxx.xxx.xxx_17.rrd -w 50,50 -c 80,80 -l M
OK - Current BW in: 0Mbps Out: 0Mbps|in=.001633Mb/s;50;80 out=.001357Mb/s;50;80
When I run the command from the test option in CCM it also returns the correct data back.

With regards to the

Code: Select all

\n
at the end of all data collected this in not on every check and I have put in place the mod_gearman and performance data tweaks outlined the FAQ but this still appears in random data returns.

Thanks in advance.

Chris
Top
tmcdonald
Posts: 9117
Joined: Mon Sep 23, 2013 8:40 am

Re: Post Upgrade Issues on 2014R1.3

Post by tmcdonald »

chriscamm wrote:[...]
This is on nearly all of my port of all switches (Currently checking over 100 switches with lots of ports)

[...] this in not on every check and I have put in place the mod_gearman and performance data tweaks outlined the FAQ
Is there any pattern as to which hosts/services are displaying this behavior? Only certain ports or devices? Does it always appear or only occasionally for a given service?
Former Nagios employee
Top
chriscamm
Posts: 72
Joined: Thu Aug 22, 2013 6:12 am

Re: Post Upgrade Issues on 2014R1.3

Post by chriscamm »

Hi,

Sent you a PM with the last 100 errors

Thanks

Chris
Top
tmcdonald
Posts: 9117
Joined: Mon Sep 23, 2013 8:40 am

Re: Post Upgrade Issues on 2014R1.3

Post by tmcdonald »

I saw the last 100 errors, but that only tells me what is *wrong* and not what is *right*. I am trying to determine a pattern, for example if every other check for a service is coming back as UNKNOWN, or maybe every 5 minutes it is OK but is otherwise UNKNOWN, if it is for a certain host only, etc. Since the log is only showing what is wrong, I can't really get a feel for what might be connecting the errors.
Former Nagios employee
Top
chriscamm
Posts: 72
Joined: Thu Aug 22, 2013 6:12 am

Re: Post Upgrade Issues on 2014R1.3

Post by chriscamm »

Hi,

Here is a snapshot

Code: Select all

n
Avaya IP 500 BandwidthThis service has commentsThis service is flapping	Unknown	8m 20s	2/5	28/07/2014 22:46:48	/var/lib/mrtg/172.20.50.250_2.rrd does not exist.\n
ESX P1 Bandwidth	Unknown	40m 2s	5/5	28/07/2014 22:41:46	/var/lib/mrtg/172.20.50.250_21.rrd does not exist.\n
ESX P2 BandwidthThis service has commentsThis service is flapping	Unknown	9m 30s	3/5	28/07/2014 22:46:48	/var/lib/mrtg/172.20.50.250_22.rrd does not exist.\n
ESX P3 BandwidthThis service has commentsThis service is flapping	Unknown	13m 22s	5/5	28/07/2014 22:46:09	/var/lib/mrtg/172.20.50.250_23.rrd does not exist.\n
ESX P4 BandwidthThis service has commentsThis service is flapping	Unknown	8m 13s	2/5	28/07/2014 22:46:48	/var/lib/mrtg/172.20.50.250_24.rrd does not exist.\n
Firewall BandwidthThis service has commentsThis service is flapping	Unknown	6m 55s	1/5	28/07/2014 22:46:44	/var/lib/mrtg/172.20.50.250_20.rrd does not exist.\n
Interlink P2 Bandwidth	Unknown	8m 57s	2/5	28/07/2014 22:46:09	/var/lib/mrtg/172.20.50.250_4.rrd does not exist.\n
AG Interlink port1 Bandwidth	Unknown	7m 33s	1/5	28/07/2014 22:46:06	/var/lib/mrtg/172.20.50.251_1.rrd does not exist.\n
Port 1 Bandwidth	Unknown	6m 55s	1/5	28/07/2014 22:46:44	/var/lib/mrtg/172.20.50.251_1.rrd does not exist.\n
Port 12 BandwidthThis service has commentsThis service is flapping	Unknown	7m 36s	1/5	28/07/2014 22:46:03	/var/lib/mrtg/172.20.50.251_12.rrd does not exist.\n
Port 13 Bandwidth	Unknown	9m 29s	3/5	28/07/2014 22:46:48	/var/lib/mrtg/172.20.50.251_13.rrd does not exist.\n
Port 14 BandwidthThis service has commentsThis service is flapping	Unknown	9m 5s	2/5	28/07/2014 22:46:09	/var/lib/mrtg/172.20.50.251_14.rrd does not exist.\n
Port 18 BandwidthThis service has commentsThis service is flapping	Unknown	41m 34s	5/5	28/07/2014 22:45:29	/var/lib/mrtg/172.20.50.251_18.rrd does not exist.\n
Port 19 BandwidthThis service has commentsThis service is flapping	Unknown	7m 33s	1/5	28/07/2014 22:46:06	/var/lib/mrtg/172.20.50.251_19.rrd does not exist.\n
Port 2 Bandwidth	Unknown	6m 55s	1/5	28/07/2014 22:46:44	/var/lib/mrtg/172.20.50.251_2.rrd does not exist.\n
Port 23 BandwidthThis service has commentsThis service is flapping	Unknown	8m 58s	2/5	28/07/2014 22:46:09	/var/lib/mrtg/172.20.50.251_23.rrd does not exist.\n
Port 4 BandwidthThis service has commentsThis service is flapping	Unknown	6m 55s	1/5	28/07/2014 22:46:44	/var/lib/mrtg/172.20.50.251_4.rrd does not exist.\n
Port 7 BandwidthThis service has commentsThis service is flapping	Unknown	15m 49s	5/5	28/07/2014 22:43:15	/var/lib/mrtg/172.20.50.251_7.rrd does not exist.\n
Port 9 BandwidthThis service has commentsThis service is flapping	Unknown	19m 41s	5/5	28/07/2014 22:46:06	/var/lib/mrtg/172.20.50.251_9.rrd does not exist.\n
I hope this is showing the info you need. 172.20.50.250 is a 24 port switch as is 172.20.50.251

Thanks

Chris
Top
User avatar
lmiltchev
Bugs find me
Posts: 13589
Joined: Mon May 23, 2011 12:15 pm

Re: Post Upgrade Issues on 2014R1.3

Post by lmiltchev »

Can you also run the following commands and show us the output?

Code: Select all

uname -a
cat /etc/*release
Be sure to check out our Knowledgebase for helpful articles and solutions!
Top
chriscamm
Posts: 72
Joined: Thu Aug 22, 2013 6:12 am

Re: Post Upgrade Issues on 2014R1.3

Post by chriscamm »

Code: Select all

[root@qualngs ~]# uname -a
Linux nagios.local 2.6.32-431.11.2.el6.x86_64 #1 SMP Tue Mar 25 19:59:55 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
[root@qualngs ~]# cat /etc/*release
CentOS release 6.5 (Final)
CentOS release 6.5 (Final)
CentOS release 6.5 (Final)
Top
User avatar
lmiltchev
Bugs find me
Posts: 13589
Joined: Mon May 23, 2011 12:15 pm

Re: Post Upgrade Issues on 2014R1.3

Post by lmiltchev »

I haven't been able to reproduce the issue so far. So you never used to get "/var/lib/mrtg/xxx.xxx.xxx.xxx_17.rrd does not exist.\n" messages prior to the upgrade to 1.3?
Can you also run the following commands and show the output?

Code: Select all

/usr/bin/check_gearman -V
LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg
Be sure to check out our Knowledgebase for helpful articles and solutions!
Top
chriscamm
Posts: 72
Joined: Thu Aug 22, 2013 6:12 am

Re: Post Upgrade Issues on 2014R1.3

Post by chriscamm »

Hi,

I don't remember ever seeing these errors before. However, I have had to install mod_gearman as the Nagios Server could not cope with the amount of checks post upgrade from 2012 to 2014

Thanks

Chris

Code: Select all

/usr/bin/check_gearman -V
check_gearman: version 1.4_nagios4 running on libgearman 0.25

[root@qualngs ~]# LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg
SNMP Error:
no response received
SNMPv2c_Session (remote host: "172.20.10.254" [172.20.10.254].161)
                   community: "public"
                  request ID: 1366168290
                 PDU bufsize: 8000 bytes
                     timeout: 2s
                     retries: 5
                     backoff: 1)
 at /usr/bin/../lib/mrtg2/SNMP_util.pm line 497
SNMPGET Problem for ifHCInOctets.1 ifHCOutOctets.1 on [email protected]:::::2:v4only
 at /usr/bin/mrtg line 2330
2014-07-30 23:12:25: WARNING: skipping because at least the query for ifHCInOctets.1 on  172.20.10.254 did not succeed
2014-07-30 23:12:25: WARNING: no data for ifHCInOctets&ifHCOutOctets:[email protected]. Skipping further queries for Host 172.20.10.254 in this round.
SNMP Error:
Received SNMP response with error code
  error status: noSuchName
  index 1 (OID: 1.3.6.1.2.1.2.2.1.10.56)
SNMPv1_Session (remote host: "172.20.10.250" [172.20.10.250].161)
                  community: "qls-priv"
                 request ID: 876173011
                PDU bufsize: 8000 bytes
                    timeout: 2s
                    retries: 5
                    backoff: 1)
 at /usr/bin/../lib/mrtg2/SNMP_util.pm line 497
SNMPGET Problem for ifInOctets.56 ifOutOctets.56 on [email protected]:::::1:v4only
 at /usr/bin/mrtg line 2330
2014-07-30 23:12:51: ERROR: Target[qualsw1_56][_IN_] ' $target->[49]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[qualsw1_56][_OUT_] ' $target->[49]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[217.111.235.2_2][_IN_] ' $target->[57]{$mode} $snmpversion' (warn): (Missing operator before  $snmpversion?)
2014-07-30 23:12:51: ERROR: Target[217.111.235.2_2][_OUT_] ' $target->[57]{$mode} $snmpversion' (warn): (Missing operator before  $snmpversion?)
2014-07-30 23:12:51: ERROR: Target[217.111.235.2_3][_IN_] ' $target->[58]{$mode} $snmpversion' (warn): (Missing operator before  $snmpversion?)
2014-07-30 23:12:51: ERROR: Target[217.111.235.2_3][_OUT_] ' $target->[58]{$mode} $snmpversion' (warn): (Missing operator before  $snmpversion?)
2014-07-30 23:12:51: ERROR: Target[217.111.235.2_5][_IN_] ' $target->[59]{$mode} $snmpversion' (warn): (Missing operator before  $snmpversion?)
2014-07-30 23:12:51: ERROR: Target[217.111.235.2_5][_OUT_] ' $target->[59]{$mode} $snmpversion' (warn): (Missing operator before  $snmpversion?)
2014-07-30 23:12:51: ERROR: Target[217.111.235.2_10][_IN_] ' $target->[60]{$mode} $snmpversion' (warn): (Missing operator before  $snmpversion?)
2014-07-30 23:12:51: ERROR: Target[217.111.235.2_10][_OUT_] ' $target->[60]{$mode} $snmpversion' (warn): (Missing operator before  $snmpversion?)
2014-07-30 23:12:51: ERROR: Target[217.111.235.2_11][_IN_] ' $target->[61]{$mode} $snmpversion' (warn): (Missing operator before  $snmpversion?)
2014-07-30 23:12:51: ERROR: Target[217.111.235.2_11][_OUT_] ' $target->[61]{$mode} $snmpversion' (warn): (Missing operator before  $snmpversion?)
2014-07-30 23:12:51: ERROR: Target[217.111.235.2_12][_IN_] ' $target->[62]{$mode} $snmpversion' (warn): (Missing operator before  $snmpversion?)
2014-07-30 23:12:51: ERROR: Target[217.111.235.2_12][_OUT_] ' $target->[62]{$mode} $snmpversion' (warn): (Missing operator before  $snmpversion?)
2014-07-30 23:12:51: ERROR: Target[217.111.235.2_13][_IN_] ' $target->[63]{$mode} $snmpversion' (warn): (Missing operator before  $snmpversion?)
2014-07-30 23:12:51: ERROR: Target[217.111.235.2_13][_OUT_] ' $target->[63]{$mode} $snmpversion' (warn): (Missing operator before  $snmpversion?)
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_1][_IN_] ' $target->[187]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_1][_OUT_] ' $target->[187]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_2][_IN_] ' $target->[188]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_2][_OUT_] ' $target->[188]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_4][_IN_] ' $target->[189]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_4][_OUT_] ' $target->[189]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_5][_IN_] ' $target->[190]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_5][_OUT_] ' $target->[190]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_6][_IN_] ' $target->[191]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_6][_OUT_] ' $target->[191]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_7][_IN_] ' $target->[192]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_7][_OUT_] ' $target->[192]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_8][_IN_] ' $target->[193]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_8][_OUT_] ' $target->[193]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_9][_IN_] ' $target->[194]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_9][_OUT_] ' $target->[194]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_10][_IN_] ' $target->[195]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_10][_OUT_] ' $target->[195]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_11][_IN_] ' $target->[196]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_11][_OUT_] ' $target->[196]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_12][_IN_] ' $target->[197]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_12][_OUT_] ' $target->[197]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_13][_IN_] ' $target->[198]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_13][_OUT_] ' $target->[198]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_14][_IN_] ' $target->[199]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.254_14][_OUT_] ' $target->[199]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.250_56][_IN_] ' $target->[251]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.20.10.250_56][_OUT_] ' $target->[251]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.17.0.3_117][_IN_] ' $target->[503]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.17.0.3_117][_OUT_] ' $target->[503]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.17.0.22_11149][_IN_] ' $target->[822]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.17.0.22_11149][_OUT_] ' $target->[822]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.17.0.22_11150][_IN_] ' $target->[823]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.17.0.22_11150][_OUT_] ' $target->[823]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.17.0.11_10625][_IN_] ' $target->[1233]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.17.0.11_10625][_OUT_] ' $target->[1233]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.17.0.11_10626][_IN_] ' $target->[1234]{$mode} ' did not eval into defined data
2014-07-30 23:12:51: ERROR: Target[172.17.0.11_10626][_OUT_] ' $target->[1234]{$mode} ' did not eval into defined data
Top
tmcdonald
Posts: 9117
Joined: Mon Sep 23, 2013 8:40 am

Re: Post Upgrade Issues on 2014R1.3

Post by tmcdonald »

Did these issues show up right after the upgrade, or only after you set up mod_gearman? The randomness of the messages makes me think maybe a core or gearman worker is messing up.
Former Nagios employee
Top
Locked

Return to “Nagios XI”