Page 1 of 1
Empty Top Talkers and Data? (cont)
Posted: Wed Dec 14, 2016 6:40 pm
by CFT6Server
In reference to this thread:
https://support.nagios.com/forum/viewto ... 8&start=10
I'm still working on this issue as it seems that the time is set correctly on the network devices (which is a Cisco device). Time is correct on the device and the times are still showing 1969.
Last when we checked:
Code: Select all
#sh clock
Load for five secs: 23%/0%; one minute: 26%; five minutes: 29%
Time source is NTP, 11:38:40.368 PST Mon Dec 12 2016
11:38:40.368 PST Mon Dec 12 2016
#
#sh ntp associations
Load for five secs: 23%/0%; one minute: 26%; five minutes: 29%
Time source is NTP, 11:38:52.249 PST Mon Dec 12 2016
Still seeing these message:
Code: Select all
1969-12-31 16:00:00.000 IGNORE Ignore TCP 10.65.64.55:61784 -> 10.65.68.6:1433 0.0.0.0:0 -> 0.0.0.0:0 184 0
1969-12-31 16:00:00.000 IGNORE Ignore TCP 10.65.68.6:1433 -> 10.65.64.55:61784 0.0.0.0:0 -> 0.0.0.0:0 168 0
Ideas? We have other NA in the environment and seems like the configuration is nearly identical.
Re: Empty Top Talkers and Data? (cont)
Posted: Thu Dec 15, 2016 12:43 pm
by tgriep
In Netflow, the sending device has to transmit certain tags to the NNA server telling it the format for the date time settings it will be receiving and your device may be not sending them because of a configuration issue or that the device has a bug causing the same issue.
What is the make and model number of the device sending the flow data?
Also, what is the firmware version it is running?
Can you post the configuration settings for the device as well?
Re: Empty Top Talkers and Data? (cont)
Posted: Thu Dec 15, 2016 4:55 pm
by CFT6Server
So this is a Cisco Switch C6824-X-LE-40G
Cisco IOS Software, c6848x Software (c6848x-IPSERVICESK9-M), Version 15.2(2)SY1
The netflow is collected by another team with another product, and that seems to be working. So it could be the date time format.... although the format might not cause the year and time to be in this pattern?
Re: Empty Top Talkers and Data? (cont)
Posted: Thu Dec 15, 2016 5:27 pm
by bwallace
Thanks for the details. To clarify, the smoking gun here is when you run nfdump -r on the nfcapd file -- here you see the bad time stamp as well so this means the device is the very likely culprit.
Going forward, please post the configuration settings for the sending device and run a simulataneus tcpdump on both the Cisco and NNA devices and send in the corresponding pcap files. This will tell us just what format /templates etc, the device is telling NA to use.
ON NNA:
tcpdump -s 0 -i any src <Cisco ip> and port <source port> -w NNA.pcap
On the Cisco device:
tcpdump -s 0 -i any dst <nna ip> and port <source port> -w Cicso.pcap
Let the captures run for about 10 minutes and send them for analysis when complete.
Re: Empty Top Talkers and Data? (cont)
Posted: Thu Dec 15, 2016 5:52 pm
by CFT6Server
I have PM'd you a short pcap on the NNA side. Looking at the data, timestamp seems correct....
pcapflow.JPG
Config
Code: Select all
#sh flow exporter NAGIOS
Load for five secs: 26%/0%; one minute: 26%; five minutes: 28%
Time source is NTP, 10:04:15.010 PST Wed Dec 14 2016
Flow Exporter NAGIOS:
Description: exporter for NAGIOS COLLECTOR
Export protocol: NetFlow Version 9
Transport Configuration:
Destination IP address: 10.65.32.9
VRF label: APP-001
Source IP address: 10.65.16.7
Source Interface: Loopback0
Transport Protocol: UDP
Destination Port: 9996
Source Port: 62425
DSCP: 0x0
TTL: 255
Output Features: Not Used
CONFIGURATION:
flow exporter NAGIOS
description exporter for NAGIOS COLLECTOR
destination 10.65.32.9 vrf APP-001
source Loopback0
transport udp 9996
template data timeout 60
Re: Empty Top Talkers and Data? (cont)
Posted: Fri Dec 16, 2016 11:15 am
by bwallace
Thanks for the capture. It was run for only 26 seconds so it is inconclusive. Sure the time stamp you pointed out is correct but that capture needs to run for at least five minutes so we can see the template that the device is sending.
For now, a capture on just the NNA machine should suffice but let it run for at least five minutes and PM it to me - thanks -
Re: Empty Top Talkers and Data? (cont)
Posted: Fri Dec 16, 2016 2:18 pm
by CFT6Server
Looks like I cannot send the file as it is > 1MB. If there's an alternative method, please provide that.
As for the template, here's a snapshot of the capture.
templateID.JPG
Looks like there's a mix of template ID: 257, 258, 259
I also did similar captures on another instance and the pcap data looks considerably different.
Site 1 is the instance we currently troubleshooting...
site1.png
Site 2 is the current NNA sources that is working fine for another instance.
site2.png
Interesting to see that the one I am having troubles with comes in as CFLOW and then working site is just raw UDP? Is this expected? or perhaps due to different model/firmware?
Re: Empty Top Talkers and Data? (cont)
Posted: Fri Dec 16, 2016 2:50 pm
by tgriep
Without seeing the configuration, we are only guessing what the issue it but the device needs to send to the NNA server a template that tells the NNA server the format of the data.
Especially if the device is sending version 9 of netflow or IPFIX.
In that template, there is a tag that has the time format in the netflow data and that is either not getting sent from your device or that it is sending the incorrect one.
The differences between the CFLOW and the UDP protocol from wireshark it that wireshark found the cflow template and decoded it for you.
The one that did not decode, did not receive the template in the time the capture was run.
Can you zip of the capture for SITE 1 and PM it to me or will it still be too large?
Re: Empty Top Talkers and Data? (cont)
Posted: Fri Dec 16, 2016 3:43 pm
by CFT6Server
So here's site 2... so you could be right here as the template is different, mainly the first_switched and last_switched fields.
site2 flow template.JPG
If you need to see the config of site 1, it was included a few replies ago, but here it is again.
Code: Select all
#sh flow exporter NAGIOS
Load for five secs: 26%/0%; one minute: 26%; five minutes: 28%
Time source is NTP, 10:04:15.010 PST Wed Dec 14 2016
Flow Exporter NAGIOS:
Description: exporter for NAGIOS COLLECTOR
Export protocol: NetFlow Version 9
Transport Configuration:
Destination IP address: 10.65.32.9
VRF label: APP-001
Source IP address: 10.65.16.7
Source Interface: Loopback0
Transport Protocol: UDP
Destination Port: 9996
Source Port: 62425
DSCP: 0x0
TTL: 255
Output Features: Not Used
CONFIGURATION:
flow exporter NAGIOS
description exporter for NAGIOS COLLECTOR
destination 10.65.32.9 vrf APP-001
source Loopback0
transport udp 9996
template data timeout 60
Re: Empty Top Talkers and Data? (cont)
Posted: Mon Dec 19, 2016 1:39 pm
by bwallace
So something of interest in the captures. You have two devices sending to NNA as one source and while this is supported, the Sysuptime is of course different between them and I wonder if this confuses nfdump during nfcapd processing time. I recommend as a test you reconfigure NNA so that 16.8 is an independent source using it's own unique port and 16.7 as a second independent source using it's own unique port. Then see if the time is accurately represented in the NNA queries / reports.
Notice this sequence in the capture (when filtered in Wireshark for cflow.template_id)
20 14:07:22.043124
10.65.16.8 52325 10.65.32.9 9996 CFLOW 116 total: 1 (v9) record Obs-Domain-ID= 1 [Data-Template:257]
SysUptime: 1276531.925917088 seconds
14.77 days
135 14:07:35.828944
10.65.16.7 62425 10.65.32.9 9996 CFLOW 116 total: 1 (v9) record Obs-Domain-ID= 1 [Data-Template:259]
SysUptime: 2298291.484244640 seconds
26 days
304 14:08:01.231619 10.65.16.7 62425 10.65.32.9 9996 CFLOW 116 total: 1 (v9) record Obs-Domain-ID= 1 [Data-Template:258]
SysUptime: 2298317.509644640 seconds
26 days
341 14:08:05.525054 10.65.16.8 52325 10.65.32.9 9996 CFLOW 116 total: 1 (v9) record Obs-Domain-ID= 1 [Data-Template:258]
SysUptime: 1276574.969397088 seconds
14 days
cflow_template_id filter.jpg