Empty Top Talkers and Data? (cont)

[CFT6Server]

In reference to this thread: https://support.nagios.com/forum/viewto ... 8&start=10

I'm still working on this issue as it seems that the time is set correctly on the network devices (which is a Cisco device). Time is correct on the device and the times are still showing 1969.

Last when we checked:

Code: Select all

 #sh clock 
Load for five secs: 23%/0%; one minute: 26%; five minutes: 29%
Time source is NTP, 11:38:40.368 PST Mon Dec 12 2016

11:38:40.368 PST Mon Dec 12 2016
 #
 #sh ntp associations 
Load for five secs: 23%/0%; one minute: 26%; five minutes: 29%
Time source is NTP, 11:38:52.249 PST Mon Dec 12 2016

Still seeing these message:

Code: Select all

1969-12-31 16:00:00.000 IGNORE  Ignore TCP        10.65.64.55:61784 ->       10.65.68.6:1433           0.0.0.0:0     ->          0.0.0.0:0          184        0
1969-12-31 16:00:00.000 IGNORE  Ignore TCP         10.65.68.6:1433  ->      10.65.64.55:61784          0.0.0.0:0     ->          0.0.0.0:0          168        0

Ideas? We have other NA in the environment and seems like the configuration is nearly identical.

[tgriep]

In Netflow, the sending device has to transmit certain tags to the NNA server telling it the format for the date time settings it will be receiving and your device may be not sending them because of a configuration issue or that the device has a bug causing the same issue.
What is the make and model number of the device sending the flow data?
Also, what is the firmware version it is running?
Can you post the configuration settings for the device as well?

[CFT6Server]

So this is a Cisco Switch C6824-X-LE-40G
Cisco IOS Software, c6848x Software (c6848x-IPSERVICESK9-M), Version 15.2(2)SY1

The netflow is collected by another team with another product, and that seems to be working. So it could be the date time format.... although the format might not cause the year and time to be in this pattern?

[bwallace]

Thanks for the details. To clarify, the smoking gun here is when you run nfdump -r on the nfcapd file -- here you see the bad time stamp as well so this means the device is the very likely culprit.

Going forward, please post the configuration settings for the sending device and run a simulataneus tcpdump on both the Cisco and NNA devices and send in the corresponding pcap files. This will tell us just what format /templates etc, the device is telling NA to use.


ON NNA:

tcpdump -s 0 -i any src <Cisco ip> and port <source port> -w NNA.pcap



On the Cisco device:

tcpdump -s 0 -i any dst <nna ip> and port <source port> -w Cicso.pcap


Let the captures run for about 10 minutes and send them for analysis when complete.

[CFT6Server]

I have PM'd you a short pcap on the NNA side. Looking at the data, timestamp seems correct....

pcapflow.JPG

Config

Code: Select all

#sh flow exporter NAGIOS
Load for five secs: 26%/0%; one minute: 26%; five minutes: 28%
Time source is NTP, 10:04:15.010 PST Wed Dec 14 2016
Flow Exporter NAGIOS:
  Description:              exporter for  NAGIOS COLLECTOR
  Export protocol:          NetFlow Version 9
  Transport Configuration:
    Destination IP address: 10.65.32.9
    VRF label:              APP-001
    Source IP address:      10.65.16.7
    Source Interface:       Loopback0
    Transport Protocol:     UDP
    Destination Port:       9996
    Source Port:            62425
    DSCP:                   0x0
    TTL:                    255
    Output Features:        Not Used
      
      CONFIGURATION: 
      
 flow exporter NAGIOS
description exporter for  NAGIOS COLLECTOR
destination 10.65.32.9 vrf APP-001
source Loopback0
transport udp 9996
template data timeout 60

[bwallace]

Thanks for the capture. It was run for only 26 seconds so it is inconclusive. Sure the time stamp you pointed out is correct but that capture needs to run for at least five minutes so we can see the template that the device is sending.

For now, a capture on just the NNA machine should suffice but let it run for at least five minutes and PM it to me - thanks -

[CFT6Server]

Looks like I cannot send the file as it is > 1MB. If there's an alternative method, please provide that.

As for the template, here's a snapshot of the capture.

templateID.JPG

Looks like there's a mix of template ID: 257, 258, 259

I also did similar captures on another instance and the pcap data looks considerably different.
Site 1 is the instance we currently troubleshooting...

site1.png

Site 2 is the current NNA sources that is working fine for another instance.

site2.png

Interesting to see that the one I am having troubles with comes in as CFLOW and then working site is just raw UDP? Is this expected? or perhaps due to different model/firmware?

[tgriep]

Without seeing the configuration, we are only guessing what the issue it but the device needs to send to the NNA server a template that tells the NNA server the format of the data.
Especially if the device is sending version 9 of netflow or IPFIX.
In that template, there is a tag that has the time format in the netflow data and that is either not getting sent from your device or that it is sending the incorrect one.
The differences between the CFLOW and the UDP protocol from wireshark it that wireshark found the cflow template and decoded it for you.
The one that did not decode, did not receive the template in the time the capture was run.

Can you zip of the capture for SITE 1 and PM it to me or will it still be too large?

[CFT6Server]

So here's site 2... so you could be right here as the template is different, mainly the first_switched and last_switched fields.

site2 flow template.JPG

If you need to see the config of site 1, it was included a few replies ago, but here it is again.

Code: Select all

#sh flow exporter NAGIOS
Load for five secs: 26%/0%; one minute: 26%; five minutes: 28%
Time source is NTP, 10:04:15.010 PST Wed Dec 14 2016
Flow Exporter NAGIOS:
  Description:              exporter for  NAGIOS COLLECTOR
  Export protocol:          NetFlow Version 9
  Transport Configuration:
    Destination IP address: 10.65.32.9
    VRF label:              APP-001
    Source IP address:      10.65.16.7
    Source Interface:       Loopback0
    Transport Protocol:     UDP
    Destination Port:       9996
    Source Port:            62425
    DSCP:                   0x0
    TTL:                    255
    Output Features:        Not Used
      
      CONFIGURATION: 
      
flow exporter NAGIOS
description exporter for  NAGIOS COLLECTOR
destination 10.65.32.9 vrf APP-001
source Loopback0
transport udp 9996
template data timeout 60

[bwallace]

So something of interest in the captures. You have two devices sending to NNA as one source and while this is supported, the Sysuptime is of course different between them and I wonder if this confuses nfdump during nfcapd processing time. I recommend as a test you reconfigure NNA so that 16.8 is an independent source using it's own unique port and 16.7 as a second independent source using it's own unique port. Then see if the time is accurately represented in the NNA queries / reports.

Notice this sequence in the capture (when filtered in Wireshark for cflow.template_id)

20 14:07:22.043124 10.65.16.8 52325 10.65.32.9 9996 CFLOW 116 total: 1 (v9) record Obs-Domain-ID= 1 [Data-Template:257]
SysUptime: 1276531.925917088 seconds
14.77 days


135 14:07:35.828944 10.65.16.7 62425 10.65.32.9 9996 CFLOW 116 total: 1 (v9) record Obs-Domain-ID= 1 [Data-Template:259]
SysUptime: 2298291.484244640 seconds
26 days

304 14:08:01.231619 10.65.16.7 62425 10.65.32.9 9996 CFLOW 116 total: 1 (v9) record Obs-Domain-ID= 1 [Data-Template:258]
SysUptime: 2298317.509644640 seconds
26 days


341 14:08:05.525054 10.65.16.8 52325 10.65.32.9 9996 CFLOW 116 total: 1 (v9) record Obs-Domain-ID= 1 [Data-Template:258]
SysUptime: 1276574.969397088 seconds
14 days

cflow_template_id filter.jpg