Interpreting NNA Data

[ahoward12]

Fortigate 800C sending Netflow to NNA on port 9045. It is the only device sending data on that port to that source in NNA. I have two interfaces (WAN1 and port3) sending data. Below is my configuration of my Fortigate 800C; I just added the the timeout lines to my configuration. Up to this point I didn't believe they were relevant to my situation.

Have we gotten anywhere on the issue of seeing of the "Total Bytes" field we spoke of earlier, being almost exactly double on some flows?

I also spoke to Fortinet support and they confirmed my setup is correct, after applying the addition of flow timeout.

Code: Select all

Cafaro_Master # config system netflow

Cafaro_Master (netflow) # show
config system netflow
    set collector-ip 10.10.99.36
    set collector-port 9045
    set source-ip 10.10.11.1
    ***set active-flow-timeout 1
    ***set inactive-flow-timeout 15
end

Cafaro_Master (netflow) # end

Cafaro_Master # config system interface

Cafaro_Master (interface) # edit wan1

Cafaro_Master (wan1) # show
config system interface
    edit "wan1"
        set vdom "root"
        set ip *public IP information*
        set allowaccess ping https http
        set vlanforward enable
        set type physical
        set netflow-sampler both
        set alias "AT&T Fiber"
        set snmp-index 5
    next
end

Cafaro_Master (wan1) # end

Cafaro_Master # config system interface

Cafaro_Master (interface) # edit port3

Cafaro_Master (port3) # show
config system interface
    edit "port3"
        set vdom "root"
        set ip 10.10.17.1 255.255.255.0
        set allowaccess ping
        set vlanforward enable
        set type physical
        set netflow-sampler both
        set alias "Millcreek EPL"
        set snmp-index 8
    next
end

[ahoward12]

After doing some more digging into the situation. I ended up removing the source altogether. Creating a new source (same port) for NetFlow. I also created a new source for sFlow (different port), again a very simple configuration. I attached two pictures, of nearly the exact same time frame (3 hours):

Netflow data says 22130.40 GB ; again the Total Bytes is double. Totally outrageous number...

sFlow data says 26.93 GB ; the Total Bytes is accurately reflected as well (26.93 GB)

You can see the screenshots and the time frames are identical. Fortinet has next to zero support for NetFlow, so I will put off blaming the device until the end. I think this is a NNA issue; the only reason I say this is because it is reporting outrageous numbers, it also is still having issues with the "bytes" and "Total Bytes" having different values.

Here is my interface configuration. NetFlow and sFlow are reflected

Code: Select all

Cafaro_Master # config system sflow

Cafaro_Master (sflow) # show
config system sflow
    set collector-ip 10.10.99.36
    set collector-port 9050
end

Cafaro_Master (sflow) # end

Cafaro_Master # config system interface

Cafaro_Master (interface) # edit wan1

Cafaro_Master (wan1) # show
config system interface
    edit "wan1"
        set vdom "root"
        set ip Public IP
        set allowaccess ping https http
        set vlanforward enable
        set type physical
        set netflow-sampler both
        set sflow-sampler enable
        set sample-rate 10
        set polling-interval 1
        set alias "AT&T Fiber"
        set snmp-index 5
    next
end

Cafaro_Master (wan1) # edit port3
Unknown action 0

Cafaro_Master (wan1) # end

Cafaro_Master # config system interface

Cafaro_Master (interface) # edit port3

Cafaro_Master (port3) # show
config system interface
    edit "port3"
        set vdom "root"
        set ip 10.10.17.1 255.255.255.0
        set allowaccess ping
        set vlanforward enable
        set type physical
        set netflow-sampler both
        set sflow-sampler enable
        set sample-rate 10
        set polling-interval 1
        set alias "Millcreek EPL"
        set snmp-index 8
    next
end

[tgriep]

There are some differences between the netflow and sflow configurations on the Fortigate
The netflow configuration has the following lines in it's settings and I think they should be removed.

Code: Select all

set source-ip 10.10.11.1
***set active-flow-timeout 1
***set inactive-flow-timeout 15

Try removing them and see if that fixes the issue.

[ahoward12]

There are some differences between the netflow and sflow configurations on the Fortigate
The netflow configuration has the following lines in it's settings and I think they should be removed.

Code: Select all

set source-ip 10.10.11.1
***set active-flow-timeout 1
***set inactive-flow-timeout 15

Try removing them and see if that fixes the issue.

Hey as I said, I didn't have those lines in there up until I created that last post on the 22nd. It has been collecting Netflow from the device for 2 weeks without those lines. They make no difference, Fortinet advised me to have them in there, I will remove them again but that is not the problem

[tgriep]

Can you post 10 of the nfcapd.xxxxxxxxx files for the Fortigate Netflow source so we can decode them and see why they could be capturing incorrect data?
You can find the files in the following folder on the NNA server under the name of the source.

Code: Select all

/usr/local/nagiosna/var/<source>/flows

Also, can you run this tcpdump command on the NNA server for about 15 minutes and post the /tmp/capture.cap file?

Code: Select all

tcpdump -i eth0 -s 65535 -w /tmp/capture.cap port 9045

You may have to replace the eth0 option with the actual Ethernet interface name on your system.
Thanks

[ahoward12]

I grabbed the last 10 Flows and did a 15-20 minute packet capture. I attached it in a 7zip file.

[tgriep]

What version of software is is installed on the Fortigate system?
I decoded the nfcapd files and it looks like it is either capturing the data incorrectly or decoding the data incorrectly as the in bound bytes are exactly the same as the outbound bytes.
That would make sense on the doubled data but it still is not correct. Also, I tried to decode the tcpdump data and even that data doesn't look good.

[ahoward12]

My Fortinet is on 5.4.3 Build 1111. I don't like to be on the latest firmware, there is one newer 5.4.4, but there is nothing in the release notes that looks to touch any of our issues.

[tgriep]

The template that the Fortigate device sending looks like it is missing some fields and the NNA server cannot decode it correctly as it uses the templates for decoding.
Even though there wasn't anything in the release notes about any changes to the netflow settings you can try and upgrade the system or stick with sflow which is working for you.

[ahoward12]

Thank for all of the feedback on narrowing down the issue. I will rely on sFlow data for now but it comparatively unreliable compared to NetFlow. I have open a support/bug ticket for the issue to be looked into by Fortinet. I'll continue to update the thread as I get information from them.