Network Analyzer Slow

[CFT6Server]

I think that would suffice. I would suggestion making this an option that is configuration in the settings so users can specify the default time period for source query interval.

[jdalrymple]

CFT6Server - I'd really like to isolate this as a shortcoming with either our software or your IO. Me being the SAN purist I am of the opinion that you probably get as good or better IO on your 10GBe NetApp than you would with internal disks. Since it seems that OnCommand reporting isn't available the only other way I'd know to test would be to get the disks inside the hardware.

Is there an option to use locally hosted VMFS just temporarily for your NNA box, or are you guys boot from SAN? Even if the performance of the local disks is worse, it would perhaps give us a quantifiable difference to work with.

Alternatively - if I could come up with a way for you to scp us a gzipped copy of your NNA VM - I could deploy it on an SSD based server here just to see the difference. I'm not sure if that's something your security policy allows though, and don't want to overstep any such boundaries. The problem is I simply have no way of generating enough network traffic to reproduce your environment in-house. I would really like to see if granting your machine the supposedly boundless IO of SSDs would resolve your performance issue.

Also open to any other possibilities you might have at your disposal to see if we can change the performance by means of modifying the disk properties.

[CFT6Server]

I will engage our storage team to see what metrics and report I can provide to you. We will also review and see if there are faster options we can use for this VM within our infrastructure.

I would love to send you a sample of our NNA, however I don't think that would be possible due to security and regulatory reasons. If there is any chance that you want to do some sort of remote session to view the instance or to see this first hand, I think that is possible.

I agree that we should isolate this a bit better. I want to see this work and would like to provide any information for you guys.

[jdalrymple]

It looks like I can generate the needed flow data in house. Myself and jolson are trying to work through possible optimization. One thing is becoming more and more and more apparent - it's not a disk IO related bottleneck.

Stay tuned - and in the meantime please let us know if you discover anything useful.

[CFT6Server]

Thanks! Our current NNA has two sources one is 4.5GB and other is 12GB each holding 2 days of data.
Also noticed that running query sometimes will just stall... so this query is looking for web traffic report for last 2 hours, but the chord diagram is done and the information just never shows. I have left this running for a while and nothing... the nmon on the right shows that the system is no longer doing anything, but the query never finishes.

Thanks again and looking forward to your findings.

[jolson]

jdalrymple and myself are working with Nagios Network Analyzer on our end to see if we can properly reproduce your results. We have collected a large amount of logs, and we see that nfdump is the bottleneck - whether this is related to the processor or the disk isn't definite yet - but we tentatively believe that the processor is the culprit.

We will keep you updated as we progress in our examination.

[CFT6Server]

I think nfdump has threads and discussions about making it muli-threaded. This is the limitation of nfdump at the moment. Unfortunately there have not been any news/development from makers of nfdump.

[jdalrymple]

Myself and jolson are still working at pinpointing or perhaps eliminating your source of pain. While we do that I'd like to make a special request of you - don't reply to this thread unless you come up with some new information. If the last reply to a forum thread isn't from a staff member it sits and haunts us in the same way the scoreboard at McDonalds haunts those workers. Just like McDonalds we don't like our orders to sit there very long so we end up crafting a useless reply like the one I'm typing now just to get it off the scoreboard

If you discover anything new - please don't hesitate to reach out.

If you don't hear from us before say Wednesday morning reach out and we'll give you a full progress update.

Otherwise let the thread sleep please.

Thanks and I promise we'll keep you posted!

[CFT6Server]

I know you guys are working hard on this. Any progress or updates? Thanks.

[jdalrymple]

I know you guys are working hard on this

I wouldn't necessarily say we're working hard, we're just fitting it in where we can. jolson and I are both convinced that this is not a problem you can just throw hardware at to solve and we're eager to find a solution that scales better so that we can be ahead of other future potential customers finding out our software can't perform in high demand environments.

I would say we're about 50-75% through making a functional *theoretical* test bed. All that sounds very vague because it is. While we're very confident that our efforts to store the data more efficiently will result in better query times, we cannot guarantee that we're right and furthermore there is no way to predict how or when this could ever make it into NNA.

All that said - we are indeed still looking at it as time permits. We do not want you to just be the first of many customers who find NNA is inadequate to handle their environment, we want to stay ahead of demand. Since we are support though it is truly an "as time permits" thing until (and if) we can convince the developers that there is a better solution than what we're doing today.