Source file is too large to run query, timing out

[snapon_admin]

We have a couple of sourced, our 2 main data center routers that have data files that are quite large (18G+) and because of this it's basically impossible to run a query on either of them since it takes so long it eventually just times out. Is there any way to alleviate this?

[tgriep]

Running reports on Network Analyzer is a single threaded process and it will use 100 percent of a CPU / Core when a report is run so make sure you have allocated enough CPU's to the system.
Another thing to try is if the system is running on a VMWare server, move it to the fastest hard drive on the VMWare server, that will help out as well when the files are read when the report is created.

You can also edit the PHP memory_limit setting and time out settings on the server so it will use more memory when a report is run that should help out on the performance and also keep it from timing out.

To do this, login to the server as root and edit the /etc/php.ini file.
Change the default setting from

Code: Select all

memory_limit = 128M

to

Code: Select all

memory_limit = 1024M

This sets it to 1 Gig of ram, if the server has more memory available, you can increase it further.

You should also increase these values from

Code: Select all

max_execution_time = 30
max_input_time = 60

to

Code: Select all

max_execution_time = 240
max_input_time = 600

Changing those will keep the PHP scripts from timing out.

Save the file and restart the Apache daemon by running
service httpd restart

Try that and see if the reports run faster for you.

[snapon_admin]

That seemed to help quite a bit, thanks. I increased it to 2G since we have 8 on this server. May still increase it further if this runs ok. Thanks for the tip!

[tgriep]

No problem, let us know the outcome.

[snapon_admin]

I increased memory_limit to 4 gig and that seems to have helped immensely, however trying to load anything via source groups is still not happening. I mentioned before that I was trying this on our 2 busiest routers, well it works pretty well when I run them individually now, but when I try to pull up the source group (just those 2 routers in the group) it takes several minutes (~10) just to get past the initial summary screen. As for the query I just tried running one aggregating by srcip and dstip for the last 4 hours and it's been thinking for about 5 minutes now and still going. So for specs, you already know I have 8Gs of RAM and it appears that this server has 4 CPU cores if i'm looking it up properly (It's a VM but I don't have access to the VM itself, just SSH access), so is this just a hardware limitation or is there anything else you can think of that I can change to make this run a little faster? Specifically, loading the summary page on any source takes awhile, but on these 2 it's painfully long, and then after loading that page I have to wait a long time to run queries on source groups. Queries on individual sources is ok now, though loading the summary page on individual sources is still fairly slow.

[tgriep]

Faster disk speed should help out in this case, the application that reads the data, has to read in all of the files and processes them and faster drives would help that.
Also, if you can increase the CPU speed allocated to the system, that will work as well. Running queries is single threaded so a faster CPU will help the performance.

[snapon_admin]

I'll have to check and see if either of those are an option. In the mean time is there a way to kill a query that's in progress? I just tried running one and it's just making the entire server hang right now and I can't do anything. Do you know if any performance increases are coming with NNA 3.0? I hate to say it, but we've had better luck with other Netflow monitors. We were hoping to be able to use something that integrates into Nagios XI as well as NNA does, but this issue has been a pretty hard one to work around.

[eloyd]

My single biggest pet peeve of NNA is its lack of scaling. If I throw 4 cores/16GB at it, I expect it to use it. It doesn't. It hums along like a game of Tetris trying to decode RSA algorithms. Please consider threading it and dynamically scaling memory usage accordingly.

[tgriep]

@snapon_admin if you are running a query that will not finish, you will have to login to the server as root and kill it off and / or restart the apache process.
There is not a way to do that from the Web Interface.

The performance issues when running reports and queries from what I heard will be addressed in the next major release of the NNA server.

[snapon_admin]

That's good to hear, is that still on track for later this year as far as you know?