Page 1 of 1

NNA stops working

Posted: Tue Mar 29, 2016 2:06 pm
by tomdchi
I am testing NNA and have a source defined and data is flowing to the server. Everything seems to be working OK but after a few minutes the web UI stops responding and I am unable to log into the console anymore. The only way to get back in is to reboot but after a few minutes it stops responding again. Any idea on how to fix this? Are there any log files other than the ones on /var/log that would show what the problem could be?
The server OS is CentOS 7

Thanks,
Tom

Re: NNA stops working

Posted: Tue Mar 29, 2016 2:07 pm
by eloyd
You could use Nagios to monitor NNA and have it restart things when they fail.

In the meantime, what is the configuration (memory, processor, etc) of the machine you're running NNA on and how many sources do you have specified?

Re: NNA stops working

Posted: Tue Mar 29, 2016 3:07 pm
by bwallace
Thanks again eloyd - tomdchi, let us know the answer to those questions whenever you have a moment.

Re: NNA stops working

Posted: Tue Mar 29, 2016 3:55 pm
by tomdchi
I wouldn't be able to use Nagios to restart things because the server is not responding at all. Not able to log into web UI or to console over internet. But this is a VM on VMware and I am able to log into the console via VMware console.
One thing I have noticed is that the problem comes up after the source is started. If I reboot and then quickly go and stop the source the server keeps running OK. Its not until after I start the source that the problem begins. Data does flow to NNA for a little while from the source. So given all of that I am wondering if its a network issue with VMware. Are there any log files I can look at to tell?

The server is a VM on VMware and I have bumped it up to 2 cores (4GHz total) and 8GB or RAM. What are the recommended number of cores and RAM for NNA if I had 50 sources?
Currently there is only a single source defined for our testing purposes.

Re: NNA stops working

Posted: Tue Mar 29, 2016 5:02 pm
by bwallace
We do not have an official Network Analyzer sizing doc available since there are to many variables to account for. However, running reports with a date range >= one week are more likely to bog things down than the number of sources.

As an example, and this is on the very low end of the spectrum, flows from my primary PC at work amount to just 5MB/day.

You'll have to determine how much data each of your sources produce on a daily basis and then do some math to see if you'll have enough room on your drive in the long run.

Run this cmd:

Code: Select all

du -h /usr/local/nagiosna/var/<your Source name>/flows
From our doc 'Nagios Network Analyzer - General Overview'
/flows - Stores the binary flow data files for the source (raw source data). Can be read using nfdump. Uses nfexpire to automatically keep only the set amount of raw data that is defined in the Source settings
https://support.nagios.com/kb/article.p ... ategory=46

As for logs on NNA for troubleshooting:

Code: Select all

/usr/local/naigosna/var/backend.log
= Hope this helps =

Re: NNA stops working

Posted: Wed Mar 30, 2016 7:51 am
by eloyd
Just a thought, but might iptables/fail2ban be blocking you for some reason? Being "unable to log into the console anymore" is a very bad thing. Indicative of system failure of some sort.

Re: NNA stops working

Posted: Wed Mar 30, 2016 1:19 pm
by jolson
the web UI stops responding and I am unable to log into the console anymore. The only way to get back in is to reboot but after a few minutes it stops responding again
One thing I have noticed is that the problem comes up after the source is started. If I reboot and then quickly go and stop the source the server keeps running OK
To me this screams of a server being hanged due to an application. Is there something special about the source that you're starting that might cause your system to freeze - is it a core switch, for example?

What I would do is freeze up the box, reboot it, and check the following logs:

Code: Select all

cat /var/log/messages
cat /var/log/secure
sar
The sar command might not work, but I do expect /var/log/messages to contain good information as that's where our netflow collectors log to. I've seen errors in that file more than any other.

Otherwise check on the disk space and free memory:

Code: Select all

df -h
df -i
free -m
Thanks!

Re: NNA stops working

Posted: Wed Mar 30, 2016 3:37 pm
by tomdchi
OK, I'm an idiot. Sorry to waste your time on this. I had the nic set to an IP already in use...DUH.
Everything is working now.
Thanks!

Re: NNA stops working

Posted: Wed Mar 30, 2016 3:46 pm
by eloyd
Glad we could help. Sometimes it's the stupidest things that take the most out of us :-)

Re: NNA stops working

Posted: Wed Mar 30, 2016 3:50 pm
by bwallace
No worries tomdchi, happens to all of us - I'll go ahead and lock this thread now.