NXI says API not working, but NNA is returning SOME data

[eloyd]

We have a customer running Nagios XI 5.4.11 and NNA 2.2.3. Going into the NNA integration component within Nagios XI and adding the NNA host works fine. API key was regenerated multiple times to ensure that that wasn't the problem. Here's what's happening:

In Nagios XI, Reports, Network Report, one (seemingly random, but always the same) source is showing up as the Top 5 talkers. The data showing on the screen is correct for that source, but the source is messed up and doesn't have actual data in NNA so it's basically blank. No big deal. However, the pulldowns for source name, source group, or reports to run are blank. Between the selection settings at the top and the data in the bottom, it says that it can't connect to the API. I'd believe that 100% if it wasn't actually seeing a valid source (name and everything).

I deleted the configuration and recreated it with a different NNA username/API key and same thing.

Customer has sudo issues right now so I can't get to /var/log/httpd/*_log on NNA box to see what the request is actually doing, but this just seems weird. All Nagios XI components were upgraded during troubleshooting and we can't figure it out.

Thoughts?

[tgriep]

Do you have access to the /var/log/httpd/*_log file on the XI server?
If so, can you look at then when you are trying to run a report and see if you get any errors.

[eloyd]

Do you have access to the /var/log/httpd/*_log file on the XI server?
If so, can you look at then when you are trying to run a report and see if you get any errors.

As written:

Customer has sudo issues right now so I can't get to /var/log/httpd/*_log on NNA box to see what the request is actually doing...

Hoping to get that fixed today, but all contact with customer has to go through secure email server so it's obnoxiously slow.

[eloyd]

Got access to XI log, still waiting for NNA log. Here's what happens when I click on the Network Report in the XI Reports section. It looks the same as when we click on our own XI server, so I'm pretty sure the problem is on the NNA side. Will update when I get those logs.

Code: Select all

==> /var/log/httpd/error_log <==
[Fri Jan 19 10:10:55 2018] [error] [client 10.100.10.2] PHP Notice:  Undefined offset: 0 in /usr/local/nagiosxi/html/includes/components/nagiosna/nagiosna-reports.php on line 479, referer: http://10.2.20.57/nagiosxi/reports/
[Fri Jan 19 10:10:55 2018] [error] [client 10.100.10.2] PHP Notice:  Undefined offset: 0 in /usr/local/nagiosxi/html/includes/components/nagiosna/nagiosna-reports.php on line 484, referer: http://10.2.20.57/nagiosxi/reports/
[Fri Jan 19 10:10:55 2018] [error] [client 10.100.10.2] PHP Warning:  array_key_exists() expects parameter 2 to be array, null given in /usr/local/nagiosxi/html/includes/components/nagiosna/nagiosna-reports.php on line 485, referer: http://10.2.20.57/nagiosxi/reports/
[Fri Jan 19 10:10:55 2018] [error] [client 10.100.10.2] PHP Warning:  array_key_exists() expects parameter 2 to be array, null given in /usr/local/nagiosxi/html/includes/components/nagiosna/nagiosna-reports.php on line 485, referer: http://10.2.20.57/nagiosxi/reports/
[Fri Jan 19 10:10:55 2018] [error] [client 10.100.10.2] PHP Warning:  Invalid argument supplied for foreach() in /usr/local/nagiosxi/html/includes/components/nagiosna/nagiosna-reports.php on line 840, referer: http://10.2.20.57/nagiosxi/reports/

Code: Select all

==> /var/log/httpd/access_log <==
10.100.10.2 - - [19/Jan/2018:10:10:55 -0600] "GET /nagiosxi/includes/components/nagiosna/nagiosna-reports.php HTTP/1.1" 200 44968 "http://10.2.20.57/nagiosxi/reports/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
10.100.10.2 - - [19/Jan/2018:10:10:55 -0600] "GET /nagiosxi/images/pdf.png HTTP/1.1" 200 18737 "http://10.2.20.57/nagiosxi/includes/components/nagiosna/nagiosna-reports.php" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
10.100.10.2 - - [19/Jan/2018:10:10:55 -0600] "GET /nagiosxi/images/jpg.png HTTP/1.1" 200 18709 "http://10.2.20.57/nagiosxi/includes/components/nagiosna/nagiosna-reports.php" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
10.100.10.2 - - [19/Jan/2018:10:10:56 -0600] "GET /nagiosxi/ajaxhelper.php?cmd=getusermeta&opts=view_rotation_enabled&nsp=56eba68a906f0eaa96ae5d2bba693c9a84a59b688a348689a1673496a483a80a HTTP/1.1" 200 1 "http://10.2.20.57/nagiosxi/includes/components/nagiosna/nagiosna-reports.php" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"

[eloyd]

Finally got sudo working on NNA box. A hit to "Network Report" from Nagios XI shows the following in the NNA httpd access_log. I see no difference (other than IPs and keys) between this and a valid, working request from our internal Nagios XI box doing the same request to our internal Network Analyzer box.

Code: Select all

==> /var/log/httpd/access_log <==
10.2.20.57 - - [19/Jan/2018:16:08:41 -0600] "GET /nagiosna/index.php/api/sources/read?token=REDACTED HTTP/1.0" 200 2227 "-" "-"
10.2.20.57 - - [19/Jan/2018:16:08:41 -0600] "GET /nagiosna/index.php/api/reports/execute?q[rid]=1&q[sid]=11&token=REDACTED HTTP/1.0" 200 48 "-" "-"
10.2.20.57 - - [19/Jan/2018:16:08:41 -0600] "GET /nagiosna/index.php/api/reports/read?q[rid]=1&token=REDACTED HTTP/1.0" 200 189 "-" "-"
10.2.20.57 - - [19/Jan/2018:16:08:42 -0600] "GET /nagiosna/index.php/api/reports/reportviz?rid=1&token=REDACTED &sid=11 HTTP/1.0" 200 456 "-" "-"
10.100.10.2 - - [19/Jan/2018:16:08:42 -0600] "GET /nagiosna/index.php/api/reports/read?token=REDACTED HTTP/1.1" 200 771 "http://10.2.20.57/nagiosxi/includes/components/nagiosna/nagiosna-reports.php" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
10.100.10.2 - - [19/Jan/2018:16:08:42 -0600] "GET /nagiosna/index.php/api/sources/read?token=REDACTED HTTP/1.1" 200 2227 "http://10.2.20.57/nagiosxi/includes/components/nagiosna/nagiosna-reports.php" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
10.100.10.2 - - [19/Jan/2018:16:08:42 -0600] "GET /nagiosna/index.php/api/graphs/queryviz?sid=11&agg1=srcip&sortby=bytes&q[begindate]=-24%20hours&q[rawquery]=&type=report&token=REDACTED &type=report&agg2=srcport HTTP/1.1" 200 73 "http://10.2.20.57/nagiosxi/includes/components/nagiosna/nagiosna-reports.php" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
10.100.10.2 - - [19/Jan/2018:16:08:42 -0600] "GET /nagiosna/index.php/api/graphs/queryviz?sid=11&agg1=srcip&sortby=bytes&q[begindate]=-24%20hours&q[rawquery]=&type=report&token=REDACTED &type=report&agg2=dstport HTTP/1.1" 200 73 "http://10.2.20.57/nagiosxi/includes/components/nagiosna/nagiosna-reports.php" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
10.100.10.2 - - [19/Jan/2018:16:08:42 -0600] "GET /nagiosna/index.php/api/graphs/queryviz?sid=11&agg1=srcip&sortby=bytes&q[begindate]=-24%20hours&q[rawquery]=&type=report&token=REDACTED &type=report&agg2=dstip HTTP/1.1" 200 73 "http://10.2.20.57/nagiosxi/includes/components/nagiosna/nagiosna-reports.php" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
10.100.10.2 - - [19/Jan/2018:16:08:43 -0600] "GET /nagiosna/index.php/api/views/get_views?token=REDACTED &q%5Bsid%5D=11 HTTP/1.1" 200 2 "http://10.2.20.57/nagiosxi/includes/components/nagiosna/nagiosna-reports.php" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"

[tgriep]

Try updating the PHP variables on the NNA system and see if that helps. This KB article shows what to change.
https://support.nagios.com/kb/article/n ... s-611.html
Any chance you can upgrade NNA to the latest version?

[eloyd]

Upgrading NNA to 2.3.1 seems to have resolved the problem, but exposed a bug in the upgrade script:

After it tried to install sourceguardian, it restarts httpd. If sourceguardian is already installed, it restarts httpd anyway. Then, it copies an http conf file to /etc/httpd/conf.d (or wherever) and restarts httpd again.

In this case, there was a race condition where the previous restart didn't finish creating processes after the second restart was started, so when the second restart tried to start things, httpd was already running and port 80 allocated, so the restart failed (even though httpd was running). End result is that the upgrade script failed.

I temporarily changed "service $httpd restart" to "service $httpd stop; sleep 5; service $httpd start" so I could control-Z in the middle and make sure it was fully stopped. Once I did that, the upgrade continued without issue and we seem to be back in business from the Nagios<->NNA integration side. Thanks.

[npolovenko]

@eloyd, I had NNA 2.2.3 installed on my VM, I cd'ed into the /tmp/ directory and downloaded NNA 2.3.1 tar.gz, went into the installation folder and ran the ./upgrade script. Everything went smooth and I didn't get any errors. Can you advise what changes should I make to my system to be able to replicate this issue?

[eloyd]

Couldn't tell you. It's possible it's unique to our customer, it may be because they activated SSL on their web site, it may be because they added something else to the http.conf that delays the startup or shutdown. Couldn't say. If it isn't a problem on a new install, then it likely isn't a problem worth dealing with.

Besides, they pay us to fix these sorts of things, which we did. You can close this out.

[npolovenko]

Sounds good, @eloyd. I will lock this topic for you .