Extremely slow acknowledgements, downtimes etc.

[Nagiusz]

Hi all,

we have a problem with our Nagios XI instance (running recent version 2024R1.3, but the problem seems to be much older).

Whenever we click acknowledgment, comment or scheduled downtime buttons, the operation takes several seconds (15-20 or even more). Seems that the item clicked appears in Nagios after such time, but it is really frustrating. I have sniffed the HTTP query and typed it in via curl in Nagios shell - the behavior is really the same.

I believe that GUI invokes /usr/local/nagios/sbin/cmd.cgi command - tried it out and as well, the response comes after several seconds.

What is more, seems that there is no such problem when making acknowledgement, comment etc. via Nagios External Commands mechanism, so the problem seems to be CGI related somehow. Our instance runs on Ubuntu 20.04 LTS with Apache/2.4.41 (PHP 7.4.3).

Any help appreciated

[danderson]

Thanks for reaching out @Nagiusz,

In your main nagios configuration file, often at /usr/local/nagios/etc/nagios.cfg, do you know what the status_update_interval is set to? The core cgi's are updated only on that interval.

Is this happening when you submit a command via the cgi or when you are interacting through the XI interface?

When you submit a command, how soon does it appear in /usr/local/nagios/var/nagios.log?

[Nagiusz]

Thanks for the reply!

Our status_update_interval is set to 10 (I have temporarily changed it to 1 but it does not help ).

Seems that it happens when sent from Nagios UI.

Regular checks are running smoothly.

[Nagiusz]

Regarding nagios.log file, entry about acknowledgement shows up after 15-20 seconds after clinking in GUI. Any idea?

[Nagiusz]

According to my deep dive into HTTP requests, it seems that all is about coreuiproxy.inc.php script, which is executed. There is sg_load function inside, seems that part of code is protected by Source Guardian and therefore is executed extremely long. How to fix it?

[danderson]

That's pretty strange. If you manually navigate to nagios core, going to https://{Your host name}/nagios, and you acknowledge it through the nagios core interface, does it still take a long time to show up in the nagios log file? I'm asking because I'm trying to determine if it's a nagios core issue or a nagios xi issue.

[Nagiusz]

Thanks for the quick response

Using Nagios core it seems that it (acknowledgement) takes as much time as Nagios XI.

BTW, we do not use Nagios (non-XI) interface at all, just entered it to check for Your question, but as I see it runs definitely much slower than Nagios XI one.

[Nagiusz]

I do not know if it matters here, but running /usr/local/nagios/sbin/cmd.cgi from the console takes also c.a. 15 seconds before giving any output. On the other hand, when I put the acknowledgement with external command, so write to /var/nagios/rw/nagios.cmd directly, it works without any delay.

[danderson]

Well I have an idea why that might take so long. How many hosts and services do you have configured roughly? It seems that every time a request is made then it reads all of the configuration data. So that might be taking a while before actually submitting the request.

If that's the case then I'll have to see what I can do to fix that. Probably easy enough in XI, where I would change it to use the nagios.cmd file instead of the CGI, but it would require some code changes.

[Nagiusz]

Thanks for the response. Indeed, we have 16k hosts and 65k services. Meanwhile I have made some PHP script which acts before CGI part, parses POST request data and puts it directly as the external command. It works quite fast, but still only workaround. However, good to know that I am not the only one thinking in this way Maybe NagiosXI should refactor the way where CGI scripts are used.