API Calls - Add Host calls not showing up

[scomdco]

Any insight into this is appreciated, if you need my system profile just let me know.

We're having a consistent issue with a group of AWS instances (70) that get launched at the same time.

Because their launched at the same time, the step in the bootstrap (User Data) they execute that sends a request to XI to add the instance as a host gets sent roughly within the same minute, which i'm wondering if that may be an issue where Nagios XI is getting overloaded with API calls, and because of this it's not acknowledging/queuing some of them.

ApplyConfig is NOT performed in each of the API calls.A single api call to apply config is sent after all the instances have completed their provisioning steps. There is on average a 8 minute gap of time between when all the instances send the Add Host API call, and the apply config call is sent.

For instance, i just launched another ASG (Auto Scaling Group) of 70 instances, and only 46 of the instances showed up in Nagios XI. Looking at the results of the bootstrap for the instances that did not show up after the config was applied, shows that the call was sent successfully, the results looks like this in each of the instances (Granted the host name is different of course)

Code: Select all

StatusCode        : 200
StatusDescription : OK
Content           : {
                        "success": "Added SWSIM-015E94 to the system. Config imported but not yet applied."
                    }
                    
RawContent        : HTTP/1.1 200 OK
                    Access-Control-Allow-Orgin: *
                    Access-Control-Allow-Methods: *
                    Keep-Alive: timeout=5, max=100
                    Connection: Keep-Alive
                    Content-Length: 92
                    Content-Type: application/json
                    Date: Fri, ...
Forms             : {}
Headers           : {[Access-Control-Allow-Orgin, *], [Access-Control-Allow-Methods, *], [Keep-Alive, timeout=5, 
                    max=100], [Connection, Keep-Alive]...}
Images            : {}
InputFields       : {}
Links             : {}
ParsedHtml        : System.__ComObject
RawContentLength  : 92

i can't see these events logged in API due to the known issue that was stated may be fixed in the upcoming 5.7 release, would y'all be able to confirm if that's still part of 5.7?
https://support.nagios.com/forum/viewto ... 16&t=57112

Please advise,
Nagios XI Version: 5.6.9
AWS instance type: c5.4xlarge

[ssax]

I'm wondering if you may be hitting a PHP limit or something (request size, etc), try running this command as root (and leave it running):

Code: Select all

tail -Fn0 /var/log/httpd/error_log /var/log/httpd/ssl_error_log

Then run through the test again, do you see any errors? (please PM the info)

That change is scheduled to be in XI 5.7.

[scomdco]

That looks to have captured some related details. I've PM'd you the output and also our system profile in case that's needed.

And thank you for the confirmation of the that as well..

[ssax]

You have crashed DB tables, please repair your DB following this KB article:

https://support.nagios.com/kb/article.php?id=24

Once that's done, please send the output of these commands (as root):
- NOTE: You may need to adjust the -h 127.0.0.1, the -uroot, and -pnagiosxi in the first command if your DB is offloaded to another server and/or you've changed the root mysql password

Code: Select all

echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -h 127.0.0.1 -uroot -pnagiosxi --table

Then test again and see if that resolves it, I'd restart mariadb to free up some connections, I'm wondering if that's what you were hitting.

[scomdco]

I've PM'd you:
The results after performing the repair
The results after repairing again following the second test round, and restarting mariadb

i observed the same errors during the second round, i'm taking that means the DB tables crashed again. Is there a configuration modification that could be done so that the DB can handle this amount of requests coming to it? We're planning to have a greater number then 70 get sent to XI nightly, so finding a resolution to this is high on our list of priorities. If this issue is something that should be escalated as a support ticket we can do that as well, just let me know.

[ssax]

Please create a ticket for this and include a link back to this forum thread:

https://support.nagios.com/tickets/

Please run these commands and attach the resulting /tmp/SYSSTAT.zip file to the ticket:

Code: Select all

mysqldump -h 127.0.0.1 -uroot -pnagiosxi --add-drop-database nagiosxi xi_sysstat > /tmp/xi_sysstat.sql
zip /tmp/SYSSTAT.zip /tmp/xi_sysstat.sql

[scomdco]

Thanks ssax, i've created the ticket and i attached the request file to the ticket as well., thanks for your time with this.

Ticket #674722

[ssax]

Locking thread, ticket received, we will continue support through the ticket.