NagiosXI API "object/servicestatus" failing with "500s"

[[email protected]]

NagiosXI API has stopped returning data for the "object/servicestatus" api endpoint.
This is the only API endpoint that is exhibiting the failure to return data on this NagiosXI instance.

I have NagiosXI instance with almost 3x the services (50k+) that is still returning data from the "servicestatus" endpoint. While it does take some time to mine the data on my larger instance, nothing fails.

[03/Nov/2021:08:53:43 -0400] 10.15.28.195 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 "GET /nagiosxi/ajaxhelper.php?cmd=getxicoreajax&opts=%7B%22func%22%3A%22get_servicestatus_table%22%2C%22args%22%3A%7B%22host%22%3A%22%22%2C%22hostgroup%22%3A%22tomcat-servers%22%2C%22servicegroup%22%3A%22%22%2C%22sortby%22%3A%22%22%2C%22sortorder%22%3A%22asc%22%2C%22records%22%3A%221000%22%2C%22page%22%3A%221%22%2C%22search%22%3A%22%22%2C%22hostattr%22%3A%220%22%2C%22serviceattr%22%3A%220%22%2C%22hoststatustypes%22%3A%220%22%2C%22servicestatustypes%22%3A%220%22%7D%7D&nsp=13e3bf972433add71c246eeb77c4492d6b44488eb20da579fc885cf3bdf731b4 HTTP/1.1" 585240
[03/Nov/2021:08:53:53 -0400] 172.30.86.185 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET /nagiosxi/api/v1/objects/servicestatus?apikey=<redacted> HTTP/1.1" -
[03/Nov/2021:08:54:05 -0400] 172.30.86.185 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 "GET /nagiosxi/ajaxhelper.php?cmd=keepalive&nsp=ade624dd56e8a277a09fc3bc28cdf7ea95e1c7a5646a96e7e3485793f0bd63ad HTTP/1.1" -
[03/Nov/2021:08:54:05 -0400] 172.30.86.185 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 "GET /nagiosxi/ajaxhelper.php?cmd=getxicoreajax&opts=%7B%22func%22%3A%22get_tray_alert_html%22%2C%22args%22%3A%22%22%7D&nsp=ade624dd56e8a277a09fc3bc28cdf7ea95e1c7a5646a96e7e3485793f0bd63ad HTTP/1.1" 1136
[03/Nov/2021:08:54:05 -0400] 172.30.86.185 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 "GET /nagiosxi/ajaxhelper.php?cmd=getxicoreajax&opts=%7B%22func%22%3A%22get_pagetop_alert_content_html%22%2C%22args%22%3A%22%22%7D&nsp=ade624dd56e8a277a09fc3bc28cdf7ea95e1c7a5646a96e7e3485793f0bd63ad HTTP/1.1" 1256
[03/Nov/2021:08:54:13 -0400] 172.30.86.185 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET /nagiosxi/api/v1/objects/hoststatus?apikey=<redacted> HTTP/1.1" 2185618
[03/Nov/2021:08:54:21 -0400] 10.15.30.57 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 "GET /nagiosxi/ajaxhelper.php?cmd=getxicoreajax&opts=%7B%22func%22%3A%22get_pagetop_alert_content_html%22%2C%22args%22%3A%22%22%7D&nsp=f4bb5f48cf56f75fbdb99287fa3e1fa776c8060c7eee3b0c6d6038dce52c6d7e HTTP/1.1" -

What gives?
How do we resolve the issue?

[gsmith]

Hi,

To get started please send up a System Profile.

1. Login to the Nagios XI GUI using a web browser.
2. Click the "Admin" > "System Profile" Menu
3. Click the "Download Profile" button
4. Save the profile.zip file and share the file in a private message and then
reply to this post to bring it up in the queue.

Thank you!

[[email protected]]

@gsmith
I have resolved the primary issue for " get object/servicestatus" although the root cause of the endpoint failures is still unknown.

Backup XI

Code: Select all

/usr/local/nagiosxi/scripts/backup_xi.sh

Validate Current Config files

Code: Select all

/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

Create Config Snapshot and Restart

Code: Select all

/usr/local/nagiosxi/scripts/reconfigure_nagios.sh

[gsmith]

Hi,

Sounds good.

Here is another way to clean up the config files.

The configuration is stored in the database, then written to the config files, and Nagios
imports the config files. If you think your config files are out of sync with your database
do this to get the latest from the database:
1. Configure, Core Config Manager
2. Under Tools in the left-hand menu select "Config File Management"
3. Hit the red "Delete Files" button
4. Hit the blue "Write Configs" button
5. Hit the white "Verify Files" button
6. Under Quick Tools in the left-hand menu hit the "Apply Configuration" link

Please let me know if you have any more questions or if I can lock
this thread.

Thanks!