Could not parse XML

[hbouma]

We have an active/standby configuration for our Nagios servers. The Standby has been reporting errors parsing the XML when monitoring the following on the Active server:
Load
I/O
Nagios XI Jobs
Nagios XI Daemons

A few minutes later, they all go back to normal.

Servers are running Nagios XI 5.5.4 on RHEL 7 64bit. We are monitoring 14,286 services and 854 servers. The issue appears to have started when we hit around 10,000 services/hosts monitored. We have the DB offloaded, rrdcached and RAMdisk enabled on the servers.

Any support or suggestions would be appreciated.

[cdienger]

The checks use a php script called check_nagiosxiserver.php and on larger systems there can be issues parsing or applying configuration unless the php.ini is tweaked. https://support.nagios.com/kb/article/n ... e-611.html covers the modifications. I usually go a higher than the recommendation and do 100000 for max_input_vars.

[hbouma]

Tweaks were already made:

max_input_time = 180
max_input_vars = 100000
memory_limit = 2048M
max_execution_time = 80

[cdienger]

What does the load on the active machine look like ? Is the problem occurring during higher than usual loads?

Try running the check with the --debug=1 option to get more info the next time it fails.

[hbouma]

Unfortunately, I can not get it to manually fail when the checks happen. It is pretty inconsistent, but frequent.

To give you an example, it has happened 8 times since Friday night, failed 2 checks each 4 minutes apart, then immediately gone to normal on the 3rd check.

[cdienger]

The debug data will be part of the $LONGSERVICEOUTPUT$ macro and you can use this in notifications to make sure you get the output - https://assets.nagios.com/downloads/nag ... iables.pdf

[hbouma]

will do.

[cdienger]

Thanks. We'll be here

[hbouma]

Thank you for your help. Changing from XML to JSON checks has helped and we have scheduled the update to 5.5.7 to fix the issue with the JSON checks.

[cdienger]

No problem. For anyone else that may come across this, configuring the checks to use the API key will also force it to use JSON where this is fixed:

--address=$HOSTADDRESS$ --url='http://xi_ip/nagiosxi/' --apikey='api_key' --mode=daemons
--address=$HOSTADDRESS$ --url='http://xi_ip/nagiosxi/' --apikey='api_key' --mode=load --warn="5,4,4" --crit="10,10,7"
--address=$HOSTADDRESS$ --url='http://xi_ip/nagiosxi/' --apikey='api_key' --mode=jobs
--address=$HOSTADDRESS$ --url='http://xi_ip/nagiosxi/' --apikey='api_key' --mode=iowait --warn="5" --crit="15"

(replacing xi_ip and api_key with the appropriate values for your environment)

5.5.7 will also fix a similar PHP warning message caused when a check is flapping:

PHP Warning: simplexml_load_string(): v7301;CPU Usage;STOPPED; Service appears to have stopped flapping (4.0% change < in /usr/local/nagiosxi/html/api/includes/utils-api.inc.php on line 55