REST API mixing sequential service DELETEs and POSTs

[rossbeehler]

Our automated deployment is creating/updating Nagios services using the REST API. Since there is no mechanism to update/modify a service, you have to delete and re-post the service. However, if you do this too quickly, as automation will do, the deletes and posts will get mixed up, and you'll only have a subset of the services. The number of services kept is random enough that it feels like it's an asynchronous processing issue behind the scenes. I can get around it with a sleep for a couple seconds after (all) the delete(s), but this feels like a horrible workaround. Am I missing something, such as a better update mechanism, etc.?

To recreate, here's the Help curl commands simply lumped together for 4 services:

Code: Select all

curl -sS -XDELETE "https://nagios.domain.name/nagiosxi/api/v1/config/service?apiKey=<redacted>&pretty=1&host_name=testapihost&service_description=PING&applyconfig=1"
curl -sS -XPOST "https://nagios.domain.name/nagiosxi/api/v1/config/service?apiKey=<redacted>&pretty=1" -d "host_name=testapihost&service_description=PING&check_command=check_ping\!3000,80%\!5000,100%&check_interval=5&retry_interval=5&max_check_attempts=2&check_period=24x7&contacts=nagiosadmin&notification_interval=5&notification_period=24x7&applyconfig=1"
curl -sS -XDELETE "https://nagios.domain.name/nagiosxi/api/v1/config/service?apiKey=<redacted>&pretty=1&host_name=testapihost&service_description=PING2&applyconfig=1"
curl -sS -XPOST "https://nagios.domain.name/nagiosxi/api/v1/config/service?apiKey=<redacted>&pretty=1" -d "host_name=testapihost&service_description=PING2&check_command=check_ping\!3000,80%\!5000,100%&check_interval=5&retry_interval=5&max_check_attempts=2&check_period=24x7&contacts=nagiosadmin&notification_interval=5&notification_period=24x7&applyconfig=1"
curl -sS -XDELETE "https://nagios.domain.name/nagiosxi/api/v1/config/service?apiKey=<redacted>&pretty=1&host_name=testapihost&service_description=PING3&applyconfig=1"
curl -sS -XPOST "https://nagios.domain.name/nagiosxi/api/v1/config/service?apiKey=<redacted>&pretty=1" -d "host_name=testapihost&service_description=PING3&check_command=check_ping\!3000,80%\!5000,100%&check_interval=5&retry_interval=5&max_check_attempts=2&check_period=24x7&contacts=nagiosadmin&notification_interval=5&notification_period=24x7&applyconfig=1"
curl -sS -XDELETE "https://nagios.domain.name/nagiosxi/api/v1/config/service?apiKey=<redacted>&pretty=1&host_name=testapihost&service_description=PING4&applyconfig=1"
curl -sS -XPOST "https://nagios.domain.name/nagiosxi/api/v1/config/service?apiKey=<redacted>&pretty=1" -d "host_name=testapihost&service_description=PING4&check_command=check_ping\!3000,80%\!5000,100%&check_interval=5&retry_interval=5&max_check_attempts=2&check_period=24x7&contacts=nagiosadmin&notification_interval=5&notification_period=24x7&applyconfig=1"

When I run that, I get a random number of PING1-4 services to actually get applied, and the configuration snapshots do not appear to be applied in the order in which the curl commands execute.

Note that this also happens if I defer applying the config until the end of all processing.

Finally, we are running Nagios XI 5.4.11.

[npolovenko]

Hello, @rossbeehler. I see that you have applyconfig=1 at the end of each request. It is very likely that the Apply Configuration process takes a while on your server and you're sending yet more configuration commands while the previous config hasn't been fully applied. This can cause all kinds of problems. Have you thought about running the commands with applyconfig=0 and only add applyconfig=1 to the last command in your list?

[rossbeehler]

Hi npolovenko,

Sorry for the long initial post, but I did mention near the end:

Note that this also happens if I defer applying the config until the end of all processing.

I.e. this bash also recreates:

Code: Select all

curl -sS -XDELETE "https://nagios.domain.name/nagiosxi/api/v1/config/service?apiKey=<redacted>&pretty=1&host_name=testapihost&service_description=PING"
curl -sS -XPOST "https://nagios.domain.name/nagiosxi/api/v1/config/service?apiKey=<redacted>&pretty=1" -d "host_name=testapihost&service_description=PING&check_command=check_ping\!3000,80%\!5000,100%&check_interval=5&retry_interval=5&max_check_attempts=2&check_period=24x7&contacts=nagiosadmin&notification_interval=5&notification_period=24x7"
curl -sS -XDELETE "https://nagios.domain.name/nagiosxi/api/v1/config/service?apiKey=<redacted>&pretty=1&host_name=testapihost&service_description=PING2"
curl -sS -XPOST "https://nagios.domain.name/nagiosxi/api/v1/config/service?apiKey=<redacted>&pretty=1" -d "host_name=testapihost&service_description=PING2&check_command=check_ping\!3000,80%\!5000,100%&check_interval=5&retry_interval=5&max_check_attempts=2&check_period=24x7&contacts=nagiosadmin&notification_interval=5&notification_period=24x7"
curl -sS -XDELETE "https://nagios.domain.name/nagiosxi/api/v1/config/service?apiKey=<redacted>&pretty=1&host_name=testapihost&service_description=PING3"
curl -sS -XPOST "https://nagios.domain.name/nagiosxi/api/v1/config/service?apiKey=<redacted>&pretty=1" -d "host_name=testapihost&service_description=PING3&check_command=check_ping\!3000,80%\!5000,100%&check_interval=5&retry_interval=5&max_check_attempts=2&check_period=24x7&contacts=nagiosadmin&notification_interval=5&notification_period=24x7"
curl -sS -XDELETE "https://nagios.domain.name/nagiosxi/api/v1/config/service?apiKey=<redacted>&pretty=1&host_name=testapihost&service_description=PING4"
curl -sS -XPOST "https://nagios.domain.name/nagiosxi/api/v1/config/service?apiKey=<redacted>&pretty=1" -d "host_name=testapihost&service_description=PING4&check_command=check_ping\!3000,80%\!5000,100%&check_interval=5&retry_interval=5&max_check_attempts=2&check_period=24x7&contacts=nagiosadmin&notification_interval=5&notification_period=24x7"
curl -sS -XPOST "https://nagios.domain.name/nagiosxi/api/v1/system/applyconfig?apikey=<redacted>"

[npolovenko]

@rossbeehler

Sorry for the long initial post, but I did mention near the end:

I apologize

Did you know that you can update services and hosts without having to delete them? You'll use the same API command that you used to create them but with updated arguments. As long as the host name and the service description are still the same.

We think the reason you're having problems is because the API delete command works asynchronously. I communicated with the Dev team and they're planning on making this command synchronous in XI 5.6.

[rossbeehler]

Hi @npolovenko,

I think I figured out what confused me and caused me to not believe POSTing actually updates an existing service, but instead went down the delete-then-re-create route: if your service_description has a backslash character in it, it will always create a new, possibly duplicate, service. My examples in this topic didnn't have that, but my actual services I'm creating for my organization (sometimes) do, and definitely did for all my initial testing.

For example, if you take the API example and change the service_description to "Name with a \ character" like this:

Code: Select all

curl -sS -XPOST "https://nagios.domain.name/nagiosxi/api/v1/config/service?apikey=<redacted>&pretty=1" -d "host_name=testapihost&service_description=Name+with+a+%5C+character&check_command=check_ping\!3000,80%\!5000,100%&check_interval=5&retry_interval=5&max_check_attempts=2&check_period=24x7&contacts=nagiosadmin&notification_interval=5&notification_period=24x7&applyconfig=1"

and run it twice, you will see two entries appear in the CCM Services page with different IDs, see the duplication in the config snapshot, etc. Do note that you DO NOT see the duplication in the normal Service Status page, which just adds to the confusion

I'll workaround this to instead use a different character (e.g. a forward slash, which works), but this is a bug you may want to report.

[npolovenko]

@rossbeehler, I also thought it's a bug at first until I tried adding a hostname with / sign in the CCM. Turns out backslash is an illegal character and not allowed in the hostname.

Untitled.png

[rossbeehler]

Hi @npolovenko, it's still a bug, just one in the (missing) validation of the REST API

[npolovenko]

@rossbeehler, Technically, we just never introduced the API config verification . API scripts do apply the configuration, and It may impact the Core if values are incorrect. I submitted a feature request to add the API config verification to the future version of XI. This will probably be implemented in XI 5.6 or 5.7.