[API] Delete method is very sloooow...

[MrWoodward]

We're currently using the JSON REST API to delete services out of hosts and are finding it very slow.

Well, submitting the service to delete is fast. That returns immediately. But if I queue up 15 services to delete and then use the apply config command, it takes 2 or more minutes to delete. An average of 12 seconds per service.

Is there any way to make deletion via the API faster? Deleting via the GUI isn't this slow.

Perhaps I'm doing something wrong? For example, when I submit things to delete, I call the "delete" method once for each service. Can I call the delete method with a list of services to delete? Would this make the deletion process faster when actually calling "apply"?

Thanks.

[scottwilkerson]

How long does an apply config take on your system through the GUI if you were to delete these same services?

The calls should be equivalent, the apply config is usually what takes a long time.

[MrWoodward]

I just timed the deletion of 5 Services and calling the "ApplyConfig" method via the API and it took about 1m 10s.

Compared to the GUI which took about 25 seconds to delete the same 5 Services and apply.

[MrWoodward]

Tried the same test with 11 services and it took the API 1m 32s to complete, while thru the GUI took only 30 seconds.

There's something wrong with the API where each additional service that gets deleted takes significantly longer, whereas deleting another service via the GUI only takes an extra 1 second.

Btw, we're monitoring /usr/local/nagiosxi/nom/checkpoints/nagioscore and /usr/local/nagiosxi/nom/checkpoints/nagioscore/errors to determine success and failure.

[scottwilkerson]

How are you timing the apply through the API? When I run mine it executes in 0.046s

Code: Select all

time curl -XPOST "http://192.168.5.60/nagiosxi/api/v1/system/applyconfig?apikey=API_KEY"
{"success":"Apply config command has been sent to the backend."}

real    0m0.046s
user    0m0.002s
sys     0m0.001s"

Maybe you could give an example of the commands?

Also, are you comparing the current timestamp to the timestamp on the the checkpoint?

I just did this on my test system

Code: Select all

[root@xidemo ~]# date +%s
1520447816
[root@xidemo ~]# curl -XPOST "http://192.168.5.60/nagiosxi/api/v1/system/applyconfig?apikey=API_KEY"
{"success":"Apply config command has been sent to the backend."}
[root@xidemo ~]# ls -trl /usr/local/nagiosxi/nom/checkpoints/nagioscore/*.txt|tail -1
-rw-r--r-- 1 nagios nagios 4178 Mar  7 12:37 /usr/local/nagiosxi/nom/checkpoints/nagioscore/1520447828.txt
[root@xidemo ~]#

1520447828 - 1520447816 = 12 seconds

[MrWoodward]

Ok, there's clearly something wrong with my "folder monitoring changes" code. I'm not sure what it is yet, but I can see that the problem is there.

Per your demonstration of using `watch` to monitor the folders, I can see that the API generally runs quickly. You can close this.

[scottwilkerson]

well, glad to hear the API is quick. Locking

[scottwilkerson]

Unlocking per OP request

[MrWoodward]

After researching the problem, I was able to come up with a partial solution to the problem.

Here's the background: We had written a script to mass delete services on a host. Some hosts have more than 50 services being monitored. And sometimes we need to decom hosts with these 50 or so services.

We used the standard DELETE "config/service" method with the "&apply_config=0" parameter so that we could delete as many services as possible in one shot.

Then after all DELETEs were called, the script immediately calls POST "system/applyconfig" command.

We found that sometimes (but not always), the REST API would accept and acknowledge receipt of this command, but then not actually execute the "apply config". And the script would wait and wait and eventually time out.

(Strangely, the "system/applyconfig" command always seems to work when ADDing services. No problem there.)

The solution was to use the "&apply_config=1" parameter on the very last service to be deleted. The REST API reliably executes the "apply config" when using this method.

The delete method is still slow-ish for us. It seems to average about 5 seconds per service deleted, which is a bit faster than before... And sometimes, it's right quick. Not sure why.

I'll keep looking into this and post back if I find anything new.

[scottwilkerson]

Thanks for sharing what you have found!