nagios restart during backup?

[elinagios]

Hello

We have had 2 occasions where during backup nagios monitoring engine will die. Both occasions there sadly has been errors in the configs.
1) Does Nagios restart services during backup? Why is this neccesary? Can we backup without service restart?
2) Is it possible to monitor the state of nagios config file with some service check?

Regards,
Mirgo

[npolovenko]

Hello, @elinagios. I'm not sure how you back up Nagios, but if you're using a scheduled backups in the GUI, or if you're using the backup_xi script then yes, it restarts Nagios to get the latest retention.dat file. It helps in large systems because it tells nagios what statuses did all host and service checks have during the last hour or so before the backup. So that way the new Nagios will not have to start checking thousands of hosts and services right away after you deploy it.
You could probably comment that part out from the backup script. But It would be better to understand what is making Nagios process to not start after the backup. What kind of errors are you seeing? Can you rerun the backup script and then send us your profile?
To send us your system profile. Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and attach it with your next post.

[elinagios]

I know what the errors are when i check the config from config manager, two times another admin has left error in the service/host config. Easy to fix, but as backup run off hours then need to deal with them off hours aswell.

Coming back to the second question, is it possible to check the nagios config via some service check so i know if someone messes up configuration before the backup restart takes the monitoring engine offline.

And yes, i use the the scheduled backups set up from the GUI.

[npolovenko]

@elinagios, Yes, there is a command to check all the configs:

Code: Select all

/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg 

You could create a plugin if you wanted to:

Code: Select all

#!/bin/bash

/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg | grep 'Total Errors'

exit 0

[elinagios]

This doesnt actually work, the command is ok, it will give you an correct output but:

1) When you make an error in config files and use apply configuration, Nagiosxi is smart and will not let you use the bad config, it will roll back to the working config, /usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg will say no errors found, thou you have a bad config ready to be writen.
2) Now Nagiosxi backup script seem to work differently, more like core config manager, config file managment, it will write the config, not verify and restart the service. When you have a bad config and use write config from config file management, then /usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg command is useful, it will say you have an error. But you still will not discover this until your service is down, bad config written by backup.

Need another way how to discover if there is an error in the config ready to be written.

[npolovenko]

@elinagios, There's no way to verify the config that hasn't been written out yet. Before the config is written is stays in MYSQL.

Nagiosxi is smart and will not let you use the bad config

Not always the case, sometimes it will write out a bad config. And sometimes, if the previously saved config is corrupted Nagios gets into a loop. It cant write out the current config because it has errors, then it goes back to the previous config which also has errors. I think that's probably what happened in your case. I'd consider deleting all the old snapshots and making a new one.

[elinagios]

Ok, thank you. I got my answer, you can close the thread.

[npolovenko]

@elinagios, You're welcome. Locking the thread.