Config Restore not successful

[safuanmansor]

Dear Support,

We are having issue with one of the nagiosxi where it cant restore the configuration changes.
Gui output : Configure snapshot restore has been scheduled

Upon checking, the ccm data in XI webportal are not updated with old configuration. Below thread not resolving the issue

1. https://support.nagios.com/forum/viewto ... 5&start=10
2. https://support.nagios.com/forum/viewto ... =16&t=8404

We have a similar case with no 1 where the cmdsubsys.log showing

CMDLINE=/usr/local/nagiosxi/scripts/ccm_snapshot.sh 1608001754 restore

Restoring CCMl snapshot

Removing old files from /usr/local/nagios/etc

/ ~

RESTORING NOM SNAPSHOT : /usr/local/nagiosxi/nom/checkpoints/nagioscore/1608001754.tar.gz

tar: usr/local/nagios/etc/pnp/check_commands: Cannot utime: Operation not permitted

tar: usr/local/nagios/etc/pnp/pages: Cannot utime: Operation not permitted

tar: usr/local/nagios/etc/pnp: Cannot utime: Operation not permitted

tar: usr/local/nagios/etc/hosts: Cannot utime: Operation not permitted

tar: usr/local/nagios/etc/import: Cannot utime: Operation not permitted

tar: usr/local/nagios/etc/services: Cannot utime: Operation not permitted

tar: usr/local/nagios/etc/static: Cannot utime: Operation not permitted

tar: usr/local/nagios/etc/objects: Cannot utime: Operation not permitted

tar: usr/local/nagios/etc: Cannot utime: Operation not permitted

tar: Exiting with failure status due to previous errors

~



--- reset_config_perms.sh ------------

> Setting script permissions

> Setting CCM script permissions

> Setting special script permissions

> Setting special component script permissions

> Setting configuration file/directory permissions

> Setting perfdata directory and RRD permissions

> Setting libexec directory permissions

> Setting Nagios XI config permissions

> Setting NOM checkpoint user:group permissions

> + Setting Nagios Core corelog.newobjects user:group permissions

> + Setting CCM configuration file user:group permissions

> + Setting Recurring Downtime file user:group permissions

> + Setting BPI configuration file user:group permissions

--------------------------------------

Restoring CCM databases...

Restore Complete.

OUTPUT=Restore Complete.

RETURNCODE=0

But the data inside ccm are not changing. I have install freshly new nagosxi 5.7.5 and it also showing the same error above but the ccm data do update upon restore.

Sudoers files are as below.

User_Alias NAGIOSXI=nagios
User_Alias NAGIOSXIWEB=apache
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios start
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios stop
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios restart
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios reload
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios status
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios checkconfig
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db start
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db stop
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db restart
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db reload
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db status
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd start
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd stop
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd restart
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd reload
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd status
NAGIOSXI ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/html/includes/components/autodiscovery/scripts/autodiscover_new.php *
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/html/includes/components/profile/getprofile.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/upgrade_to_latest.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/change_timezone.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/manage_services.sh *
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/reset_config_perms.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/backup_xi.sh *
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/messages
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/httpd/error_log
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/mysqld.log
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/html/includes/components/autodiscovery/scripts/autodiscover_new.php *
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/html/includes/components/profile/getprofile.sh
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/nmap *
NAGIOSXIWEB ALL = NOPASSWD:/etc/init.d/snmptt restart
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/scripts/repair_databases.sh
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/scripts/manage_services.sh *
NAGIOSXI ALL = NOPASSWD:/usr/local/nagios/libexec/check_as400 *
NAGIOSXI ALL = NOPASSWD:/bin/timeout *
#NAGIOSXI ALL = NOPASSWD:/usr/local/nagios/libexec/check_nrpe *
NAGIOSXI ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/scripts/components/autodiscover_new.php *
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/components/getprofile.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/manage_ssl_config.sh *
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/scripts/components/autodiscover_new.php *
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/scripts/components/getprofile.sh
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/scripts/components/autodiscover_new.php *

Nagios version. 5.6.7

config.inc.php look normal with proper password.
Is there any other place that i should be looking for?

Thanks
Safuan

[vtrac]

Hi safuanmansor,
It seems like the error was related to the "tar" command (below).
Telling me that "tar" is unable to restore the original timestamp of the files ... or if you are extracting onto a non-linux file system (vfat or ntfs).

Code: Select all

echo "RESTORING NOM SNAPSHOT : $ss"
tar -pxzf "$ss"

Please run the following commands and upload the results:

Code: Select all

chage -l nagios
grep nag /etc/group
ls -la /usr/local/nagios/etc
df -Th /usr/local/nagios/etc

Regards,
Vinh

[safuanmansor]

Hi Vinh,

Thanks for the reply,

The output for the command are as below,

chage -l nagios
------------------------------------------------------------
Last password change : Jul 16, 2018
Password expires : never
Password inactive : never
Account expires : never
Minimum number of days between password change : -1
Maximum number of days between password change : -1
Number of days of warning before password expires : -1

grep nag /etc/group
-----------------------------------------------------------
apache48:nagios
nagcmd1001:apache,nagios,snmptt
nagios1002:apache,nagios,snmptt

ls -la /usr/local/nagios/etc

image003.png

df -Th /usr/local/nagios/etc
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/datavg-lv_apps xfs 200G 108G 93G 54% /usr/local

I have pm u the system profile if you required

[vtrac]

Hi safuanmansor,
I was able to reproduced this on my Nagios XI machine as well.
First, I have confirmed that everything under "/usr/local/nagios/etc" folder were restored correctly and the Config restore was successful.

However, the "Cannot Utime" error was because user "apache" do not have permission to set the "restored" time for "directories".

I will bring this up to our development team.

Best Regards,
Vinh

[safuanmansor]

Hi Vinh,

Glad that you can reproduced it on your XI, it surely help in this troubleshooting.
When you said it is confirmed that everything under "/usr/local/nagios/etc" folder were restored correctly and the Config restore was successful.
What do you mean by that? Is my system actually OK and properly restored? Correct me if i'm wrong ya.

Normally what I'll do is, after i realize that the system is in wrong configuration or i accidentally deleted any host or service from the ccm portal, i will do the config snapshot restoration and ccm will restore to the previous state and i will verify the deleted host/service are back inside the ccm and then only i click the "Apply configuration" button .

Thanks
Safuan

[vtrac]

Hi safuanmansor,
I tested by restoring my config from the "Configuration Snapshots" (below).

Config_Restore.png

My config was restored even though the "cmdsubsys.log" said that it "cannot utime" these directories below.

Code: Select all

tar: usr/local/nagios/etc/static: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc/import: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc/hosts: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc/services: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc/pnp/check_commands: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc/pnp/pages: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc/pnp: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc: Cannot utime: Operation not permitted
tar: Exiting with failure status due to previous errors

This error is related to "apache" do not have permission to update the "TIME" on the "static", "import", "hosts", "services" ......
However, all files inside the "/usr/local/nagiosxi/nom/checkpoints/nagioscore/1608001754.tar.gz" tar file should be restored correctly.

You can check by running the below command and compare that with what you have on your system.
All file should match except the timestamp of directories under the "/usr/local/nagios/etc":

Code: Select all

tar -pztvf /usr/local/nagiosxi/nom/checkpoints/nagioscore/1608001754.tar.gz

Yes, your "config restore" should be fine!!

Best Regards,
Vinh

[safuanmansor]

Hi Vinh.

I think we are missing something here,
( Take a scenario where i accidentally delete 8 contact out of 100 contact and i didnt exactly know which contact was deleted)
*** All file should match except the timestamp of directories under the "/usr/local/nagios/etc": ***
Assuming that the files inside the /usr/local/nagios/etc is all restored. The nagios run the monitoring base on the configuration files that was restored. That is great but.....
Upon checking at the core config manager. The configuration are not same (the total number of contact are still 92). If i were to update a host or service , save it and click at the apply config. The new configuration will take place with 92 number of contact in CCM, not base on the number of contact from the previous/restored configuration files.

Hope thats explain the current issue that i'm facing right now. Please highlight if i missing any step to properly restore the data in CCM.

Thanks
Safuan

[vtrac]

Hi safuanmansor,
I was able to reproduced it from my XI where I deleted a user and then tried to restore using the "Configuration Snapshots" with no success even though the "contacts.cfg" do listed the "deleted" user.

I then contacted our development team and here are their replied:

Code: Select all

A Nagios XI user normally creates a contact in the CCM/Core (you can create a nagios xi user without a contact if you want actually) but if you remove the user from XI and restore an old CCM backup, it will not re-create the user in XI, they are not actually connected to one another, the CCM restore is just restoring the config files for Core at a specific point in time

You would have to do a full XI restore to get users back

There should also be daily backups for XI databases in "/store/backups/mysql/nagiosxi/" which customers could use to restore the db to a version that had the users in it.

So, to get those users back you must do the full Nagios XI DB restore.

Here's the URL on how to backup and restore your Nagios XI:
https://assets.nagios.com/downloads/nag ... ios-XI.pdf

Best Regards,
Vinh

[safuanmansor]

Hi Vinh,

We aware about the user can be contact while contact doesn't necessary a user. The discussion purely within CCM configuration data that was not restore during the restoration process.
Upon future investigation by reviewing the ccm_restore.sh script, part of the process of backup is by doing dumping of nagiosql database. We found out that the mysqldump command is failing on out system explaining the failed to restore of CCM data in nagios. We will fix that first and update back the status here so that it may help someone with the same situation.

Thanks.
Safuan

[vtrac]

Hi Safuan,
Thank you for your reply.

Yes, please update back with the status as it may help someone with the same (future) situation.

Regards,
Vinh