CCM config sometime visible and sometime invisible

[rajsshah]

Hi Team
There is a strange issue happening from last 2 days . When I go to CCM and select on hosts/services , some time I see my existing config and some time it is entirely blank ( no config at all ) . Any fix for this ??

[rajsshah]

Also when I try to restore the old config .. I see below in the logs


PROCESS COMMAND: CMD=1113, DATA=1564040301 restore
CMDLINE=/usr/local/nagiosxi/scripts/ccm_snapshot.sh 1564040301 restore
Restoring CCMl snapshot
Removing old files from /usr/local/nagios/etc
/ ~
RESTORING NOM SNAPSHOT : /usr/local/nagiosxi/nom/checkpoints/nagioscore/1564040301.tar.gz
tar: usr/local/nagios/etc/static: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc/import: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc/hosts: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc/services: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc/pnp/check_commands: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc/pnp/pages: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc/pnp: Cannot utime: Operation not permitted
tar: usr/local/nagios/etc: Cannot utime: Operation not permitted
tar: Exiting with failure status due to previous errors


Also see the Snap ..

[rajsshah]

Can any one please update ??? This is production issue for us . Same issue is happening with HOST and Service status . Some time it is showing the OLD status ( status of 2-3 hours back ) ..

[benjaminsmith]

Hi @rajsshah,

Have you made any changes to your server as it looks like the permissions are not correct. Please run the commands below and post the full output.

Code: Select all

ll /usr/local/nagios/etc
ll /usr/local/nagiosxi/scripts

Same issue is happening with HOST and Service status . Some time it is showing the OLD status ( status of 2-3 hours back ) ..

Do you see the same time difference in the core interface at: http://ipaddress/nagios.

Lastly, please send us your system profile for us to review. Thanks.

To send us your system profile.
Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and share in a private message and then reply to this post to bring it up in the queue.

[cdienger]

This sound like a permission issue. Make sure sudoers contains:

Code: Select all

User_Alias      NAGIOSXI=nagios
User_Alias      NAGIOSXIWEB=apache
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios start
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios stop
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios restart
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios reload
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios status
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios checkconfig
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db start
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db stop
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db restart
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db reload
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db status
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd start
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd stop
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd restart
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd reload
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd status
NAGIOSXI ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/scripts/components/autodiscover_new.php *
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/components/getprofile.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/upgrade_to_latest.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/change_timezone.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/manage_services.sh *
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/reset_config_perms.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/manage_ssl_config.sh *
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/backup_xi.sh *
NAGIOSXIWEB ALL = NOPASSWD:/etc/init.d/snmptt restart
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/messages
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/httpd/error_log
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/mysqld.log
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/scripts/components/autodiscover_new.php *
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/scripts/components/getprofile.sh
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/scripts/repair_databases.sh
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/scripts/manage_services.sh *

and check the permissions under /usr/local/nagios/etc/, /usr/local/nagios/etc/pnp/ and /usr/local/nagios/. The permissions on the directories it is complaining about should be:

Code: Select all

drwxrwxr-x 2 apache nagios  4096 Jul 19 16:11 hosts
drwxrwxr-x 2 apache nagios     6 Jul 18 14:15 import
drwxrwxr-x 2 apache nagios    42 Jul 19 16:11 nrpe
drwxrwxr-x 4 apache nagios  4096 Jul 19 16:11 pnp
drwxrwxr-x 2 apache nagios  4096 Jul 19 16:11 services
drwxrwxr-x 2 apache nagios    65 Jul 19 16:11 static

Code: Select all

drwxrwxr-x 2 apache nagios   36 Jul 19 16:11 check_commands
drwxrwxr-x 2 apache nagios   35 Jul 19 16:11 pages

Code: Select all

drwxrwxr-x  8 apache nagios 4096 Jul 19 16:11 etc

[rajsshah]

[root@XXXX~]# ll /usr/local/nagios/etc
total 208
-rw-rw-r-- 1 apache nagios 891 Jul 8 18:05 cgi.cfg
-rw-rw-r-- 1 apache nagios 21943 Jul 26 15:10 commands.cfg
-rw-rw-r-- 1 apache nagios 1263 Jul 26 15:10 contactgroups.cfg
-rw-rw-r-- 1 apache nagios 1181 Jul 26 15:10 contacts.cfg
-rw-rw-r-- 1 apache nagios 1610 Jul 26 15:10 contacttemplates.cfg
-rw-rw-r-- 1 apache nagios 796 Jul 26 15:10 hostdependencies.cfg
-rw-rw-r-- 1 apache nagios 794 Jul 26 15:10 hostescalations.cfg
-rw-rw-r-- 1 apache nagios 786 Jul 26 15:10 hostextinfo.cfg
-rw-rw-r-- 1 apache nagios 4419 Jul 26 15:10 hostgroups.cfg
drwxrwxr-x 2 apache nagios 8192 Jul 29 10:50 hosts
-rw-rw-r-- 1 apache nagios 22207 Jul 26 15:10 hosttemplates.cfg
drwxrwxr-x 2 apache nagios 6 Jul 24 16:41 import
-rw-rw-r-- 1 apache nagios 5770 Jul 29 10:50 nagios.cfg
-rw-rw-r-- 1 apache nagios 2229 Jun 28 13:13 ndo2db.bak
-rw-rw-r-- 1 apache nagios 2242 Jul 24 17:08 ndo2db.cfg
-rw-rw-r-- 1 apache nagios 4827 Jun 28 10:02 ndomod.cfg
-rw-rw-r-- 1 apache nagios 12463 Jun 28 10:03 nrpe.cfg
-rw-rw-r-- 1 apache nagios 5345 Jun 28 10:03 nsca.cfg
drwxrwxr-x 4 apache nagios 246 Jul 29 10:50 pnp
-rw-rw-r-- 1 apache nagios 0 Jun 28 10:12 recurringdowntime.cfg
-rw-rw-r-- 1 apache nagios 210 Jun 28 10:00 resource.cfg
-rw-rw-r-- 1 apache nagios 1627 Jun 28 10:03 send_nsca.cfg
-rw-rw-r-- 1 apache nagios 802 Jul 26 15:10 servicedependencies.cfg
-rw-rw-r-- 1 apache nagios 800 Jul 26 15:10 serviceescalations.cfg
-rw-rw-r-- 1 apache nagios 792 Jul 26 15:10 serviceextinfo.cfg
-rw-rw-r-- 1 apache nagios 790 Jul 26 15:10 servicegroups.cfg
drwxrwxr-x 2 apache nagios 8192 Jul 29 10:50 services
-rw-rw-r-- 1 apache nagios 23927 Jul 26 15:10 servicetemplates.cfg
drwxrwxr-x 2 apache nagios 68 Jul 29 10:50 static
-rw-rw-r-- 1 apache nagios 1788 Jul 26 15:10 timeperiods.cfg

[root@XXXX~]# ll /usr/local/nagiosxi/scripts
total 480
drwxr-xr-x 3 nagios nagios 35 Jun 28 10:03 automation
-r-xr-x--- 1 root nagios 7766 Jul 24 16:41 backup_xi.sh
-r-xr-x--- 1 nagios nagios 8165 Jul 24 16:41 ccm_delete_object.php
-r-xr-x--- 1 nagios nagios 1041 Jul 24 16:41 ccm_export.php
-r-xr-x--- 1 nagios nagios 1630 Jul 24 16:41 ccm_import.php
-r-xr-x--- 1 nagios nagios 3180 Jul 24 16:41 ccm_snapshot.sh
-r-xr-x--- 1 root nagios 1800 Jul 24 16:41 change_timezone.sh
drwxr-xr-x 2 root nagios 55 Jun 28 10:03 components
-rwxr-xr-x 1 nagios nagios 2839 Jul 24 16:41 contact_notification_handler.php
-rwxr-xr-x 1 nagios nagios 266 Jul 24 16:41 handle_nagioscore_event.php
-rwxr-xr-x 1 nagios nagios 3903 Jul 24 16:41 handle_nagioscore.inc.php
-rwxr-xr-x 1 nagios nagios 267 Jul 24 16:41 handle_nagioscore_notification.php
-r-xr-x--- 1 root nagios 1244 Jul 24 16:41 import_xiconfig.php
-rwxr-xr-x 1 nagios nagios 705 Jul 24 16:41 initialize_mibs.php
-r-xr-x--- 1 root nagios 3428 Jul 24 16:41 manage_services.sh
-r-xr-x--- 1 root nagios 3815 Jul 24 16:41 manage_ssl_config.sh
-rwxr-xr-x 1 nagios nagios 283685 Jul 24 16:41 nagiosql_defaults.sql
-rwxr-xr-x 1 nagios nagios 506 Jul 24 16:41 nom_create_nagioscore_checkpoint_cond.sh
-rwxr-xr-x 1 nagios nagios 986 Jul 24 16:41 nom_create_nagioscore_checkpoint.sh
-rwxr-xr-x 1 nagios nagios 785 Jul 24 16:41 nom_create_nagioscore_errorpoint.sh
-rwxr-xr-x 1 nagios nagios 968 Jul 24 16:41 nom_restore_nagioscore_checkpoint.sh
-rwxr-xr-x 1 nagios nagios 947 Jul 24 16:41 nom_restore_nagioscore_checkpoint_specific.sh
-rwxr-xr-x 1 nagios nagios 4627 Jul 24 16:41 nom_trim_checkpoints.sh
-rwxr-xr-x 1 nagios nagios 7779 Jul 24 16:41 nxti_import.php
-rwxr-xr-x 1 nagios nagios 1351 Jul 24 16:41 nxti.php
-rwxr-xr-x 1 nagios nagios 12484 Jul 24 16:41 parse_core_eventlog.php
-rwxr-xr-x 1 nagios nagios 6716 Jul 24 16:41 patch_ndoutils.php
-rwxr-xr-x 1 nagios nagios 971 Jul 24 16:41 reconfigure_nagios.sh
-rwxr-xr-x 1 nagios nagios 4224 Jul 24 16:41 remove_historical_data.sh
-r-xr-x--- 1 root nagios 1705 Jul 24 16:41 repair_databases.sh
-rwxr-xr-x 1 nagios nagios 3895 Jul 24 16:41 repairmysql.sh
-r-xr-x--- 1 root nagios 4835 Jul 24 16:41 reset_config_perms.sh
-rwxr-xr-x 1 nagios nagios 1859 Jul 24 16:41 reset_nagiosadmin_password.php
-rwxr-xr-x 1 nagios nagios 1881 Jul 24 16:41 restart_nagios_with_export.sh
-rwxr-xr-x 1 nagios nagios 4027 Jul 24 16:41 restore_defaults.sh
-rwxr-xr-x 1 nagios nagios 13105 Jul 24 16:41 restore_xi.sh
-rwxr-xr-x 1 nagios nagios 1908 Jul 24 16:41 send_to_auditlog.php
-rwxr-xr-x 1 nagios nagios 1345 Jul 24 16:41 unlock_user_account.php
-rwxr-xr-x 1 nagios nagios 722 Jul 24 16:41 update_check.php
-r-xr-x--- 1 root nagios 2914 Jul 24 16:41 upgrade_to_latest.sh

[rajsshah]

Also while updating contact notification I get below :

There were 2 errors while processing this request
Error while inserting data into the database:
Table 'nagiosql.tbl_lnkcontacttovariabledefinition' doesn't exist


I can see the table in the database . However the name is in uppercase . "tbl_lnkContactToVariabledefinition" . In the error the table name is in small case.

[tgriep]

How many Host and Services is the server checking?
How many users are logged in to the system and are they doing a lot of changes to the system?

First, run the following as root to repair the databases to see if that fixes the issue.

Code: Select all

mysqlcheck -f -r -u root -pnagiosxi --all-databases --use-frm

If not, login to the server in a shell and run the following commands to see is there are any errors when running the command line version of the Apply Config. Post all of the output.

Code: Select all

su - nagios
cd /usr/local/nagiosxi/scripts
./reconfigure_nagios.sh

If you cannot post your System Profile, try restarting the nagios stack to see if that fixes the issue.
Run this as root.

Code: Select all

service npcd stop
service nagios stop
service ndo2db stop
service crond stop
if grep --quiet pgsql /usr/local/nagiosxi/html/config.inc.php; then service postgresql stop; fi;
service mysqld restart
rm -f /usr/local/nagios/var/rw/nagios.cmd
rm -f /usr/local/nagios/var/nagios.lock
rm -f /var/run/nagios.lock
rm -f /usr/local/nagios/var/ndo.sock
rm -f /usr/local/nagios/var/ndo2db.lock
rm -f /var/lib/mrtg/mrtg_l
rm -f /usr/local/nagiosxi/var/*.lock
for i in `ipcs -q | grep nagios |awk '{print $2}'`; do ipcrm -q $i; done
pkill -9 -u nagios
pkill python
if grep --quiet pgsql /usr/local/nagiosxi/html/config.inc.php; then service postgresql start; fi;
service httpd restart
service ndo2db start
service nagios start
service npcd start
service crond start

[rajsshah]

You can close this incident . The issue was that I was using perconaxtradb as backend for nagios . I have reverted the config to use single mysql isntance using DRBD.

[scottwilkerson]

You can close this incident . The issue was that I was using perconaxtradb as backend for nagios . I have reverted the config to use single mysql isntance using DRBD.

Great!

Closing