Nagios Support Forum

Support for Nagios products and services
https://support.nagios.com/forum/

Weird Renaming Problem

https://support.nagios.com/forum/viewtopic.php?t=33103

Page 2 of 2

Re: Weird Renaming Problem

Posted: Tue Jun 02, 2015 9:00 am
by WillemDH
We resolved all warnings after a normal apply configuration and doubled all values you mentioned in php.ini.

Code: Select all

; Maximum execution time of each script, in seconds
; http://www.php.net/manual/en/info.configuration.php#ini.max-execution-time
max_execution_time = 120

; Maximum amount of time each script may spend parsing request data. It's a good
; idea to limit this time on productions servers in order to eliminate unexpectedly
; long running scripts.
; Default Value: -1 (Unlimited)
; Development Value: 60 (60 seconds)
; Production Value: 60 (60 seconds)
; http://www.php.net/manual/en/info.configuration.php#ini.max-input-time
max_input_time = 120

; Maximum input variable nesting level
; http://www.php.net/manual/en/info.configuration.php#ini.max-input-nesting-level
;max_input_nesting_level = 64

; Maximum amount of memory a script may consume (128MB)
; http://www.php.net/manual/en/ini.core.php#ini.memory-limit
memory_limit = 512M
But I'm still experiencing exactly the same issue as before. Applying after renaming a host takes ages and then apply configuration fails with error:

Error: Template 'dig_srv_win_drv_A_usage_prio2' specified in service definition could not be not found (config file '/usr/local/nagios/etc/services/testserver.cfg', starting on line 52)

testserver.cfg does not exist on the Nagios XI server.

Tail of cmdsubsys.log

Code: Select all

 tail -f /usr/local/nagiosxi/var/cmdsubsys.log
/usr/bin/wget --load-cookies=nagiosql.cookies http://localhost/nagiosxi/includes/components/ccm/ --no-check-certificate --post-data 'cmd=apply&type=writeConfig' -O nagiosql.export.monitoring
--2015-06-02 15:43:11--  http://localhost/nagiosxi/includes/components/ccm/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: “nagiosql.export.monitoring”

     0K ........
PROCESSED 1 COMMANDS
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
............................................................tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated

PROCESSED 0 COMMANDS
.                                              74.4 =2m15s

2015-06-02 15:45:27 (74.4 B/s) - “nagiosql.export.monitoring” saved [10078]

NAGIOSQL WRITE CONFIGS FAILED!\n
OUTPUT=NAGIOSQL WRITE CONFIGS FAILED!\n
RETURNCODE=5

PROCESSED 1 COMMANDS
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
............................................................tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated

PROCESSED 0 COMMANDS
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
When I try rolling back to the latest config snapshot, tailing cmdsubsys.log again gives me:

Code: Select all

tail -f /usr/local/nagiosxi/var/cmdsubsys.log
Another reconfigure process is still running, sleeping...
Another reconfigure process is still running, sleeping...
Another reconfigure process is still running, sleeping...
Another reconfigure process is still running, sleeping...
Another reconfigure process is still running, sleeping...
Another reconfigure process is still running, sleeping...
Another reconfigure process is still running, sleeping...
Another reconfigure process is still running, sleeping...
Another reconfigure process is still running, sleeping...
Another reconfigure process is still running, sleeping...
After a long time, the configuration gets applied, but the host is not renamed it seems.
Please advice how to continue troubleshoot this.

Re: Weird Renaming Problem

Posted: Tue Jun 02, 2015 9:12 am
by tgriep
Could you run the following commands and post the output back?

Code: Select all

ls -lR /usr/local/nagiosxi/scripts
ps aux

Re: Weird Renaming Problem

Posted: Wed Jun 03, 2015 1:33 am
by WillemDH
Tom,

Code: Select all

/usr/local/nagiosxi/scripts
/usr/local/nagiosxi/scripts:
total 976
-rwxr-x---. 1 nagios nagios   4653 Jan 28  2014 backup_xi_old.sh
-rwxr-x---  1 nagios nagios   4696 Feb 25 12:49 backup_xi.sh
-rwxr-x---  1 nagios nagios   1538 Feb 25 12:49 change_timezone.sh
-rw-r-----  1 nagios nagios   2532 Feb 25 12:49 contact_notification_handler.php
-rwxr-x---. 1 nagios nagios    652 Feb 25 12:49 export_nagiosql.sh
-rwxr-x---. 1 nagios nagios    824 Feb 25 12:49 handle_nagioscore_event.php
-rwxr-x---. 1 nagios nagios    832 Feb 25 12:49 handle_nagioscore_notification.php
-rwxr-x---. 1 nagios nagios    432 Feb 25 12:49 import_nagiosql.sh
-rw-r-----  1 nagios nagios    908 Feb 25 12:49 import_xiconfig.php
-rwxr-x---. 1 nagios nagios    153 Feb 25 12:49 kill_rrdtool.sh
-rwxr-x---  1 nagios nagios   1823 Feb 25 12:49 manage_services.sh
-rwxrwxr-x  1 nagios nagios    161 Jun  2 16:03 nagiosql.cookies
-rwxrwxr-x. 1 nagios nagios 271780 Feb 25 12:49 nagiosql_defaults.sql
-rwxrwxr-x  1 nagios nagios  38983 May 28 13:38 nagiosql.delete.contact
-rwxrwxr-x. 1 nagios nagios    952 Feb 25 12:49 nagiosql_delete_contact.php
-rwxrwxr-x  1 nagios nagios  31864 Aug 20  2013 nagiosql.delete.host
-rwxrwxr-x. 1 nagios nagios   1991 Feb 25 12:49 nagiosql_delete_host.php
-rwxrwxr-x. 1 nagios nagios    509 Feb 25 12:49 nagiosql_delete_object.sh
-rwxrwxr-x  1 nagios nagios 157971 May 29 09:43 nagiosql.delete.service
-rwxrwxr-x. 1 nagios nagios   2846 Feb 25 12:49 nagiosql_delete_service.php
-rwxrwxr-x  1 nagios nagios  23433 Dec 23  2013 nagiosql.delete.timeperiod
-rwxrwxr-x. 1 nagios nagios    955 Feb 25 12:49 nagiosql_delete_timeperiod.php
-rwxrwxr-x  1 nagios nagios   7400 Jun 12  2014 nagiosql.export.additional
-rwxrwxr-x. 1 nagios nagios   1334 Feb 25 12:49 nagiosql_exportall.php
-rwxrwxr-x  1 nagios nagios  14445 Jun  2 16:03 nagiosql.export.monitoring
-rwxrwxr-x. 1 nagios nagios   1333 Feb 25 12:49 nagiosql_importall.php
-rwxrwxr-x  1 nagios nagios 227320 Jun  2 15:20 nagiosql.import.monitoring
-rwxrwxr-x  1 nagios nagios  11589 Jun  2 16:03 nagiosql.login
-rwxrwxr-x. 1 nagios nagios   1849 Feb 25 12:49 nagiosql_login.php
-rwxrwxr-x  1 nagios nagios   2246 Feb 25 12:49 nagiosql_snapshot.sh
-rwxrwxr-x. 1 nagios nagios    262 Feb 25 12:49 nagiosql_trim_backups.sh
-rwxr-x---. 1 nagios nagios    519 Feb 25 12:49 nom_create_nagioscore_checkpoint_cond.sh
-rwxr-x---. 1 nagios nagios    826 Feb 25 12:49 nom_create_nagioscore_checkpoint.sh
-rwxr-x---. 1 nagios nagios    635 Feb 25 12:49 nom_create_nagioscore_errorpoint.sh
-rwxr-x---. 1 nagios nagios    788 Feb 25 12:49 nom_restore_nagioscore_checkpoint.sh
-rwxr-x---. 1 nagios nagios    839 Feb 25 12:49 nom_restore_nagioscore_checkpoint_specific.sh
-rwxr-x---. 1 nagios nagios   2972 Feb 25 12:49 nom_trim_nagioscore_checkpoints.sh
-rwxr-x---. 1 nagios nagios   4621 Feb 25 12:49 parse_core_eventlog.php
-rwxr-x---. 1 nagios nagios   6077 Feb 25 12:49 patch_ndoutils.php
-rwxr-x---. 1 nagios nagios    600 Feb 25 12:49 reconfigure_nagios.sh
-rwxr-x---  1 nagios nagios    522 Feb 25 12:49 repair_databases.sh
-rwxr-x---. 1 nagios nagios   1222 Feb 25 12:49 repairmysql.sh
-rwsr-xr-x. 1 root   nagios   6798 Feb 25 12:49 reset_config_perms
-rw-r-----. 1 nagios nagios    280 Feb 25 12:49 reset_config_perms.c
-rwsr-xr-x. 1 root   nagios   1347 Feb 25 12:49 reset_config_perms.sh
-rwxr-x---. 1 nagios nagios   1155 Feb 25 12:49 reset_nagiosadmin_password.php
-rwxr-x---. 1 nagios nagios   1621 Feb 25 12:49 restart_nagios_with_export.sh
-rwxr-x---. 1 nagios nagios    765 Feb 25 12:49 restore_defaults.sh
-rwxr-x---. 1 nagios nagios   5731 Feb 25 12:49 restore_xi.sh
-rw-r-----  1 nagios nagios   1847 Feb 25 12:49 send_to_auditlog.php
-rw-r-----. 1 nagios nagios   2379 Feb 25 12:49 uninstall_xi.sh
-rwxr-x---  1 nagios nagios   2348 Feb 25 12:49 upgrade_to_latest.sh
The ps -aux contains too many sensitive info, so I pm'ed you the output. So how can I find out where this testserver is located?

Grtz

Willem

Re: Weird Renaming Problem

Posted: Wed Jun 03, 2015 8:47 am
by tgriep
I am thinking that the mysql database might be corrupted. Check the /var/log/mysqld.log file for any errors. If there are, do this to repair it in a shell on the XI server.

Code: Select all

cd /usr/local/nagiosxi/scripts
./repair_databases.sh
One thing to try is to create the template that is missing "dig_srv_win_drv_A_usage_prio2" and try the renaming tool.

Re: Weird Renaming Problem

Posted: Thu Jun 04, 2015 2:11 am
by WillemDH
Tom,

I checked the /var/log/mysqld.log => No logs since 23/04

These are the last entries:

Code: Select all

150413 14:01:35 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150413 14:02:52 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150413 14:02:53  InnoDB: Initializing buffer pool, size = 8.0M
150413 14:02:53  InnoDB: Completed initialization of buffer pool
150413 14:02:53  InnoDB: Started; log sequence number 0 44233
150413 14:02:53 [Note] Event Scheduler: Loaded 0 events
150413 14:02:53 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
150422 10:14:54 [Note] /usr/libexec/mysqld: Normal shutdown

150422 10:14:54 [Note] Event Scheduler: Purging the queue. 0 events
150422 10:14:55  InnoDB: Starting shutdown...
150422 10:15:00  InnoDB: Shutdown completed; log sequence number 0 44233
150422 10:15:00 [Note] /usr/libexec/mysqld: Shutdown complete

150422 10:15:00 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150422 10:17:24 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150422 10:17:25  InnoDB: Initializing buffer pool, size = 8.0M
150422 10:17:25  InnoDB: Completed initialization of buffer pool
150422 10:17:25  InnoDB: Started; log sequence number 0 44233
150422 10:17:26 [Note] Event Scheduler: Loaded 0 events
150422 10:17:26 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
150422 10:52:31 [Note] /usr/libexec/mysqld: Normal shutdown

150422 10:52:31 [Note] Event Scheduler: Purging the queue. 0 events
150422 10:52:33  InnoDB: Starting shutdown...
150422 10:52:38  InnoDB: Shutdown completed; log sequence number 0 44233
150422 10:52:38 [Note] /usr/libexec/mysqld: Shutdown complete

150422 10:52:38 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150422 10:54:11 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150422 10:54:12  InnoDB: Initializing buffer pool, size = 8.0M
150422 10:54:12  InnoDB: Completed initialization of buffer pool
150422 10:54:12  InnoDB: Started; log sequence number 0 44233
150422 10:54:12 [Note] Event Scheduler: Loaded 0 events
150422 10:54:12 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
150423 14:38:55 [Note] /usr/libexec/mysqld: Normal shutdown

150423 14:38:55 [Note] Event Scheduler: Purging the queue. 0 events
150423 14:38:56  InnoDB: Starting shutdown...
150423 14:38:57  InnoDB: Shutdown completed; log sequence number 0 44233
150423 14:38:57 [Note] /usr/libexec/mysqld: Shutdown complete

150423 14:38:57 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150423 14:49:24 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150423 14:49:25  InnoDB: Initializing buffer pool, size = 8.0M
150423 14:49:25  InnoDB: Completed initialization of buffer pool
150423 14:49:25  InnoDB: Started; log sequence number 0 44233
150423 14:49:25 [Note] Event Scheduler: Loaded 0 events
150423 14:49:25 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
I tried creating the dig_srv_win_drv_A_usage_prio2 service template and then renamed a host. Same problem. Applying configuration takes a very long time. THis is the logging from cmdsubsys

Code: Select all

tail -f /usr/local/nagiosxi/var/cmdsubsys.log
...........................................................
PROCESSED 0 COMMANDS
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
................................................PROCESSING COMMAND ID 18119...
PROCESS COMMAND: CMD=17, DATA=
APPLYING NAGIOSCORE CONFIG...
CMDLINE=cd /usr/local/nagiosxi/scripts && ./reconfigure_nagios.sh
URL: http://localhost/nagiosxi/includes/components/ccm/
CMDLINE
--2015-06-04 09:00:50--  http://localhost/nagiosxi/includes/components/ccm/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: “nagiosql.login”

     0K .......... .                                            934K=0.01s

2015-06-04 09:00:50 (934 KB/s) - “nagiosql.login” saved [11589]

/usr/bin/wget --save-cookies nagiosql.cookies --keep-session-cookies http://localhost/nagiosxi/includes/components/ccm/ --no-check-certificate --post-data 'submit=Login&hidelog=true&loginSubmitted=true&username=nagiosxi&password=pasw' -O nagiosql.loginLOGIN SUCCESSFUL!
IMPORTING CONFIG FILES...URL: http://localhost/nagiosxi/includes/components/ccm/
Array
(
)
RESETTING PERMS
SETUID ROOT OK
URL: http://localhost/nagiosxi/includes/components/ccm/
CMDLINE
--2015-06-04 09:00:51--  http://localhost/nagiosxi/includes/components/ccm/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: “nagiosql.login”

     0K .......... .                                           1.01M=0.01s

2015-06-04 09:00:51 (1.01 MB/s) - “nagiosql.login” saved [11589]

/usr/bin/wget --save-cookies nagiosql.cookies --keep-session-cookies http://localhost/nagiosxi/includes/components/ccm/ --no-check-certificate --post-data 'submit=Login&hidelog=true&loginSubmitted=true&username=nagiosxi&password=pasw' -O nagiosql.loginLOGIN SUCCESSFUL!
URL: http://localhost/nagiosxi/includes/components/ccm/
CMDLINE:
/usr/bin/wget --load-cookies=nagiosql.cookies http://localhost/nagiosxi/includes/components/ccm/ --no-check-certificate --post-data 'cmd=apply&type=writeConfig' -O nagiosql.export.monitoring
--2015-06-04 09:00:51--  http://localhost/nagiosxi/includes/components/ccm/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: “nagiosql.export.monitoring”

     0K ........tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
............................................................tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
.
PROCESSED 0 COMMANDS
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated

PROCESSED 0 COMMANDS
.                                              74.4 =2m15s

2015-06-04 09:03:07 (74.4 B/s) - “nagiosql.export.monitoring” saved [10078]

NAGIOSQL WRITE CONFIGS FAILED!\n
OUTPUT=NAGIOSQL WRITE CONFIGS FAILED!\n
RETURNCODE=5

PROCESSED 1 COMMANDS
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
............................................................tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
.
PROCESSED 0 COMMANDS
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
.
PROCESSED 0 COMMANDS
In the gui, I still get this error: Error: Template 'dig_srv_win_drv_A_usage_prio2' specified in service definition could not be not found (config file '/usr/local/nagios/etc/services/testserver.cfg', starting on line 52)

For the first time I did not try to revert to the last ok config cnapshot and tried to apply one more time. tailing cmdsubsys.log again gave

Code: Select all

Another reconfigure process is still running, sleeping...
Another reconfigure process is still running, sleeping...
Another reconfigure process is still running, sleeping...
Another reconfigure process is still running, sleeping...
So I guess this implies the apply config after renaming a host actually takes even longer. And it seems the config actually gets applied the second time (after a long time)!

Code: Select all

Another reconfigure process is still running, sleeping...
Another reconfigure process is still running, sleeping...
Another reconfigure process is still running, sleeping...
Another reconfigure process is still running, sleeping...
Another reconfigure process is still running, sleeping...
Another reconfigure process is still running, sleeping...
Another reconfigure process is still running, sleeping...
RESETTING PERMS
SETUID ROOT OK
URL: http://localhost/nagiosxi/includes/components/ccm/
CMDLINE
--2015-06-04 09:12:50--  http://localhost/nagiosxi/includes/components/ccm/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: “nagiosql.login”

     0K .......... .                                           1.07M=0.01s

2015-06-04 09:12:50 (1.07 MB/s) - “nagiosql.login” saved [11589]

/usr/bin/wget --save-cookies nagiosql.cookies --keep-session-cookies http://localhost/nagiosxi/includes/components/ccm/ --no-check-certificate --post-data 'submit=Login&hidelog=true&loginSubmitted=true&username=nagiosxi&password=passw' -O nagiosql.loginLOGIN SUCCESSFUL!
URL: http://localhost/nagiosxi/includes/components/ccm/
CMDLINE:
/usr/bin/wget --load-cookies=nagiosql.cookies http://localhost/nagiosxi/includes/components/ccm/ --no-check-certificate --post-data 'cmd=apply&type=writeConfig' -O nagiosql.export.monitoring
--2015-06-04 09:12:51--  http://localhost/nagiosxi/includes/components/ccm/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: “nagiosql.export.monitoring”

     0K ........tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated

PROCESSED 0 COMMANDS
.. .......... .                                1.38K=15s

2015-06-04 09:13:06 (1.38 KB/s) - “nagiosql.export.monitoring” saved [21688]

WRITE CONFIGS SUCCESSFUL!
OUTPUT:
Nagios Core 4.0.8
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 08-12-2014
License: GPL

Website: http://www.nagios.org
Reading configuration data...
   Read main config file okay...
   Read object config files okay...

Running pre-flight check on configuration data...

Checking objects...
        Checked 12689 services.
        Checked 744 hosts.
        Checked 114 host groups.
        Checked 83 service groups.
        Checked 65 contacts.
        Checked 16 contact groups.
        Checked 284 commands.
        Checked 79 time periods.
        Checked 0 host escalations.
        Checked 0 service escalations.
Checking for circular paths...
        Checked 744 hosts
        Checked 0 service dependencies
        Checked 0 host dependencies
        Checked 79 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 0
Total Errors:   0

Things look okay - No serious problems were detected during the pre-flight check
RET: 0
Running configuration check...
Stopping nagios:. done.
Starting nagios: done.
OUTPUT=Starting nagios: done.
RETURNCODE=0

PROCESSED 1 COMMANDS
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
............................................................tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
.
PROCESSED 0 COMMANDS
And the host was correctly renamed. The weird thing when looking at config changes is that a whole bunch of hosts that have nothing to do with the renamed hosts have also been rewritten.. I don't understand what's going on here.

EDIT:
SO I decided to give my XI prod server a reboot and did all the above tests again. It seems things are better now. Renaming a host actually works the first apply, but it still takes a very long time. Looking at the config changes of the apply show he wrote all the config files of all my 700 hosts, so I guess that's why it takes so long. This does not seem 'normal' to me? Or is it by design that after renaming one hosts, all hosts are rewritten? I would suspent only the affected host and service cfg (of the renamed host) should be written?

Grtz

Re: Weird Renaming Problem

Posted: Thu Jun 04, 2015 9:09 am
by lmiltchev
This does not seem 'normal' to me? Or is it by design that after renaming one hosts, all hosts are rewritten? I would suspent only the affected host and service cfg (of the renamed host) should be written?
I will discuss this with our developers and get back to you. FYI, I can verify that renaming a single host via the Renaming tool changes the status of ALL hosts/services in the CCM to "Not Applied".

Re: Weird Renaming Problem

Posted: Thu Jun 04, 2015 9:33 am
by lmiltchev
Update: I heard back from the devs. This is "by design". If you name-change a parent, the child configs need to be updated as well. So instead of tediously checking for everywhere an association could be made, it's easier to just rewrite all. Besides, you can make many changes before applying the configuration (don't need to apply it after each small change). Select "Do not apply config at the end of the wizard" check-box in Stage 3 of the Bulk Renaming Tool.
All times are UTC-05:00
Page 2 of 2

Powered by phpBB® Forum Software © phpBB Limited