manual ndo2db restart required after "apply changes"

[kendallchenoweth]

On a particular Nagios XI 2014 R2.0, I'm finding that when I apply changes, I need to manually restart ndo2db service before Nagios XI services (stuff under Monitoring Process Engine) starts. I can click on the start for the services, but that won't work until I manually restart ndo2db (service ndo2db restart.)

What information from my system can I provide you to help me solve this problem?

Thanks!

[jdalrymple]

I think starting with the output of

Code: Select all

/usr/local/nagiosxi/scripts/reconfigure_nagios.sh

would be a good start. Does this happen consistently or sporadically?

[kendallchenoweth]

It happens consistently and doesn't matter if I've made a change or just re-applied a non changing config.

Code: Select all

URL: https://localhost/nagiosxi/includes/components/ccm/
CMDLINE
/usr/bin/wget --save-cookies nagiosql.cookies --keep-session-cookies https://localhost/nagiosxi/includes/components/ccm/ --no-che
ck-certificate --post-data 'submit=Login&hidelog=true&loginSubmitted=true&username=nagiosxi&password=guo64u' -O nagiosql.login--2
015-04-09 10:15:02--  https://localhost/nagiosxi/includes/components/ccm/
Resolving localhost... 127.0.0.1
Connecting to localhost|127.0.0.1|:443... connected.
WARNING: cannot verify localhost’s certificate, issued by “/C=US/ST=MA/L=Natick/O=the MathWorks/OU=SSG/CN=ec2-50-19-171-201.compu
te1.amazonaws.com/[email protected]”:
  Self-signed certificate encountered.
WARNING: certificate common name “ec2-50-19-171-201.compute1.amazonaws.com” doesn’t match requested host name “localhost”.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: “nagiosql.login”

    [ <=>                                                                                    ] 10,409      --.-K/s   in 0.02s   

2015-04-09 10:15:03 (557 KB/s) - “nagiosql.login” saved [10409]

LOGIN SUCCESSFUL!
IMPORTING CONFIG FILES...URL: https://localhost/nagiosxi/includes/components/ccm/
Array
(
)
SETUID ROOT OK
RESETTING PERMS
URL: https://localhost/nagiosxi/includes/components/ccm/
CMDLINE
/usr/bin/wget --save-cookies nagiosql.cookies --keep-session-cookies https://localhost/nagiosxi/includes/components/ccm/ --no-che
ck-certificate --post-data 'submit=Login&hidelog=true&loginSubmitted=true&username=nagiosxi&password=guo64u' -O nagiosql.login--2
015-04-09 10:15:03--  https://localhost/nagiosxi/includes/components/ccm/
Resolving localhost... 127.0.0.1
Connecting to localhost|127.0.0.1|:443... connected.
WARNING: cannot verify localhost’s certificate, issued by “/C=US/ST=MA/L=Natick/O=the MathWorks/OU=SSG/CN=ec2-50-19-171-201.compu
te1.amazonaws.com/[email protected]”:
  Self-signed certificate encountered.
WARNING: certificate common name “ec2-50-19-171-201.compute1.amazonaws.com” doesn’t match requested host name “localhost”.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: “nagiosql.login”

    [ <=>                                                                                    ] 10,409      --.-K/s   in 0.02s   

2015-04-09 10:15:03 (488 KB/s) - “nagiosql.login” saved [10409]

LOGIN SUCCESSFUL!
URL: https://localhost/nagiosxi/includes/components/ccm/
CMDLINE:
/usr/bin/wget --load-cookies=nagiosql.cookies https://localhost/nagiosxi/includes/components/ccm/ --no-check-certificate --post-d
ata 'cmd=apply&type=writeConfig' -O nagiosql.export.monitoring
--2015-04-09 10:15:04--  https://localhost/nagiosxi/includes/components/ccm/
Resolving localhost... 127.0.0.1
Connecting to localhost|127.0.0.1|:443... connected.
WARNING: cannot verify localhost’s certificate, issued by “/C=US/ST=MA/L=Natick/O=the MathWorks/OU=SSG/CN=ec2-50-19-171-201.compu
te1.amazonaws.com/[email protected]”:
  Self-signed certificate encountered.
WARNING: certificate common name “ec2-50-19-171-201.compute1.amazonaws.com” doesn’t match requested host name “localhost”.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: “nagiosql.export.monitoring”

    [   <=>                                                                                  ] 8,278       --.-K/s              
    [      <=>                                                                               ] 13,543      2.95K/s   in 4.5s    

2015-04-09 10:15:08 (2.95 KB/s) - “nagiosql.export.monitoring” saved [13543]

WRITE CONFIGS SUCCESSFUL!
OUTPUT: 
Nagios Core 4.0.8
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 08-12-2014
License: GPL

Website: http://www.nagios.org
Reading configuration data...
   Read main config file okay...
   Read object config files okay...

Running pre-flight check on configuration data...

Checking objects...
	Checked 461 services.
	Checked 114 hosts.
	Checked 113 host groups.
	Checked 31 service groups.
	Checked 31 contacts.
	Checked 23 contact groups.
	Checked 197 commands.
	Checked 11 time periods.
	Checked 18 host escalations.
	Checked 11594 service escalations.
Checking for circular paths...
	Checked 114 hosts
	Checked 0 service dependencies
	Checked 0 host dependencies
	Checked 11 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 0
Total Errors:   0

Things look okay - No serious problems were detected during the pre-flight check
RET: 0
Running configuration check...done.
Stopping nagios: .done.
Starting nagios: done.
[nagios@ip-10-5-189-249 scripts]$ ^C
[nagios@ip-10-5-189-249 scripts]$ exit

Script done on Thu 09 Apr 2015 10:15:37 AM EDT

[kendallchenoweth]

It does restart successfully but not as quick as I'd like. I'm going to try to time how long it takes and see if the restart is all of the time.

[lmiltchev]

Can you also run the following command and show us the output?

Code: Select all

grep ndo2db /var/log/messages | tail -50

Are you using mod gearman or mk livestatus?

[abrist]

I need to manually restart ndo2db service before Nagios XI services (stuff under Monitoring Process Engine) starts.

That is really odd as most of those services - performance grapher, database maint, etc are not restarted when apply config is run and they should be checked by a cron.
Does ndo2db stop on an apply config?
Could you maybe post some more in depth descriptions of the behavior you are experiencing (screenshots may help here as well).

[kendallchenoweth]

I am getting a lost database connection (remote AWS database) when this happens.

Code: Select all

May  4 10:46:18 ip-10-5-189-249 nagios: ndomod: Please check remote ndo2db log, database connection or SSL Parameters
May  4 10:49:04 ip-10-5-189-249 nagios: ndomod: Please check remote ndo2db log, database connection or SSL Parameters
May  4 10:49:04 ip-10-5-189-249 nagios: ndomod: Please check remote ndo2db log, database connection or SSL Parameters
May  4 11:01:40 ip-10-5-189-249 nagios: ndomod: Please check remote ndo2db log, database connection or SSL Parameters
May  4 11:06:37 ip-10-5-189-249 nagios: ndomod: Please check remote ndo2db log, database connection or SSL Parameters
May  4 11:13:43 ip-10-5-189-249 nagios: ndomod: Please check remote ndo2db log, database connection or SSL Parameters
May  4 11:15:06 ip-10-5-189-249 nagios: ndomod: Please check remote ndo2db log, database connection or SSL Parameters
May  4 11:25:18 ip-10-5-189-249 nagios: ndomod: Please check remote ndo2db log, database connection or SSL Parameters
May  4 11:26:39 ip-10-5-189-249 nagios: ndomod: Please check remote ndo2db log, database connection or SSL Parameters
May  4 11:27:39 ip-10-5-189-249 nagios: ndomod: Please check remote ndo2db log, database connection or SSL Parameters
May  4 11:29:40 ip-10-5-189-249 nagios: ndomod: Please check remote ndo2db log, database connection or SSL Parameters
May  4 11:33:18 ip-10-5-189-249 nagios: ndomod: Please check remote ndo2db log, database connection or SSL Parameters
May  4 12:08:46 ip-10-5-189-249 nagios: ndomod: Please check remote ndo2db log, database connection or SSL Parameters
May  4 12:14:23 ip-10-5-189-249 nagios: ndomod: Please check remote ndo2db log, database connection or SSL Parameters
May  4 12:23:06 ip-10-5-189-249 nagios: ndomod: Please check remote ndo2db log, database connection or SSL Parameters
May  4 12:25:10 ip-10-5-189-249 nagios: ndomod: Please check remote ndo2db log, database connection or SSL Parameters
May  4 12:25:15 ip-10-5-189-249 nagios: ndomod: Please check remote ndo2db log, database connection or SSL Parameters
May  4 12:30:18 ip-10-5-189-249 nagios: ndomod: Please check remote ndo2db log, database connection or SSL Parameters
May  6 10:05:20 ip-10-5-189-249 ndo2db: Error: mysql_query() failed for 'UPDATE nagios_conninfo SET disconnect_time=NOW(), last_checkin_time=NOW(), data_end_time=FROM_UNIXTIME(0), bytes_processed='0', lines_processed='0', entries_processed='0' WHERE conninfo_id='0''
May  6 10:05:20 ip-10-5-189-249 ndo2db: mysql_error: 'MySQL server has gone away'
May  6 10:05:20 ip-10-5-189-249 ndo2db: Error: Connection to MySQL database has been lost!
May  6 10:08:31 ip-10-5-189-249 nagios: ndomod: Please check remote ndo2db log, database connection or SSL Parameters
May  6 10:16:20 ip-10-5-189-249 nagios: ndomod: Please check remote ndo2db log, database connection or SSL Parameters
May  6 10:17:13 ip-10-5-189-249 nagios: ndomod: Please check remote ndo2db log, database connection or SSL Parameters
May  6 10:19:10 ip-10-5-189-249 nagios: ndomod: Please check remote ndo2db log, database connection or SSL Parameters

Based on some previous posts, I've read it might have something to do with ndomod.cfg. Here is my ndomod.cfg.

Code: Select all

#####################################################################
# NDOMOD CONFIG FILE
#####################################################################


# INSTANCE NAME
# This option identifies the "name" associated with this particular
# instance of Nagios and is used to seperate data coming from multiple
# instances.  Defaults to 'default' (without quotes).

instance_name=localhost



# OUTPUT TYPE
# This option determines what type of output sink the NDO NEB module
# should use for data output.  Valid options include:
#   file       = standard text file
#   tcpsocket  = TCP socket
#   unixsocket = UNIX domain socket (default)

#output_type=file
#output_type=tcpsocket
output_type=unixsocket



# OUTPUT
# This option determines the name and path of the file or UNIX domain 
# socket to which output will be sent if the output type option specified
# above is "file" or "unixsocket", respectively.  If the output type
# option is "tcpsocket", this option is used to specify the IP address
# of fully qualified domain name of the host that the module should
# connect to for sending output.

#output=/usr/local/nagios/var/ndo.dat
#output=127.0.0.1
output=/usr/local/nagios/var/ndo.sock



# TCP PORT
# This option determines what port the module will connect to in
# order to send output.  This option is only vlaid if the output type
# option specified above is "tcpsocket".

tcp_port=5668



# OUTPUT BUFFER
# This option determines the size of the output buffer, which will help
# prevent data from getting lost if there is a temporary disconnect from
# the data sink.  The number of items specified here is the number of
# lines (each of variable size) of output that will be buffered.

output_buffer_items=5000



# BUFFER FILE
# This option is used to specify a file which will be used to store the
# contents of buffered data which could not be sent to the NDO2DB daemon
# before Nagios shuts down.  Prior to shutting down, the NDO NEB module
# will write all buffered data to this file for later processing.  When
# Nagios (re)starts, the NDO NEB module will read the contents of this
# file and send it to the NDO2DB daemon for processing.

buffer_file=/usr/local/nagios/var/ndomod.tmp



# FILE ROTATION INTERVAL
# This option determines how often (in seconds) the output file is
# rotated by Nagios.  File rotation is handled by Nagios by executing
# the command defined by the file_rotation_command option.  This
# option has no effect if the output_type option is a socket.

file_rotation_interval=14400



# FILE ROTATION COMMAND
# This option specified the command (as defined in Nagios) that is
# used to rotate the output file at the interval specified by the
# file_rotation_interval option.  This option has no effect if the
# output_type option is a socket.
#
# See the file 'misccommands.cfg' for an example command definition
# that you can use to rotate the log file.

#file_rotation_command=rotate_ndo_log



# FILE ROTATION TIMEOUT
# This option specified the maximum number of seconds that the file
# rotation command should be allowed to run before being prematurely
# terminated.

file_rotation_timeout=60



# RECONNECT INTERVAL
# This option determines how often (in seconds) that the NDO NEB
# module will attempt to re-connect to the output file or socket if
# a connection to it is lost.

reconnect_interval=15



# RECONNECT WARNING INTERVAL
# This option determines how often (in seconds) a warning message will
# be logged to the Nagios log file if a connection to the output file
# or socket cannot be re-established.

#reconnect_warning_interval=15
reconnect_warning_interval=900



# DATA PROCESSING OPTION
# This option determines what data the NDO NEB module will process. 
# Do not mess with this option unless you know what you're doing!!!!
# Read the source code (include/ndbxtmod.h) to determine what values
# to use here.  Values from source code should be OR'ed to get the
# value to use here.  A value of -1 will cause all data to be processed.
# Read the source code (include/ndomod.h) and look for "NDOMOD_PROCESS_"
# to determine what values to use here.  Values from source code should
# be OR'ed to get the value to use here.  A value of -1 will cause all
# data to be processed. 

# Process everything
#data_processing_options=-1

#no timed event, no host check, no service check
data_processing_options=67108669


# CONFIG OUTPUT OPTION
# This option determines what types of configuration data the NDO
# NEB module will dump from Nagios.  Values can be OR'ed together.
# Values: 
# 	  0 = Don't dump any configuration information
#         1 = Dump only original config (from config files)
#         2 = Dump config only after retained information has been restored
#         3 = Dump both original and retained configuration

config_output_options=2

[kendallchenoweth]

I changed the output_type from unixsocket to tcpsocket and, for now, the problem seems to have gone away. I will do more testing. Because we're using a remote database does it make more sense to use TCP socket?

While this solves the problem of ndo2db not being required after apply changes, it breaks something else because the Nagios XI status is out of sync with the nagios core status. I've backed out this change and am still having the same problem.

[tgriep]

Could you post the contents of the ndo2db.cfg file?
It can be found here.

Code: Select all

/usr/local/nagios/etc/ndo2db.cfg

[kendallchenoweth]

Here it is... I didn't list the actual host name and password below but I confirmed that I can log onto the database okay.

Code: Select all

#####################################################################
# NDO2DB DAEMON CONFIG FILE
#####################################################################


lock_file=/usr/local/nagios/var/ndo2db.lock

ndo2db_user=nagios
ndo2db_group=nagios

socket_type=unix

socket_name=/usr/local/nagios/var/ndo.sock

tcp_port=5668


db_servertype=mysql
db_host=nagiosxi-qa01.abcme34236ni.us-east-3.rds.amazonaws.com
db_port=3306

db_name=nagios
db_prefix=nagios_

db_user=ndoutils
db_pass=something_else



## TABLE TRIMMING OPTIONS
# Several database tables containing Nagios event data can become quite large
# over time.  Most admins will want to trim these tables and keep only a
# certain amount of data in them.  The options below are used to specify the
# age (in MINUTES) that data should be allowd to remain in various tables
# before it is deleted.  Using a value of zero (0) for any value means that
# that particular table should NOT be automatically trimmed.

# Keep timed events for 24 hours
max_timedevents_age=1440

# Keep system commands for 1 week
max_systemcommands_age=10080

# Keep service checks for 1 week
max_servicechecks_age=10080

# Keep host checks for 1 week
max_hostchecks_age=10080

# Keep event handlers for 31 days
max_eventhandlers_age=44640





# DEBUG LEVEL
# This option determines how much (if any) debugging information will
# be written to the debug file.  OR values together to log multiple
# types of information.
# Values: -1 = Everything
#          0 = Nothing
#          1 = Process info
#	   2 = SQL queries

debug_level=1



# DEBUG VERBOSITY
# This option determines how verbose the debug log out will be.
# Values: 0 = Brief output
#         1 = More detailed
#         2 = Very detailed

debug_verbosity=2



# DEBUG FILE
# This option determines where the daemon should write debugging information.

debug_file=/usr/local/nagios/var/ndo2db.debug



# MAX DEBUG FILE SIZE
# This option determines the maximum size (in bytes) of the debug file.  If
# the file grows larger than this size, it will be renamed with a .old
# extension.  If a file already exists with a .old extension it will
# automatically be deleted.  This helps ensure your disk space usage doesn't
# get out of control when debugging.

max_debug_file_size=1000000