Nagios 3.5.1 Slow Start

[sebastiaopburnay]

Hi,

I have a new Nagios 3.5.1 host, running on an 64bit Ubuntu Server 12.04 LTS.

This nagios is configured not to do active checks, instead it only accepts passive checks from remote Nagios' servers and it has enabled a ndomod broker module in order to store data on a remote MySQL server.

So, this server is only for
- Receiving passive checks from other Nagios servers (nsca)
- Dealing with notifications (nagios)
- Store data in a third party MySQL Server (ndoutils)

The strange thing is the time that the nagios process takes to get a PID from the time I run 'service nagios start' on the CLI (there are 331 hosts totaling 2277 services).

Below, I show the entries on nagios.log, where you can see by the epoch timestamps that the process took 3 minutes from starting to finishing daemonizing

Code: Select all

[1390317258] Nagios 3.5.0 starting... (PID=9174)
[1390317258] Local time is Tue Jan 21 15:14:18 UTC 2014
[1390317258] LOG VERSION: 2.0
[1390317258] ndomod: NDOMOD 1.5.2 (06-08-2012) Copyright (c) 2009 Nagios Core Development Team and Community Contributors
[1390317258] ndomod: Successfully connected to data sink.  0 queued items to flush.
[1390317258] Event broker module '/usr/local/nagios/bin/ndomod.o' initialized successfully.
[1390317258] Warning: Service 'CPU_load' on host 'MyServer01' has no default contacts or contactgroups defined!
[1390317258] Warning: Service 'Drive_C' on host 'MyServer01' has no default contacts or contactgroups defined!
[1390317258] Warning: Service 'Memory_usage' on host 'MyServer01' has no default contacts or contactgroups defined!
[1390317258] Warning: Service 'SMO_Backoffice' on host 'MyServer01' has no default contacts or contactgroups defined!
[1390317258] Warning: Service 'SMO_Scheduler' on host 'MyServer01' has no default contacts or contactgroups defined!
[1390317258] Warning: Service 'Uptime' on host 'MyServer01' has no default contacts or contactgroups defined!
[1390317258] Warning: Service 'Apache' on host 'MyServer01' has no default contacts or contactgroups defined!
[1390317258] Warning: Service 'HTTP' on host 'MyServer01' has no default contacts or contactgroups defined!
[1390317258] Warning: Service 'PING' on host 'MyServer01' has no default contacts or contactgroups defined!
[1390317258] Warning: Service 'SMO_Broker' on host 'MyServer01' has no default contacts or contactgroups defined!
[1390317258] Warning: Service 'SMO_Server' on host 'MyServer01' has no default contacts or contactgroups defined!
[1390317258] Warning: Service 'PING' on host 'MyServer02' has no default contacts or contactgroups defined!
[1390317258] Warning: Service 'SQL_Port' on host 'MyServer02' has no default contacts or contactgroups defined!
[1390317258] Warning: Service 'PING' on host 'MyServer03' has no default contacts or contactgroups defined!
[1390317258] Warning: Service 'SQL_Port' on host 'MyServer03' has no default contacts or contactgroups defined!
[1390317258] Warning: Service 'Current_Load' on host 'MyServer04' has no default contacts or contactgroups defined!
[1390317258] Warning: Service 'Current_Users' on host 'MyServer04' has no default contacts or contactgroups defined!
[1390317258] Warning: Service 'Root_Partition' on host 'MyServer04' has no default contacts or contactgroups defined!
[1390317258] Warning: Service 'Swap_Usage' on host 'MyServer04' has no default contacts or contactgroups defined!
[1390317258] Warning: Service 'Total_Processes' on host 'MyServer04' has no default contacts or contactgroups defined!
[1390317258] Warning: Service 'HTTP' on host 'MyServer04' has no default contacts or contactgroups defined!
[1390317258] Warning: Service 'PING' on host 'MyServer04' has no default contacts or contactgroups defined!
[1390317258] Warning: Service 'SSH' on host 'MyServer04' has no default contacts or contactgroups defined!
[1390317258] Warning: Host 'MyServer01' has no default contacts or contactgroups defined!
[1390317258] Warning: Host 'MyServer02' has no default contacts or contactgroups defined!
[1390317258] Warning: Host 'MyServer03' has no default contacts or contactgroups defined!
[1390317258] Warning: Host 'MyServer04' has no default contacts or contactgroups defined!
[1390317438] Finished daemonizing... (New PID=9182)

Any suggestions?

Best regards,
Sebastião

[slansing]

Hmm, doesn't appear to be an NDO sync wait issue, is this only happening on this server? It sounds like this is the only one of it's kind that you have. How much additional output is in the log on that same timestamp? It looks like it is being flooded by configuration warnings.

[sebastiaopburnay]

Well, other versions of nagios (3.3.x) could have much more WARNINGs of this nature and still be faster daemonizing themselves... this is quite intriguing

[lmiltchev]

Did you check the mysqld log on the remote server for errors (crashed tables)?

Code: Select all

tail -50 /var/log/mysqld.log

[sebastiaopburnay]

Did you check the mysqld log on the remote server for errors (crashed tables)?

I have it a bit different on my /etc/mysql/my.cf:

Code: Select all

###################################################################
# * Logging and Replication
#################################################################### 
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.          
# As of 5.1 you can enable the log at runtime!
general_log_file        = /var/log/mysql/mysql.log
general_log             = 1

log_error                = /var/log/mysql/error.log

So, whenever I have crashed tables (very typical) I already have a nagios check for grepping crashed tables in the error.log.

I wasn't blaming ndoutils, but thinking of it, this can be due to some entry in ndomod.cfg

Two of them seem specially suspicious:

Code: Select all

# DATA PROCESSING OPTION
# This option determines what data the NDO NEB module will process.
# Do not mess with this option unless you know what you're doing!!!!
# Read the source code (include/ndbxtmod.h) to determine what values
# to use here.  Values from source code should be OR'ed to get the
# value to use here.  A value of -1 will cause all data to be processed.
# Read the source code (include/ndomod.h) and look for "NDOMOD_PROCESS_"
# to determine what values to use here.  Values from source code should
# be OR'ed to get the value to use here.  A value of -1 will cause all
# data to be processed.

data_processing_options=-1

# CONFIG OUTPUT OPTION
# This option determines what types of configuration data the NDO
# NEB module will dump from Nagios.  Values can be OR'ed together.
# Values:
#         0 = Don't dump any configuration information
#         1 = Dump only original config (from config files)
#         2 = Dump config only after retained information has been restored
#         3 = Dump both original and retained configuration

config_output_options=1

[sebastiaopburnay]

Thank you very nuch lmiltchev.

Thanks to your suggestion of problems regarding the database,

I've changed the entry:

Code: Select all

config_output_options=1

to

Code: Select all

config_output_options=2

After that, the 'service nagios restart' command on the CLI only took a few secconds to restart nagios

[tmcdonald]

That seems a lot more appropriate! Is your issue resolved now? If so I will be marking this thread as solved and locking it.

[sebastiaopburnay]

I will be marking this thread as solved and locking it.

Yes, its resolved.

The reason I had that config option was related to another thread I've opened earlyer: http://support.nagios.com/forum/viewtop ... =7&t=22272

By the way, How do I mark it as closed/solved?

[abrist]

Just let us know in the thread that it is resolved and we will lock it with all due haste.