I have an issue : all is empty in Nagios XI home dashboard.
0 hosts, 0 services... (see capture)
I have made repair_databases script (ok), checked that tables from mysql "nagios" (ndo) database are not empty...
ndo2db is running and populating tables.
[root@xxxxxx ~]# service nagios stop
Stopping nagios (via systemctl): [ OK ]
[root@xxxxxx ~]# service ndo2db stop
Stopping ndo2db (via systemctl): [ OK ]
[root@xxxxxx ~]# mv /usr/local/nagios/var/ndo2db.lock /usr/local/nagios/var/ndo2db.lock.bak
mv: impossible d'évaluer « /usr/local/nagios/var/ndo2db.lock »: Aucun fichier ou dossier de ce type (no such file in french)
[root@xxxxxx ~]# mv /usr/local/nagios/var/ndo.sock /usr/local/nagios/var/ndo.sock.bak
mv: impossible d'évaluer « /usr/local/nagios/var/ndo.sock »: Aucun fichier ou dossier de ce type
[root@xxxxxx ~]# mv /usr/local/nagios/var/ndo.sock /usr/local/nagios/var/ndo.sock.bak
mv: impossible d'évaluer « /usr/local/nagios/var/ndo.sock »: Aucun fichier ou dossier de ce type
[root@xxxxxx ~]# service ndo2db start
Starting ndo2db (via systemctl): [ OK ]
[root@xxxxxx ~]# service nagios start
Starting nagios (via systemctl): [ OK ]
Does not work : 0 service 0 host displayed. (but thruk... are ok)
Tried to purge all tables (it s our dev environment) :
set foreign_key_checks=0; delete from nagios.nagios_acknowledgements;
set foreign_key_checks=0; delete from nagios.nagios_commands;
set foreign_key_checks=0; delete from nagios.nagios_commenthistory;
set foreign_key_checks=0; delete from nagios.nagios_comments;
set foreign_key_checks=0; delete from nagios.nagios_configfiles;...
set foreign_key_checks=0; delete from nagios.nagios_configfilevariables;
...
Tables are well populated after the restart of nagios+ndo2db, but XI is still displaying nothing.
[1540854000] CURRENT SERVICE STATE: localhost;Service Status - ndo2db;WARNING;HARD;4;[Nous espérons que vous avez reçu de votre administrateur système local les consignes traditionnelles. Généralement, elles se concentrent sur ces trois éléments :
[1540854907] SERVICE NOTIFICATION: nagiosadmin;localhost;Service Status - ndo2db;WARNING;notify_service_by_email;[Nous espérons que vous avez reçu de votre administrateur système local les consignes traditionnelles. Généralement, elles se concentrent sur ces trois éléments :
[1540858597] SERVICE NOTIFICATION: nagiosadmin;localhost;Service Status - ndo2db;WARNING;notify_service_by_email;[Nous espérons que vous avez reçu de votre administrateur système local les consignes traditionnelles. Généralement, elles se concentrent sur ces trois éléments :
[1540862277] SERVICE NOTIFICATION: nagiosadmin;localhost;Service Status - ndo2db;WARNING;notify_service_by_email;[Nous espérons que vous avez reçu de votre administrateur système local les consignes traditionnelles. Généralement, elles se concentrent sur ces trois éléments :
[1540865937] SERVICE NOTIFICATION: nagiosadmin;localhost;Service Status - ndo2db;WARNING;notify_service_by_email;[Nous espérons que vous avez reçu de votre administrateur système local les consignes traditionnelles. Généralement, elles se concentrent sur ces trois éléments :
[1540869627] SERVICE NOTIFICATION: nagiosadmin;localhost;Service Status - ndo2db;WARNING;notify_service_by_email;[Nous espérons que vous avez reçu de votre administrateur système local les consignes traditionnelles. Généralement, elles se concentrent sur ces trois éléments :
[1540873297] SERVICE NOTIFICATION: nagiosadmin;localhost;Service Status - ndo2db;WARNING;notify_service_by_email;[Nous espérons que vous avez reçu de votre administrateur système local les consignes traditionnelles. Généralement, elles se concentrent sur ces trois éléments :
[1540876977] SERVICE NOTIFICATION: nagiosadmin;localhost;Service Status - ndo2db;WARNING;notify_service_by_email;[Nous espérons que vous avez reçu de votre administrateur système local les consignes traditionnelles. Généralement, elles se concentrent sur ces trois éléments :
[1540880647] SERVICE NOTIFICATION: nagiosadmin;localhost;Service Status - ndo2db;WARNING;notify_service_by_email;[Nous espérons que vous avez reçu de votre administrateur système local les consignes traditionnelles. Généralement, elles se concentrent sur ces trois éléments :
[1540884327] SERVICE NOTIFICATION: nagiosadmin;localhost;Service Status - ndo2db;WARNING;notify_service_by_email;[Nous espérons que vous avez reçu de votre administrateur système local les consignes traditionnelles. Généralement, elles se concentrent sur ces trois éléments :
[1540887997] SERVICE NOTIFICATION: nagiosadmin;localhost;Service Status - ndo2db;WARNING;notify_service_by_email;[Nous espérons que vous avez reçu de votre administrateur système local les consignes traditionnelles. Généralement, elles se concentrent sur ces trois éléments :
[1540891667] SERVICE NOTIFICATION: nagiosadmin;localhost;Service Status - ndo2db;WARNING;notify_service_by_email;[Nous espérons que vous avez reçu de votre administrateur système local les consignes traditionnelles. Généralement, elles se concentrent sur ces trois éléments :
[1540895327] SERVICE NOTIFICATION: nagiosadmin;localhost;Service Status - ndo2db;WARNING;notify_service_by_email;[Nous espérons que vous avez reçu de votre administrateur système local les consignes traditionnelles. Généralement, elles se concentrent sur ces trois éléments :
[1540898997] SERVICE NOTIFICATION: nagiosadmin;localhost;Service Status - ndo2db;WARNING;notify_service_by_email;[Nous espérons que vous avez reçu de votre administrateur système local les consignes traditionnelles. Généralement, elles se concentrent sur ces trois éléments :
[1540902657] SERVICE NOTIFICATION: nagiosadmin;localhost;Service Status - ndo2db;WARNING;notify_service_by_email;[Nous espérons que vous avez reçu de votre administrateur système local les consignes traditionnelles. Généralement, elles se concentrent sur ces trois éléments :
[1540906317] SERVICE NOTIFICATION: nagiosadmin;localhost;Service Status - ndo2db;WARNING;notify_service_by_email;[Nous espérons que vous avez reçu de votre administrateur système local les consignes traditionnelles. Généralement, elles se concentrent sur ces trois éléments :
[1540909985] SERVICE NOTIFICATION: nagiosadmin;localhost;Service Status - ndo2db;WARNING;notify_service_by_email;[Nous espérons que vous avez reçu de votre administrateur système local les consignes traditionnelles. Généralement, elles se concentrent sur ces trois éléments :
[1540913646] SERVICE NOTIFICATION: nagiosadmin;localhost;Service Status - ndo2db;WARNING;notify_service_by_email;[Nous espérons que vous avez reçu de votre administrateur système local les consignes traditionnelles. Généralement, elles se concentrent sur ces trois éléments :
[1540917306] SERVICE NOTIFICATION: nagiosadmin;localhost;Service Status - ndo2db;WARNING;notify_service_by_email;[Nous espérons que vous avez reçu de votre administrateur système local les consignes traditionnelles. Généralement, elles se concentrent sur ces trois éléments :
For Maria DB : Old issues with space (before our incident) : The DB is up and running, and as i said, NDO tables are well populated. Tables were repaired
[root@infnxddat021 ~]# tail -50 /var/log/mariadb/mariadb.log
2018-08-20 12:04:12 140201291060992 [ERROR] mysqld: Disk full (/tmp/#sql_68b_3.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
2018-08-20 12:04:12 140201291060992 [ERROR] mysqld: Disk full (/tmp/#sql_68b_3.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
2018-08-20 12:06:34 140200824362752 [Warning] mysqld: Retry in 60 secs. Message reprinted in 600 secs
2018-08-20 12:07:21 140200487073536 [ERROR] mysqld: Disk full (/tmp/#sql_68b_3.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
2018-08-20 12:07:21 140200487073536 [ERROR] mysqld: Disk full (/tmp/#sql_68b_3.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
2018-08-20 12:07:21 140200487073536 [ERROR] mysqld: Disk full (/tmp/#sql_68b_3.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
2018-08-20 12:07:21 140200487073536 [ERROR] mysqld: Disk full (/tmp/#sql_68b_3.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
2018-08-20 12:07:21 140200487073536 [ERROR] mysqld: Disk full (/tmp/#sql_68b_3.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
2018-08-20 12:07:21 140200487073536 [ERROR] mysqld: Disk full (/tmp/#sql_68b_3.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
2018-08-20 12:07:21 140200487073536 [ERROR] mysqld: Disk full (/tmp/#sql_68b_3.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
2018-08-20 12:07:21 140200487073536 [ERROR] mysqld: Disk full (/tmp/#sql_68b_3.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
2018-08-20 12:07:21 140200487073536 [ERROR] mysqld: Disk full (/tmp/#sql_68b_3.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
2018-08-20 12:07:21 140200487073536 [ERROR] mysqld: Disk full (/tmp/#sql_68b_3.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
2018-08-20 12:07:21 140200487073536 [ERROR] mysqld: Disk full (/tmp/#sql_68b_3.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
2018-08-20 12:07:58 140198912378624 [ERROR] mysqld: Disk full (/tmp/#sql_68b_3.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
2018-08-20 12:07:58 140198912378624 [ERROR] mysqld: Disk full (/tmp/#sql_68b_3.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
2018-08-20 12:07:58 140198912378624 [ERROR] mysqld: Disk full (/tmp/#sql_68b_3.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
2018-08-20 12:07:58 140198912378624 [ERROR] mysqld: Disk full (/tmp/#sql_68b_3.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
2018-08-20 12:07:58 140198912378624 [ERROR] mysqld: Disk full (/tmp/#sql_68b_3.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
2018-08-20 12:07:58 140198912378624 [ERROR] mysqld: Disk full (/tmp/#sql_68b_3.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
2018-08-20 12:07:58 140198912378624 [ERROR] mysqld: Disk full (/tmp/#sql_68b_3.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
2018-08-20 12:07:58 140198912378624 [ERROR] mysqld: Disk full (/tmp/#sql_68b_3.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
2018-08-20 12:07:58 140198912378624 [ERROR] mysqld: Disk full (/tmp/#sql_68b_3.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
2018-08-20 12:07:58 140198912378624 [ERROR] mysqld: Disk full (/tmp/#sql_68b_3.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
2018-08-20 12:07:58 140198912378624 [ERROR] mysqld: Disk full (/tmp/#sql_68b_3.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
2018-08-20 12:08:34 140198912378624 [ERROR] mysqld: Disk full (/tmp/#sql_68b_3.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
2018-08-20 12:08:34 140198912378624 [ERROR] mysqld: Disk full (/tmp/#sql_68b_3.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
2018-08-20 12:08:34 140198912378624 [ERROR] mysqld: Disk full (/tmp/#sql_68b_3.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
2018-08-20 12:08:34 140198912378624 [ERROR] mysqld: Disk full (/tmp/#sql_68b_3.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
2018-08-20 12:08:34 140198912378624 [ERROR] mysqld: Disk full (/tmp/#sql_68b_3.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
2018-08-20 12:08:34 140198912378624 [ERROR] mysqld: Disk full (/tmp/#sql_68b_3.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
2018-08-20 12:08:34 140198912378624 [ERROR] mysqld: Disk full (/tmp/#sql_68b_3.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
2018-08-20 12:08:34 140198912378624 [ERROR] mysqld: Disk full (/tmp/#sql_68b_3.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
2018-08-20 12:08:34 140198912378624 [ERROR] mysqld: Disk full (/tmp/#sql_68b_3.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
2018-08-20 12:08:34 140198912378624 [ERROR] mysqld: Disk full (/tmp/#sql_68b_3.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
2018-08-20 12:08:34 140198912378624 [ERROR] mysqld: Disk full (/tmp/#sql_68b_3.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
2018-08-20 12:15:15 140198916679424 [Note] /usr/sbin/mysqld (initiated by: unknown): Normal shutdown
2018-08-20 12:15:15 140198916679424 [Note] Event Scheduler: Purging the queue. 0 events
2018-08-20 12:15:15 140200843912960 [Note] InnoDB: FTS optimize thread exiting.
2018-08-20 12:15:35 140198916679424 [Warning] /usr/sbin/mysqld: Forcing close of thread 4842792 user: 'nagios'
2018-08-20 12:15:35 140198916679424 [ERROR] mysqld: Got an error writing communication packets
2018-08-20 12:15:36 140198916679424 [Note] InnoDB: Starting shutdown...
2018-08-20 12:15:36 140200966375168 [Note] InnoDB: Dumping buffer pool(s) to /app/mysql/ib_buffer_pool
2018-08-20 12:15:36 140200966375168 [Note] InnoDB: Instance 0, restricted to 2048 pages due to innodb_buf_pool_dump_pct=25
2018-08-20 12:15:36 140200966375168 [Note] InnoDB: Buffer pool(s) dump completed at 180820 12:15:36
2018-08-20 12:15:38 140198916679424 [Note] InnoDB: Shutdown completed; log sequence number 84456222
2018-08-20 12:15:38 140198916679424 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2018-08-20 12:15:39 140198916679424 [Note] /usr/sbin/mysqld: Shutdown complete
Last edited by sigmainformatique on Tue Oct 30, 2018 12:02 pm, edited 1 time in total.
[INF_S01LX01SA@infnxdsup020 ~]$ cat /usr/local/nagios/etc/ndo2db.cfg
#####################################################################
# NDO2DB DAEMON CONFIG FILE
#####################################################################
lock_file=/usr/local/nagios/var/ndo2db.lock
ndo2db_user=nagios
ndo2db_group=nagios
socket_type=unix
socket_name=/usr/local/nagios/var/ndo.sock
tcp_port=5668
db_servertype=mysql
db_host=172.31.210.50
#db_host=localhost
db_port=3306
db_name=nagios
db_prefix=nagios_
db_user=nagios
db_pass=nagios
## TABLE TRIMMING OPTIONS
# Several database tables containing Nagios event data can become quite large
# over time. Most admins will want to trim these tables and keep only a
# certain amount of data in them. The options below are used to specify the
# age (in MINUTES) that data should be allowd to remain in various tables
# before it is deleted. Using a value of zero (0) for any value means that
# that particular table should NOT be automatically trimmed.
# Keep timed events for 24 hours
max_timedevents_age=1440
# Keep system commands for 1 week
max_systemcommands_age=10080
# Keep service checks for 1 week
max_servicechecks_age=10080
# Keep host checks for 1 week
max_hostchecks_age=10080
# Keep event handlers for 31 days
max_eventhandlers_age=44640
# DEBUG LEVEL
# This option determines how much (if any) debugging information will
# be written to the debug file. OR values together to log multiple
# types of information.
# Values: -1 = Everything
# 0 = Nothing
# 1 = Process info
# 2 = SQL queries
debug_level=0
# DEBUG VERBOSITY
# This option determines how verbose the debug log out will be.
# Values: 0 = Brief output
# 1 = More detailed
# 2 = Very detailed
debug_verbosity=2
# DEBUG FILE
# This option determines where the daemon should write debugging information.
debug_file=/usr/local/nagios/var/ndo2db.debug
# MAX DEBUG FILE SIZE
# This option determines the maximum size (in bytes) of the debug file. If
# the file grows larger than this size, it will be renamed with a .old
# extension. If a file already exists with a .old extension it will
# automatically be deleted. This helps ensure your disk space usage doesn't
# get out of control when debugging.
max_debug_file_size=1000000
Last edited by sigmainformatique on Tue Oct 30, 2018 12:17 pm, edited 1 time in total.
[INF_S01LX01SA@infnxdsup020 ~]$ cat /usr/local/nagios/etc/ndomod.cfg
#####################################################################
# NDOMOD CONFIG FILE
#####################################################################
# INSTANCE NAME
# This option identifies the "name" associated with this particular
# instance of Nagios and is used to seperate data coming from multiple
# instances. Defaults to 'default' (without quotes).
instance_name=localhost
# OUTPUT TYPE
# This option determines what type of output sink the NDO NEB module
# should use for data output. Valid options include:
# file = standard text file
# tcpsocket = TCP socket
# unixsocket = UNIX domain socket (default)
#output_type=file
#output_type=tcpsocket
output_type=unixsocket
# OUTPUT
# This option determines the name and path of the file or UNIX domain
# socket to which output will be sent if the output type option specified
# above is "file" or "unixsocket", respectively. If the output type
# option is "tcpsocket", this option is used to specify the IP address
# of fully qualified domain name of the host that the module should
# connect to for sending output.
#output=/usr/local/nagios/var/ndo.dat
#output=127.0.0.1
output=/usr/local/nagios/var/ndo.sock
# TCP PORT
# This option determines what port the module will connect to in
# order to send output. This option is only vlaid if the output type
# option specified above is "tcpsocket".
tcp_port=5668
# OUTPUT BUFFER
# This option determines the size of the output buffer, which will help
# prevent data from getting lost if there is a temporary disconnect from
# the data sink. The number of items specified here is the number of
# lines (each of variable size) of output that will be buffered.
output_buffer_items=5000
# BUFFER FILE
# This option is used to specify a file which will be used to store the
# contents of buffered data which could not be sent to the NDO2DB daemon
# before Nagios shuts down. Prior to shutting down, the NDO NEB module
# will write all buffered data to this file for later processing. When
# Nagios (re)starts, the NDO NEB module will read the contents of this
# file and send it to the NDO2DB daemon for processing.
buffer_file=/usr/local/nagios/var/ndomod.tmp
# FILE ROTATION INTERVAL
# This option determines how often (in seconds) the output file is
# rotated by Nagios. File rotation is handled by Nagios by executing
# the command defined by the file_rotation_command option. This
# option has no effect if the output_type option is a socket.
file_rotation_interval=14400
# FILE ROTATION COMMAND
# This option specified the command (as defined in Nagios) that is
# used to rotate the output file at the interval specified by the
# file_rotation_interval option. This option has no effect if the
# output_type option is a socket.
#
# See the file 'misccommands.cfg' for an example command definition
# that you can use to rotate the log file.
#file_rotation_command=rotate_ndo_log
# FILE ROTATION TIMEOUT
# This option specified the maximum number of seconds that the file
# rotation command should be allowed to run before being prematurely
# terminated.
file_rotation_timeout=60
# RECONNECT INTERVAL
# This option determines how often (in seconds) that the NDO NEB
# module will attempt to re-connect to the output file or socket if
# a connection to it is lost.
reconnect_interval=15
# RECONNECT WARNING INTERVAL
# This option determines how often (in seconds) a warning message will
# be logged to the Nagios log file if a connection to the output file
# or socket cannot be re-established.
#reconnect_warning_interval=15
reconnect_warning_interval=900
# DATA PROCESSING OPTION
# This option determines what data the NDO NEB module will process.
# Do not mess with this option unless you know what you're doing!!!!
# Read the source code (include/ndbxtmod.h) to determine what values
# to use here. Values from source code should be OR'ed to get the
# value to use here. A value of -1 will cause all data to be processed.
# Read the source code (include/ndomod.h) and look for "NDOMOD_PROCESS_"
# to determine what values to use here. Values from source code should
# be OR'ed to get the value to use here. A value of -1 will cause all
# data to be processed.
# Process everything
#data_processing_options=-1
#no timed event, no host check, no service check
data_processing_options=67108669
# CONFIG OUTPUT OPTION
# This option determines what types of configuration data the NDO
# NEB module will dump from Nagios. Values can be OR'ed together.
# Values:
# 0 = Don't dump any configuration information
# 1 = Dump only original config (from config files)
# 2 = Dump config only after retained information has been restored
# 3 = Dump both original and retained configuration
config_output_options=2
I didn't see any issues with configs/logs with the exception of the "No space left on device" error, but you said it was an old issue, so I would ignore it.
Have you tried restarting nagios and ndo2db to see if this is going to fix the issue?