Error processing config file '/usr/local/nagios/etc/ndo2db.c

[nyoung]

My nagiosxi 2014R2.5 vm ran out of disk sapce. I was able to extend the LVM but I still cant start the Database Backend.

I tried to run database_repair.sh but it returns an error-

Code: Select all

===============
REPAIR COMPLETE
===============
ndo2db was not running... could not stop
Starting ndo2db:Error processing config file '/usr/local/nagios/etc/ndo2db.cfg'.
 done.
Running configuration check...done.
Stopping nagios: .done.
Starting nagios: done.

I see that ndo2db.cfg is empty. I assume this is not correct.

Why did this happen?
How can I resolve this?

[jdalrymple]

Do you have any backups on your system, or off?

They'd be under /store/backups/nagiosxi

If you have a backup, it will have your ndo2db.cfg.

How it happened - I'd guess your system did an fsck and "repaired" that file? That's just my best guess.

[ssax]

Here is the default from 2014R2.5.

ndo2db.zip

[abrist]

Did you run out of disk space?

Code: Select all

df -h
df -i

You will need to repopulate the file, though I am weary that other configs may have issues:

Code: Select all

#####################################################################
# NDO2DB DAEMON CONFIG FILE
#####################################################################


lock_file=/usr/local/nagios/var/ndo2db.lock

ndo2db_user=nagios
ndo2db_group=nagios

socket_type=unix

socket_name=/usr/local/nagios/var/ndo.sock

tcp_port=5668


db_servertype=mysql
db_host=localhost
db_port=3306

db_name=nagios
db_prefix=nagios_

db_user=ndoutils
db_pass=n@gweb



## TABLE TRIMMING OPTIONS
# Several database tables containing Nagios event data can become quite large
# over time.  Most admins will want to trim these tables and keep only a
# certain amount of data in them.  The options below are used to specify the
# age (in MINUTES) that data should be allowd to remain in various tables
# before it is deleted.  Using a value of zero (0) for any value means that
# that particular table should NOT be automatically trimmed.

# Keep timed events for 24 hours
max_timedevents_age=1440

# Keep system commands for 1 week
max_systemcommands_age=10080

# Keep service checks for 1 week
max_servicechecks_age=10080

# Keep host checks for 1 week
max_hostchecks_age=10080

# Keep event handlers for 31 days
max_eventhandlers_age=44640





# DEBUG LEVEL
# This option determines how much (if any) debugging information will
# be written to the debug file.  OR values together to log multiple
# types of information.
# Values: -1 = Everything
#          0 = Nothing
#          1 = Process info
#          2 = SQL queries

debug_level=0



# DEBUG VERBOSITY
# This option determines how verbose the debug log out will be.
# Values: 0 = Brief output
#         1 = More detailed
#         2 = Very detailed

debug_verbosity=1



# DEBUG FILE
# This option determines where the daemon should write debugging information.

debug_file=/usr/local/nagios/var/ndo2db.debug



# MAX DEBUG FILE SIZE
# This option determines the maximum size (in bytes) of the debug file.  If
# the file grows larger than this size, it will be renamed with a .old
# extension.  If a file already exists with a .old extension it will
# automatically be deleted.  This helps ensure your disk space usage doesn't
# get out of control when debugging.

max_debug_file_size=1000000

[nyoung]

thanks for the quick reply guys. I'll try to repopulate the file.

Yes, I ran out of disk space. But that should be resolved now-

Code: Select all

# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
                       96G  7.6G   84G   9% /
tmpfs                 939M     0  939M   0% /dev/shm
/dev/sda1             477M   66M  386M  15% /boot

Is there a way I can validate the integrity of other config files?

[jdalrymple]

We don't really have anything that can validate, especially since we don't have anything to validate against.

I would try to restart the Nagios process now, and if it fails I recommend looking towards that backup directory to see if you have a fairly recent copy you can restore.

[ssax]

You could always perform an upgrade to the latest version or same version, you would want to take backups/snapshots first in case of issues.

Example:

Code: Select all

rm -rf /tmp/nagiosxi
cd /tmp
wget http://assets.nagios.com/downloads/nagiosxi/2014/xi-2014r2.7.tar.gz
tar zxf xi-2014r2.7.tar.gz
cd nagiosxi
./upgrade

[nyoung]

thanks everyone.
After I repopulated the file, it looked ok

Code: Select all

===============
REPAIR COMPLETE
===============
Stopping ndo2db: head: cannot open `/usr/local/nagios/var/ndo2db.lock' for reading: No such file or directory
done.
Starting ndo2db: done.
Running configuration check...done.
Stopping nagios: .done.
Starting nagios: done.

However I was unable to see any host/service details. It showed this error.

Code: Select all

SQL: SQL Error [ndoutils] : Table 'nagios_programstatus' is marked as crashed and should be repaired SQL: SQL Error [ndoutils] : Table 'nagios_programstatus' is marked as crashed and should be repaired SQL: SQL Error [ndoutils] : Table 'nagios_programstatus' is marked as crashed and should be repaired SQL: SQL Error [ndoutils] : Table 'nagios_services' is marked as crashed and should be repaired SQL: SQL Error [ndoutils] : Table 'nagios_programstatus' is marked as crashed and should be repaired SQL: SQL Error [ndoutils] : Table 'nagios_services' is marked as crashed and should be repaired 

So then I

Code: Select all

service mysqld stop

Code: Select all

./repairmysql.sh nagios

Code: Select all

service mysqld start

and it looks ok now.
Which is the correct script- repair_databases.sh or repairmysql.sh?

Will performing an upgrade make my configurations correct again?

[jdalrymple]

repairmysql.sh is a portion of the repair_databases.sh script. repair_databases.sh is the one you should run.

The upgrade process will not repair any site-specific data. The problem you encountered is very common, may people deploy our virtual appliance without taking disk space into consideration. The confusing part is why the alerts that it sends out to the default contact (nagiosadmin) are always ignored or lost

Sounds like you're in good shape now though and we can lock this topic?

[nyoung]

yes, please lock it. Thanks.

If I ran the ./upgrade successfully, would that "restore" all my corrupted system files? I noticed that most of my service cfg files are zero byte. I will recover those from a backup.