Table './nagios/nagios_contactnotifications' is marked as cr

[saptech]

I deleted all the rows in the table as we don't use the values for report etc.

mysql> select count(*) from nagios_contactnotifications;
+----------+
| count(*) |
+----------+
| 0 |
+----------+

but the page still shows the report (monthly , quarterly etc.) even after restarting every nagios process. After 30 min more or less I checked again . The DB shows these errors

mysql> select count(*) from nagios_contactnotifications;
ERROR 144 (HY000): Table './nagios/nagios_contactnotifications' is marked as crashed and last (automatic?) repair failed

and the web page:

SQL: SQL Error [ndoutils] : Table './nagios/nagios_contactnotifications' is marked as crashed and last (automatic?) repair failed SQL: SQL Error [ndoutils] : Table './nagios/nagios_contactnotifications' is marked as crashed and last (automatic?) repair failed

[tgriep]

Could you PM me a new System Profile or open a support ticket and email in the profile to [email protected]?
To send us your system profile. login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and email it back to us.

Also, can you run the following command and email back the output?

Code: Select all

echo 'describe nagios_contactnotifications;' | mysql  -t -pnagiosxi nagios

[saptech]

Hi,

I sent the info as requested via PM, could you please let me know if you received them?

[tgriep]

Thanks, I did receive the profile and the only error that I can see other than the crashes mysql table are some proxy server errors.
If your server is behind a proxy server, following this link below to setup the XI server to use the proxy server.
https://support.nagios.com/kb/article.php?id=147
Try setting that and then repair the mysql database and see if the issue is fixed.

[saptech]

I fixed the proxy configuration, repaired the DB but still after more or less 15 minutes the errors appears again. Attached the new system profile

[tgriep]

Can you run the following command and post the output?

Code: Select all

du -h /var/lib/mysql

Also, post these files as well.

Code: Select all

/usr/local/nagiosxi/html/config.inc.php
/etc/my.cnf

The only thing I can think of is maybe the backup or the antivirus software is not allowing the mysql table to be updated and that is causing the errors.
Take a look in all of the log files in the /var/log folder around the time the error came back and see if you can find any errors that could cause it.

[saptech]

# du -h /var/lib/mysql

Code: Select all

4.0K    /var/lib/mysql/test
1.9M    /var/lib/mysql/nagiosql
53M     /var/lib/mysql/nagios
980K    /var/lib/mysql/mysql
84M     /var/lib/mysql

# cat /usr/local/nagiosxi/html/config.inc.php

Code: Select all

<?php
//
// Copyright (c) 2008-2010 Nagios Enterprises, LLC.  All rights reserved.
//
// $Id: config.inc.php 202 2010-07-13 21:22:57Z egalstad $

// base url
$cfg['base_url'] = "/nagiosxi"; // do not include http(s) or host name - this is the base from "http://localhost"

// base root directory where XI is installed
$cfg['root_dir'] = "/usr/local/nagiosxi";

// directory where scripts are installed
$cfg['script_dir'] = "/usr/local/nagiosxi/scripts";

$cfg['xidpe_dir'] = '/var/nagiosramdisk/spool/xidpe/';
$cfg['perfdata_spool'] = '/var/nagiosramdisk/spool/perfdata/';

// nom checkpoints
$cfg['nom_checkpoints_dir'] = "/usr/local/nagiosxi/nom/checkpoints/nagioscore/";

// force http/https
$cfg['use_https'] = true; // determines whether cron jobs and other scripts will force the use of HTTPS instead of HTTP

// allow for different http port for subsystem calls
$cfg['port_number'] = false;

// default server, db, connection settings
$cfg['dbtype'] = ''; // this setting is no longer used - use settings below
$cfg['dbserver'] = 'localhost'; // this setting is no longer used - use settings below

// db-specific connection information
$cfg['db_info'] = array(
    "nagiosxi" => array(
        "dbtype" => 'pgsql',
        "dbserver" => '',
        "user" => 'nagiosxi',
        "pwd" => 'n@gweb',
        "db" => 'nagiosxi',
        "dbmaint" => array( // variables affecting maintenance of db
            "max_auditlog_age" => 30, // max time (in DAYS) to keep audit log entries
            "max_commands_age" => 480, // max time (minutes) to keep commands
            "max_events_age" => 480, // max time (minutes) to keep events
            "optimize_interval" => 60, // time (in minutes) between db optimization runs
            "repair_interval" => 0, // time (in minutes) between db repair runs
        ),
    ),
    "ndoutils" => array(
        "dbtype" => 'mysql',
        "dbserver" => 'localhost',
        "user" => 'ndoutils',
        "pwd" => 'n@gweb',
        "db" => 'nagios',
        "dbmaint" => array( // variables affecting maintenance of ndoutils db

            "max_externalcommands_age" => 7, // max time (in DAYS) to keep external commands
            "max_logentries_age" => 90, // max time (in DAYS) to keep log entries
            "max_statehistory_age" => 730, // max time (in DAYS) to keep state history information
            "max_notifications_age" => 90, // max time (in DAYS) to keep notifications
            "max_timedevents_age" => 5, // max time (minutes) to keep timed events
            "max_systemcommands_age" => 5, // max time (minutes) to keep system commands
            "max_servicechecks_age" => 5, // max time (minutes) to keep service checks
            "max_hostchecks_age" => 5, // max time (minutes) to keep host checks
            "max_eventhandlers_age" => 5, // max time (minutes) to keep event handlers
            "optimize_interval" => 60, // time (in minutes) between db optimization runs
            "repair_interval" => 0, // time (in minutes) between db repair runs
        ),
    ),
    "nagiosql" => array(
        "dbtype" => 'mysql',
        "dbserver" => 'localhost',
        "user" => 'nagiosql',
        "pwd" => 'n@gweb',
        "db" => 'nagiosql',
        "dbmaint" => array( // variables affecting maintenance of db
            "max_logbook_age" => 480, // max time (minutes) to keep log book records
            "optimize_interval" => 60, // time (in minutes) between db optimization runs
            "repair_interval" => 0, // time (in minutes) between db repair runs
        ),
    ),
);

// db-specific table prefixes
$cfg['db_prefix'] = array(
    "ndoutils" => "nagios_", // prefix for NDOUtils tables
    "nagiosxi" => "xi_", // prefix for XI tables
    "nagiosql" => "tbl_", // prefix for NagiosQL tables
);

// component info
$cfg['component_info'] = array(
    "nagioscore" => array(
        "cgi_dir" => "/usr/local/nagios/sbin",
        "import_dir" => "/usr/local/nagios/etc/import",
        "plugin_dir" => "/usr/local/nagios/libexec",
        "cgi_config_file" => "/usr/local/nagios/etc/cgi.cfg",
        "cmd_file" => "/usr/local/nagios/var/rw/nagios.cmd",
        "log_file" => "/usr/local/nagios/var/nagios.log",
        "nom_checkpoint_interval" => 1440, // time (in minutes) between nom checkpoints
    ),
    "pnp" => array(
        "perfdata_dir" => "/usr/local/nagios/share/perfdata",
        "share_dir" => "/usr/local/nagios/share/pnp",
        "direct_url" => "/nagios/pnp",
        "username" => 'nagiosxi', // don't change this!
        "password" => 'nagiosadmin', // this gets reset when security credentials are reset after installation
    ),
    "perfdata" => array(
        "rrdtool_path" => "/usr/bin/rrdtool",
    ),
    "nagiosql" => array(
        "dir" => "/var/www/html/nagiosql",
        "direct_url" => "/nagiosql",
        "username" => 'nagiosxi', // don't change this!
        "password" => 'n@gweb', // this gets reset when security credentials are reset after installation
    ),
    "nagvis" => array(
        "share_dir" => "/usr/local/nagios/share/nagvis",
        "direct_url" => "/nagios/nagvis",
        "username" => 'nagiosadmin', // don't change this!
        "password" => 'nagiosadmin', // this gets reset when security credentials are reset after installation
    ),
);

$cfg['demo_mode'] = false; // is this in demo mode

$cfg['dashlet_refresh_multiplier'] = 1000; // milliseconds (1 second = 1000)

// REFRESH RATES FOR VARIOUS DASHLETS (IN SECONDS UNLESS THE MULTIPLIER IS CHANGED)
$cfg['dashlet_refresh_rates'] = array(
    "available_updates" => 24 * 60 * 60, // 24 hours
    "systat_eventqueuechart" => 5,
    "sysstat_monitoringstats" => 30,
    "systat_monitoringperf" => 30,
    "sysstat_monitoringproc" => 30,
    "perfdata_chart" => 60, // performance graphs
    "network_outages" => 30,
    "host_status_summary" => 60,
    "service_status_summary" => 60,
    "hostgroup_status_overview" => 60,
    "hostgroup_status_grid" => 60,
    "servicegroup_status_overview" => 60,
    "servicegroup_status_grid" => 60,
    "hostgroup_status_summary" => 60,
    "servicegroup_status_summary" => 60,
    "sysstat_componentstates" => 7,
    "sysstat_serverstats" => 5,
    "network_outages_summary" => 30,
    "network_health" => 30,
    "host_status_tac_summary" => 30,
    "service_status_tac_summary" => 30,
    "feature_status_tac_summary" => 30,
    "admin_tasks" => 60,
    "getting_started" => 60,
    "pagetop_alert_content" => 30, // not a dashlet yet, sits in page header
    "tray_alert" => 30, // sites in page footer
);


// MEMCACHED SETUP
$cfg['memcached_enable'] = false; // should we use memcached or not?
$cfg['memcached_hosts'] = array('127.0.0.1', '192.168.1.3'); // one or more memcached servers
$cfg['memcached_port'] = 11211; // default memcached port
$cfg['memcached_compress'] = false; // use true to store items compressed
$cfg['memcached_ttl'] = 10; // max number of seconds data (from SELECT statements) should be cached


// HTTP BASIC AUTHENTICATION INFO -- USED BY SUBSYSTEM
$cfg['use_basic_authentication'] = false; // is HTTP Basic authentication being used? if so, set the two variables below...
$cfg['subsystem_basic_auth_username'] = 'nagiosxi'; // subsystem credentials
$cfg['subsystem_basic_auth_password'] = 'somepassword';

$cfg['default_language'] = 'en_US'; // default language
$cfg['default_theme'] = ''; // default theme

// available languages
$cfg['languages'] = array(
    "en_US" => "English",
);


/*********   DO NOT MODIFY ANYTHING BELOW THIS LINE   **********/

$cfg['default_instance_id'] = 1; // default ndoutils instance to read from
$cfg['default_result_records'] = 100000; // max number of records to return by default

$cfg['online_help_url'] = "http://support.nagios.com/"; // comment this out to disable online help links
$cfg['feedback_url'] = "http://api.nagios.com/feedback/";
$cfg['privacy_policy_url'] = "http://www.nagios.com/legal/privacypolicy/";

//$cfg['db_version']=101;
$cfg['db_version'] = 113;
//$cfg['product_version']='2009RC1';

$cfg['subsystem_ticket'] = "12345"; // default - this gets reset...

$cfg['htaccess_file'] = "/usr/local/nagiosxi/etc/htpasswd.users";
$cfg['htpasswd_path'] = "/usr/bin/htpasswd";

$cfg['enable_analytics'] = 1;

// Secure notification %responseurl%
// setting this to 1 will force users to login in the response URL
// $cfg['secure_response_url']=1;

// Globally disable in page help system
// $cfg['disable_helpsystem']=1;

///////// keep these in order /////////

// include generic db defs
require_once(dirname(__FILE__) . '/includes/db.inc.php');

// include generic  definitions
require_once(dirname(__FILE__) . '/db/common.inc.php');
// include db-specific definitions
//require_once(dirname(__FILE__).'/db/'.$cfg['dbtype'].'.inc.php');

# cat /etc/my.cnf
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock

# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

# Settings user and group are ignored when systemd is used (fedora >= 15).
# If you need to run mysqld under a different user or group,
# customize your systemd unit file for mysqld according to the
# instructions in http://fedoraproject.org/wiki/Systemd
user=mysql

# Semisynchronous Replication
# http://dev.mysql.com/doc/refman/5.5/en/replication-semisync.html
# uncomment next line on MASTER
;plugin-load=rpl_semi_sync_master=semisync_master.so
# uncomment next line on SLAVE
;plugin-load=rpl_semi_sync_slave=semisync_slave.so

# Others options for Semisynchronous Replication
;rpl_semi_sync_master_enabled=1
;rpl_semi_sync_master_timeout=10
;rpl_semi_sync_slave_enabled=1

# http://dev.mysql.com/doc/refman/5.5/en/performance-schema.html
;performance_schema


[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid


#
# include all files from the config directory
#
!includedir /etc/my.cnf.d

[tgriep]

Can you post these log files from the XI server so we can view them?

Code: Select all

/var/log/mysqld.log
/var/log/messages
/var/lib/pgsql/data/pg_log/postgresql-Fri.log

I need to see the full logs as there may be more information that could help out on the issue.

[saptech]

here attached

[tgriep]

It looks like in the past, the MYSQL database version was upgraded but the Native SQL tables didn't get updated.
Login as root, run the following to upgrade those tables. First, we will stop the processed that access the databases.

Code: Select all

service crond stop
service nagios stop
service ndo2db stop

Now, run this to upgrade the native SQL tables

Code: Select all

mysql_upgrade -u root -pnagiosxi

Then run this to repair the databases

Code: Select all

cd /usr/local/nagiosxi/scripts
./repair_databases.sh

Then run this to optimize the database
Optimize Databases

Code: Select all

mysqlcheck -o -u root -pnagiosxi --databases nagios

Then run the repair again.

Code: Select all

./repair_databases.sh

Now you can start up the processes.

Code: Select all

service ndo2db start
service nagios start
service crond start

Try this and post here is it fixed the SQL table for you.