Databse Error

[tgriep]

Either zip the file to make is smaller or try this instead.

Code: Select all

ps -ef --cols=300 >/tmp/info.txt
tail -1000  /var/log/messages | grep ndo2db  >>/tmp/info.txt
tail -500 /usr/local/nagios/var/nagios.log |grep ndo2db >>/tmp/info.txt

[sandeepatil]

Please find attached requested output.

[tgriep]

For some reason, mariadb is not starting and spawning a child copy like my Nagios system.

This if how mariadb runs on my system

Code: Select all

mysql     1966     1  0 Apr26 ?        00:00:00 /bin/sh /usr/bin/mysqld_safe --basedir=/usr
mysql     2523  1966  0 Apr26 ?        03:49:04 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/r
un/mariadb/mariadb.pid --socket=/var/lib/mysql/mysql.sock

This is your system

Code: Select all

mysql     4600     1  0 11:08 ?        00:00:00 /usr/libexec/mysqld --basedir=/usr

Did someone install a different startup script for mariadb on the server?

Edit your /etc/my.cfg file

Below this line

Code: Select all

symbolic-links=0

add these lines

Code: Select all

[mysqld_safe]
log-error=/var/log/mariadb/mariadb.log
pid-file=/var/run/mariadb/mariadb.pid

Save the file and restart mariadb by running

Code: Select all

systemctl restart mariadb.service

The to see if the child process runs, run this as root

Code: Select all

ps -ef |grep mysql

You should see 2 mysql processes running, if not, post this file

Code: Select all

/etc/systemd/system/multi-user.target.wants/mariadb.service

[sandeepatil]

Not able to 2 mysql process running,

Please find requested file,

"/etc/systemd/system/multi-user.target.wants/mariadb.service"

Code: Select all

[root@nodea system]# cat mariadb.service
# It's not recommended to modify this file in-place, because it will be
# overwritten during package upgrades.  If you want to customize, the
# best way is to create a file "/etc/systemd/system/mariadb.service",
# containing
#       .include /usr/lib/systemd/system/mariadb.service
#       ...make your changes here...
# or create a file "/etc/systemd/system/mariadb.service.d/foo.conf",
# which doesn't need to include ".include" call and which will be parsed
# after the file mariadb.service itself is parsed.
#
# For more info about custom unit files, see systemd.unit(5) or
# http://fedoraproject.org/wiki/Systemd#How_do_I_customize_a_unit_file.2F_add_a_custom_unit_file.3F

# For example, if you want to increase mysql's open-files-limit to 10000,
# you need to increase systemd's LimitNOFILE setting, so create a file named
# "/etc/systemd/system/mariadb.service.d/limits.conf" containing:
#       [Service]
#       LimitNOFILE=10000

# Note: /usr/lib/... is recommended in the .include line though /lib/...
# still works.
# Don't forget to reload systemd daemon after you change unit configuration:
# root> systemctl --system daemon-reload

# Use [mysqld.INSTANCENAME] as sections in my.cnf to configure this instance.

[Unit]
Description=MariaDB 10.1 database server
After=syslog.target
After=network.target

[Service]
Type=notify
User=mysql
Group=mysql

ExecStartPre=/usr/libexec/mysql-check-socket
ExecStartPre=/usr/libexec/mysql-prepare-db-dir %n
# MYSQLD_OPTS here is for users to set in /etc/systemd/system/[email protected]/MY_SPECIAL.conf
# Note: we set --basedir to prevent probes that might trigger SELinux alarms,
# per bug #547485
ExecStart=/usr/libexec/mysqld --basedir=/usr $MYSQLD_OPTS $_WSREP_NEW_CLUSTER
ExecStartPost=/usr/libexec/mysql-check-upgrade
ExecStopPost=/usr/libexec/mysql-wait-stop

# Setting this to true can break replication and the Type=notify settings
# See also bind-address mysqld option.
PrivateNetwork=false

KillMode=process
KillSignal=SIGTERM

# Don't want to see an automated SIGKILL ever
SendSIGKILL=no

# Restart crashed server only, on-failure would also restart, for example, when
# my.cnf contains unknown option
Restart=on-abort
RestartSec=5s

UMask=007

# Give a reasonable amount of time for the server to start up/shut down
TimeoutSec=300

# Place temp files in a secure directory, not /tmp
PrivateTmp=true

[Install]
WantedBy=multi-user.target
[root@nodea system]#

[tgriep]

The 2 mysql processes may not be an issue. The install on your server isn't the same as most other systems and it didn't look right to me.
One thing I notice is that the system is running XWindows and Gnome, I recommend disabling then so the system will gain some of the performance back for the other processes.

All I see now it that is looks like the mysql database is either getting randomly restarted or crashing but I do not have enough information as to why that could be happening.

Run the following commands as root and post the /tmp/info.txt file.

Code: Select all

uname -a >/tmp/info.txt
cat /etc/*release* >>/tmp/info.txt
chage -l nagios >>/tmp/info.txt
ipcs -q >>/tmp/info.txt
tail -10 /var/log/mariadb/mariadb.log >>/tmp/info.txt

Can you also, Private Message your System Profile to me?
To PM your system profile. Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and PM it back to me.

[sandeepatil]

Please find requested details,

Code: Select all

Linux nodea.techmahindra.com 3.10.0-327.36.3.el7.x86_64 #1 SMP Mon Oct 24 16:09:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
CentOS Linux release 7.2.1511 (Core)
Derived from Red Hat Enterprise Linux 7.2 (Source)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.2.1511 (Core)
CentOS Linux release 7.2.1511 (Core)
cpe:/o:centos:centos:7
Last password change                                    : Aug 31, 2016
Password expires                                        : never
Password inactive                                       : never
Account expires                                         : never
Minimum number of days between password change          : 0
Maximum number of days between password change          : 99999
Number of days of warning before password expires       : 7

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages
0x25010080 148242432  nagios     600        0            0
0x12010080 148570113  nagios     600        0            0

2017-06-26  4:34:39 140252283750528 [Note] InnoDB: Completed initialization of buffer pool
2017-06-26  4:34:39 140252283750528 [Note] InnoDB: Highest supported file format is Barracuda.
2017-06-26  4:34:40 140252283750528 [Note] InnoDB: 128 rollback segment(s) are active.
2017-06-26  4:34:40 140252283750528 [Note] InnoDB: Waiting for purge to start
2017-06-26  4:34:40 140252283750528 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.31-77.0 started; log sequence number 1680347
2017-06-26  4:34:40 140251485959936 [Note] InnoDB: Dumping buffer pool(s) not yet started
2017-06-26  4:34:40 140252283750528 [Note] Plugin 'FEEDBACK' is disabled.
2017-06-26  4:34:40 140252283750528 [Note] Server socket created on IP: '::'.
2017-06-26  4:34:40 140252283750528 [Note] /usr/libexec/mysqld: ready for connections.
Version: '10.1.17-MariaDB'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Server

Note able to login in Nagios XI GUI, display only below message. Using browser with URL "http://<IP Adddress>/nagiosxi/"

Databse Error
A database connection error has been detected, please follow the repair prompt below. If the issue persists, please contact Nagios support.

Run the following from the CLI as root to attempt to repair the DB:

/usr/local/nagiosxi/scripts/repair_databases.sh

[tgriep]

Run the following as root to repair the database again.

Code: Select all

service nagios stop
service ndo2db stop
service crond stop
service mariadb stop
for i in `ipcs -q | grep nagios |awk '{print $2}'`; do ipcrm -q $i; done
service mariadb start
mysqlcheck -f -r -u root -pnagiosxi --all-databases
service ndo2db start
service nagios start
service crond start

If you receive the database error when you login to the Web interface again and if you have a support contract, can you open a ticket by sending an email in to [email protected] ?

[sandeepatil]

We have restored old backup and now server is up and now able to login in GUI.

Please find attached "System Profile" .zip file and "System Status" screenshot.

Most component showing down and server memory utilization very high, need you help to all component get up and minimize server utilization.

[tgriep]

It looks like the cron daemon it not running right on the server so login to the server as root and run the following to stop and start cron.

Code: Select all

service crond stop
killall -9 crond
service crond start

Give the system 15 minutes to run and see if it start to respond correctly.

[sandeepatil]

Issue resolved after restoring previous backup.

Please close the request.