Nagios maria db failed to start - Console not coming up

[deek]

Hi,

One of our nagios console is not coming up and showing that the server "try running repair_database.sh script.

[nagios@servexxxx ~]$ sudo /usr/local/nagiosxi/scripts/manage_services.sh start mysqld
Job for mariadb.service failed because the control process exited with error code. See "systemctl status mariadb.service" and "journalctl -xe" for details.

Attached some log file for reference

Thanks
Deepan.

[tgriep]

What do you see if you run these commands?

Code: Select all

systemctl status mariadb.service
journalctl -xe

Post the full output.
Also, run these as root and post the output as well.

Code: Select all

chage -l nagios
df -h
df -i

Thanks

[amane]

Please find below output for given command

[nagios@servexxxx ~]$ systemctl status mariadb.service
? mariadb.service - MariaDB database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Tue 2019-10-15 17:21:22 EDT; 7h ago
Process: 2967 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID (code=exited, status=1/FAILURE)
Process: 2966 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, status=0/SUCCESS)
Process: 2931 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n (code=exited, status=0/SUCCESS)
Main PID: 2966 (code=exited, status=0/SUCCESS)


[nagios@servexxxx ~]$ journalctl -xe
Hint: You are currently not seeing messages from other users and the system.
Users in the 'systemd-journal' group can see all messages. Pass -q to
turn off this notice.
No journal files were opened due to insufficient permissions.


[nagios@servexxxx ~]$ chage -l nagios
Last password change : Mar 15, 2018
Password expires : never
Password inactive : never
Account expires : never
Minimum number of days between password change : 7
Maximum number of days between password change : 99999
Number of days of warning before password expires : 7

[nagios@servexxxx ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 9.8G 0 9.8G 0% /dev
tmpfs 9.8G 0 9.8G 0% /dev/shm
tmpfs 9.8G 745M 9.0G 8% /run
tmpfs 9.8G 0 9.8G 0% /sys/fs/cgroup
/dev/mapper/systemvg-rootlv 2.0G 61M 1.8G 4% /
/dev/mapper/systemvg-usrlv 7.8G 5.5G 2.0G 75% /usr
/dev/sda1 488M 180M 273M 40% /boot
/dev/mapper/systemvg-optlv 2.0G 157M 1.7G 9% /opt
/dev/mapper/appvg-nagios_store_lv 50G 9.1G 40G 19% /store
/dev/mapper/systemvg-cvlv 2.5G 7.9M 2.4G 1% /opt/commvault
/dev/mapper/systemvg-homelv 2.0G 8.9M 1.8G 1% /home
/dev/mapper/appvg-bmclv 9.8G 37M 9.2G 1% /opt/bmc
/dev/mapper/systemvg-tmplv 2.0G 9.5M 1.8G 1% /tmp
/dev/mapper/systemvg-locallv 2.4G 1.1G 1.3G 47% /usr/local
/dev/mapper/systemvg-varlv 2.1G 1.4G 628M 69% /var
/dev/mapper/systemvg-tripwirelv 1.1G 318M 689M 32% /usr/local/tripwire
/dev/mapper/systemvg-varloglv 2.0G 577M 1.3G 31% /var/log
/dev/mapper/appvg-nagios_mysql_lv 50G 2.7G 47G 6% /var/lib/mysql
/dev/mapper/systemvg-simpanalv 976M 2.6M 907M 1% /var/log/simpana
/dev/mapper/systemvg-auditlv 278M 132M 126M 52% /var/log/audit
/dev/mapper/appvg-nagioslv 25G 6.3G 19G 26% /usr/local/nagios
tmpfs 2.0G 0 2.0G 0% /run/user/9588

[nagios@servexxxx ~]$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
devtmpfs 2546360 503 2545857 1% /dev
tmpfs 2549253 1 2549252 1% /dev/shm
tmpfs 2549253 866 2548387 1% /run
tmpfs 2549253 16 2549237 1% /sys/fs/cgroup
/dev/mapper/systemvg-rootlv 131072 3258 127814 3% /
/dev/mapper/systemvg-usrlv 524288 100846 423442 20% /usr
/dev/sda1 32768 355 32413 2% /boot
/dev/mapper/systemvg-optlv 131072 8785 122287 7% /opt
/dev/mapper/appvg-nagios_store_lv 3276800 272 3276528 1% /store
/dev/mapper/systemvg-cvlv 170688 11 170677 1% /opt/commvault
/dev/mapper/systemvg-homelv 131072 491 130581 1% /home
/dev/mapper/appvg-bmclv 655360 11 655349 1% /opt/bmc
/dev/mapper/systemvg-tmplv 131072 727 130345 1% /tmp
/dev/mapper/systemvg-locallv 163840 19448 144392 12% /usr/local
/dev/mapper/systemvg-varlv 147456 5109 142347 4% /var
/dev/mapper/systemvg-tripwirelv 72288 1404 70884 2% /usr/local/tripwire
/dev/mapper/systemvg-varloglv 139264 205 139059 1% /var/log
/dev/mapper/appvg-nagios_mysql_lv 3276800 604 3276196 1% /var/lib/mysql
/dev/mapper/systemvg-simpanalv 65536 11 65525 1% /var/log/simpana
/dev/mapper/systemvg-auditlv 24576 17 24559 1% /var/log/audit
/dev/mapper/appvg-nagioslv 1638400 22521 1615879 2% /usr/local/nagios
tmpfs 2549253 1 2549252 1% /run/user/9588

[tgriep]

There is a ticket that was opened for this issue so do you want to continue to troubleshoot the issue here or in the ticket?