Nagios Support Forum

Hi,

One of our nagios console is not coming up and showing that the server "try running repair_database.sh script.

[nagios@servexxxx ~]$ sudo /usr/local/nagiosxi/scripts/manage_services.sh start mysqld
Job for mariadb.service failed because the control process exited with error code. See "systemctl status mariadb.service" and "journalctl -xe" for details.

Attached some log file for reference

Thanks
Deepan.

What do you see if you run these commands? Post the full output.
Also, run these as root and post the output as well. Thanks

Please find below output for given command

[nagios@servexxxx ~]$ systemctl status mariadb.service
? mariadb.service - MariaDB database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Tue 2019-10-15 17:21:22 EDT; 7h ago
Process: 2967 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID (code=exited, status=1/FAILURE)
Process: 2966 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, status=0/SUCCESS)
Process: 2931 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n (code=exited, status=0/SUCCESS)
Main PID: 2966 (code=exited, status=0/SUCCESS)


[nagios@servexxxx ~]$ journalctl -xe
Hint: You are currently not seeing messages from other users and the system.
Users in the 'systemd-journal' group can see all messages. Pass -q to
turn off this notice.
No journal files were opened due to insufficient permissions.


[nagios@servexxxx ~]$ chage -l nagios
Last password change : Mar 15, 2018
Password expires : never
Password inactive : never
Account expires : never
Minimum number of days between password change : 7
Maximum number of days between password change : 99999
Number of days of warning before password expires : 7

[nagios@servexxxx ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 9.8G 0 9.8G 0% /dev
tmpfs 9.8G 0 9.8G 0% /dev/shm
tmpfs 9.8G 745M 9.0G 8% /run
tmpfs 9.8G 0 9.8G 0% /sys/fs/cgroup
/dev/mapper/systemvg-rootlv 2.0G 61M 1.8G 4% /
/dev/mapper/systemvg-usrlv 7.8G 5.5G 2.0G 75% /usr
/dev/sda1 488M 180M 273M 40% /boot
/dev/mapper/systemvg-optlv 2.0G 157M 1.7G 9% /opt
/dev/mapper/appvg-nagios_store_lv 50G 9.1G 40G 19% /store
/dev/mapper/systemvg-cvlv 2.5G 7.9M 2.4G 1% /opt/commvault
/dev/mapper/systemvg-homelv 2.0G 8.9M 1.8G 1% /home
/dev/mapper/appvg-bmclv 9.8G 37M 9.2G 1% /opt/bmc
/dev/mapper/systemvg-tmplv 2.0G 9.5M 1.8G 1% /tmp
/dev/mapper/systemvg-locallv 2.4G 1.1G 1.3G 47% /usr/local
/dev/mapper/systemvg-varlv 2.1G 1.4G 628M 69% /var
/dev/mapper/systemvg-tripwirelv 1.1G 318M 689M 32% /usr/local/tripwire
/dev/mapper/systemvg-varloglv 2.0G 577M 1.3G 31% /var/log
/dev/mapper/appvg-nagios_mysql_lv 50G 2.7G 47G 6% /var/lib/mysql
/dev/mapper/systemvg-simpanalv 976M 2.6M 907M 1% /var/log/simpana
/dev/mapper/systemvg-auditlv 278M 132M 126M 52% /var/log/audit
/dev/mapper/appvg-nagioslv 25G 6.3G 19G 26% /usr/local/nagios
tmpfs 2.0G 0 2.0G 0% /run/user/9588

[nagios@servexxxx ~]$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
devtmpfs 2546360 503 2545857 1% /dev
tmpfs 2549253 1 2549252 1% /dev/shm
tmpfs 2549253 866 2548387 1% /run
tmpfs 2549253 16 2549237 1% /sys/fs/cgroup
/dev/mapper/systemvg-rootlv 131072 3258 127814 3% /
/dev/mapper/systemvg-usrlv 524288 100846 423442 20% /usr
/dev/sda1 32768 355 32413 2% /boot
/dev/mapper/systemvg-optlv 131072 8785 122287 7% /opt
/dev/mapper/appvg-nagios_store_lv 3276800 272 3276528 1% /store
/dev/mapper/systemvg-cvlv 170688 11 170677 1% /opt/commvault
/dev/mapper/systemvg-homelv 131072 491 130581 1% /home
/dev/mapper/appvg-bmclv 655360 11 655349 1% /opt/bmc
/dev/mapper/systemvg-tmplv 131072 727 130345 1% /tmp
/dev/mapper/systemvg-locallv 163840 19448 144392 12% /usr/local
/dev/mapper/systemvg-varlv 147456 5109 142347 4% /var
/dev/mapper/systemvg-tripwirelv 72288 1404 70884 2% /usr/local/tripwire
/dev/mapper/systemvg-varloglv 139264 205 139059 1% /var/log
/dev/mapper/appvg-nagios_mysql_lv 3276800 604 3276196 1% /var/lib/mysql
/dev/mapper/systemvg-simpanalv 65536 11 65525 1% /var/log/simpana
/dev/mapper/systemvg-auditlv 24576 17 24559 1% /var/log/audit
/dev/mapper/appvg-nagioslv 1638400 22521 1615879 2% /usr/local/nagios
tmpfs 2549253 1 2549252 1% /run/user/9588

There is a ticket that was opened for this issue so do you want to continue to troubleshoot the issue here or in the ticket?