mysqldump failing after changing cert

[markmerchant]

Desc
mysqldump hanging after changing certificate referenced in ssl.conf. We are unable
to login. System down.

Ver
Nagios XI 5.4.7

OS
uname -a
Linux cvglpnagios01.xx.xxx 3.10.0-693.11.1.el7.x86_64 #1 SMP Fri Oct 27 05:39:05 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux

ps -ef | grep mysq

Code: Select all

    mysql     1442     1  0 Jan21 ?        00:00:00 /bin/sh /usr/bin/mysqld_safe --basedir=/usr
    mysql     1877  1442  0 Jan21 ?        01:57:59 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/mariadb.pid --socket=/var/lib/mysql/mysql.sock
    root      7909  7894  0 Jan28 ?        00:00:00 /bin/bash /root/scripts/automys lbackup
    root      8077  7909  0 Jan28 ?        00:00:00 mysqldump --user=root --password=x xxxxxx --host=localhost --port=3306 --quote-names --opt --databases nagios
    root     10793 10781  0 07:00 ?        00:00:00 /bin/bash /root/scripts/automys lbackup
    root     10866 10793  0 07:00 ?        00:00:00 mysqldump --user=root --password=x xxxxxx --host=localhost --port=3306 --quote-names --opt --databases nagios

[kyang]

Is changing the path to a new certificate the only thing you did? Was the old cert expired?

Anything notable in the mysqld log files? Or mariadb log files?

Code: Select all

tail /var/log/mysqld.log

OR

Code: Select all

tail /var/log/mariadb.log

Better yet, could you send us your profile?

Nagios XI Profile --> On the XI Home Page click "Admin" > "System Profile" --> "Download Profile" button
Save the profile.zip file and upload it here or PM me.

If you receive a PROFILE BUILD FAILED
Please follow this article,

https://support.nagios.com/kb/article.p ... ategory=44

After you PM the profile please update this thread so we know you sent it unless you post the profile on here. Thanks!

[markmerchant]

I was finally able to run the repair_databases script and get the server back
on line. As I recall, we had trouble with the sudoers thing before -- it's not
something I can change.

If my theory is right, my Dev and QA boxes will stop working Sunday
night when their DBs get maintenance. Will post logs then.

[kyang]

Sounds good, let us know!

[markmerchant]

How do I run the DB maintenance manually? A script I can kick off before Sunday? Thx.

[kyang]

Here's the script to run db maint manually.

Code: Select all

/usr/local/nagiosxi/cron/dbmaint.php

[markmerchant]

How could this possibly fail?

Code: Select all

sudo /usr/local/nagiosxi/cron/dbmaint.php
    /bin/env: php -q: No such file or directory

[Francesco]

How could this possibly fail?

Code: Select all

sudo /usr/local/nagiosxi/cron/dbmaint.php
    /bin/env: php -q: No such file or directory

Sorry but I did not read all the thread.

Is PHP installed and in path?

Code: Select all

[service@nagiosde02 ~]$ /bin/env php -q
^D
[service@nagiosde02 ~]$ /bin/env foobar -q
/bin/env: foobar: No such file or directory

Usually this error is when the user are not able to access the interpreter

[kyang]

Thanks for the help Francesco!

In addition to that, what are the permissions on the script?

Code: Select all

ll /usr/local/nagiosxi/cron/dbmaint.php

markmerchant, let us know!

[markmerchant]

I am once again, in the situation where I can't "Apply changes".

ll /usr/local/nagiosxi/cron/dbmaint.php
-rwxr-xr-x 1 nagios 437 14447 Feb 2 12:12 /usr/local/nagiosxi/cron/dbmaint.php

==== -rw-r----- 1 nagios nagios 766273 Feb 2 12:32 ./nagios/var/nagios.log
[1517592716] wproc: GLOBAL SERVICE EVENTHANDLER job 767 from worker Core Worker 21271 is a non-check helper but exited with return code 1
[1517592716] wproc: early_timeout=0; exited_ok=1; wait_status=256; error_code=0;
[1517592716] wproc: stdout line 01: UNABLE TO CONNECT TO DB - EXITING!
[1517592731] SERVICE ALERT: bluba01;ICMP for Network Hostgroups;OK;SOFT;2;PING OK - Packet loss = 0%, RTA = 3.81 ms
[1517592753] SERVICE ALERT: floba01.td.afg;ICMP for Network Hostgroups;WARNING;SOFT;1;PING WARNING - Packet loss = 16%, RTA = 3.03 ms


I can get it ti run if I change the first line:
#!/usr/bin/php -q