No hosts or services showing in CCM & additional error

[caf_infra]

Nagiosxi version 5.5.2, running on Centos 7

Hi,

We've got a problem with our Nagios Xi installation. The CCM isn't showing any objects at all, though the system is still correctly monitoring our hosts and services.

When I tried the trouble shooting steps in another post (Delete - Write - Verify -Restart) with a similar issue, I get the error "Cannot find the Nagios binary or no rights for execution!". I have checked the perms on the /usr/local/nagios/bin/nagios file, and ran chmod +x against it (perms are now 775, file ownership is nagios:nagios)

None of the file systems on the nagios system are full.

Any advice please?

[caf_infra]

Hi,

Would really like some advice, because we also need to reboot the nagios server for other reasons, and I want to be confident that it'll work once it's been rebooted.

Thanks

[jbrunkow]

You might just want to start by trying to repair the database by running the script below.

Code: Select all

/usr/local/nagiosxi/scripts/repair_databases.sh

DOC

If that doesn't work, send us a profile and we can examine that to determine what might be going awry. You can download the profile by navigating to System Profile under System Config on the left pane > and clicking the Download Profile button on that page. Then please reply to this topic to bring it up in our queue.

[caf_infra]

HI,

I've tried that, no change. I'm not sure it is a database issue, because it's continuing to monitor everything atm.

I've tried the reset_config_perms.sh script too because some of the errors we're getting imply a perms issue.

What's the best way to get the profile to you? Can we email it rather than uploading to the forum?

[jbrunkow]

Dang! It is fairly common for issues like these to be related to the database. Thanks for trying it.

Yes, of course! You could send it to me in a private message by clicking the PM icon next to one of my comments.

[caf_infra]

Sent, thank you.

[jbrunkow]

It looks like the database log in your profile is indicating a problem finding your table settings.

Code: Select all

[ERROR] mysqld: Can't find file: './nagiosql/tbl_settings.frm' (errno: 117)

Can you please look to see if there is anything in that folder on your server?

Code: Select all

cd /var/lib/mysql/nagiosql && ls

This error could also indicate a permissions issue with that folder or the files in it. If you run the following command, the first column in the output will show you your permissions to that file.

Code: Select all

ls -l tbl_settings.frm

If you have a backup, it may be easier to just revert XI to the last working state it was in. The only drawback is that you would lose performance data gathered since that backup was made. Please refer to the following linked documentation on how to do so.
BACKUP DOCUMENT

[caf_infra]

Everything is coming up in Red and Black (seems to indicate a broken file according to the bash documentation) in the CLI and with 'Structure Needs Cleaning' Errors. I can't see any sign of that file anywhere, not even a broken remnant.

As I said in my PM, we've got the same issue with files showing as red/black and with that 'Structure needs cleaning' error in other parts of the installation.

[jbrunkow]

Do you have a backup for that table that you can roll back to? You can check with the following command.

Code: Select all

ls /store/backups

As for the missing tbl_settings.frm file, you might be able to create it using the following command.

Code: Select all

echo 'repair table tbl_settings use_frm;' | mysql -h 127.0.0.1 -uroot -pnagiosxi nagiosql

Then check to see if that worked.

Code: Select all

cd /var/lib/mysql/nagiosql && ls

Several sources online are reporting that xfs_repair and e2fsck resolved similar file corruption errors. I recommend that you perform a backup of the damaged file systems, and then try to repair them using one of the two utilities. One or the other may already be present on your machine depending on the operating system...