Latest Alerts is showing back dates (duration) in services

[biswajit.banerjee]

Hi Team,

We are facing issue in Latest Alerts in our Nagios XI, as it is showing back dates duration time (7675d 0h 1m 25s) in some of the devices. For your reference we are attaching a screenshot. Please provide solution as soon as possible.

[benjaminsmith]

Hi @biswajit.banerjee,

Let's check the all the time settings to make sure nothing is out of sync. Please post the output of the following commands to the thread.

Code: Select all

date
php -r 'echo date("D M j G:i:s T Y")."\n";'
echo "SELECT NOW();" | mysql -u root -pnagiosxi

If those are all correct, then send the system profile we'll review the logs for any errors. Thanks, Benjamin

To send us your system profile.
Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button

[biswajit.banerjee]

Hi Benjamin,

Thanks for the reply.
I am sending the output of the following commands.

[root@appserver1 ~]# date
Wed Jan 13 03:50:30 IST 2021
[root@appserver1 ~]# php -r 'echo date("D M j G:i:s T Y")."\n";'
Wed Jan 13 3:50:43 IST 2021
[root@appserver1 ~]# echo "SELECT NOW();" | mysql -u root -pnagiosxi
NOW()
2021-01-13 03:50:50

I am sending my system profile for your reference.

Moderator's Note: The profile has been shared with the support team but has been removed from the public forum.

[benjaminsmith]

Hi @biswajit.banerjee,

The time settings look good and the timestamps in the Nagios log look right. This particular dashlet is pulling from the database, and since your database is offloaded, the log was not in the system profile. Can you retrieve the database log from the remote server and post it to the thread.

The other issue here is the message queues are not getting processed as they should.

------ Message Queues --------
key msqid owner perms used-bytes messages
0x71000002 19300352 nagios 600 126724096 123754

Please follow the steps in the following guide to increasing the message queues on this server and restart Nagios as directed for Cent 6.

NDOUtils - Message Queue Exceeded

Also, I would recommend planning a migration in the near future since Cent 6 is now at the end of life and no longer receiving updates.

Best Regards,
Benjamin

[biswajit.banerjee]

Hi Benjamin,

Please find the database logs attached with this post.

[biswajit.banerjee]

Hi Benjamin,

We are waiting for your response, Please check it on priority.

[ssax]

Were those mysql logs from the XI server or your offloaded DB server? We need them from the offloaded DB server (192.168.1.50) if they were from the XI server.

You have an offloaded DB server, please run this command against it:
- You may need to change the root mysql password in the command below

Code: Select all

echo "SELECT NOW();" | mysql -h 192.168.1.50 -uroot -pnagiosxi

What does the date show on 192.168.1.50?

Code: Select all

date

Please run these commands as root on the XI server, and let me know if it resolves the issue:

Code: Select all

service httpd stop
service crond stop
service npcd stop
service nagios stop
service ndo2db stop
service mod-gearman2-worker stop || service mod-gearman-worker stop
service gearmand stop
pkill -9 -u nagios
pkill -9 -u apache
for i in $(ipcs -q | grep nagios |awk '{print $2}'); do ipcrm -q $i; done
rm -f /usr/local/nagiosxi/var/dbmaint.lock
rm -f /usr/local/nagiosxi/var/event_handler.lock
rm -f /usr/local/nagiosxi/scripts/reconfigure_nagios.lock
rm -f /usr/local/nagios/var/ndo2db.lock
rm -f /usr/local/nagios/var/ndo2db.pid
rm -f /usr/local/nagios/var/ndo2db.sock
rm -f /usr/local/nagios/var/ndo.sock
rm -f /us/local/nagiosxi/var/subsys/ndo2db
rm -f /var/run/nagios/nagios.lock
rm -f /var/run/nagios.lock
rm -f /usr/local/nagios/var/nagios.lock
rm -f /var/run/httpd/httpd.pid
rm -f /usr/local/nagiosxi/var/subsys/npcd.pid
service gearmand start
service mod-gearman2-worker start || service mod-gearman-worker start
service ndo2db start
service nagios start
service npcd start
service crond start
service httpd start
service snmptt restart

Additionally, please send the output of these commands:
- NOTE: You may need to adjust the -uroot and -pnagiosxi in the command if you've changed the root mysql password

Code: Select all

echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -h 192.168.1.50 -uroot -pnagiosxi --table

You should also SSH into every one of your mod_gearman workers and check the date/restart the worker service to see if that helps.

[biswajit.banerjee]

Hi Benjamin,

1. Were those mysql logs from the XI server or your offloaded DB server? We need them from the offloaded DB server (192.168.1.50) if they were from the XI server.

Yes, those mysql logs were from offloaded DB Server.

2. You have an offloaded DB server, please run this command against it:

echo "SELECT NOW();" | mysql -h 192.168.1.50 -uroot -pnagiosxi

Output:
[root@appserver1 ~]# echo "SELECT NOW();" | mysql -h 192.168.1.50 -uroot -p'jaqc31l'
ERROR 1045 (28000): Access denied for user 'root'@'192.168.1.3' (using password: YES)

3. What does the date show on 192.168.1.50?
Sun Jan 24 02:00:41 IST 2021

4. Please run these commands as root on the XI server, and let me know if it resolves the issue:
Run those command which was suggested by you, but it didn't resolve my problem.

5. Additionally, please send the output of these commands:

echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -h 192.168.1.50 -uroot -pnagiosxi --table

Output:

ERROR 1045 (28000): Access denied for user 'root'@'192.168.1.50' (using password: YES)

6. You should also SSH into every one of your mod_gearman workers and check the date/restart the worker service to see if that helps.

I have checked the date on my both mod_gearman workers and found that one worker has date "27 Jan 2000" and after that I have changed the date according to the current year and month and restart the nagios and gearman services on both mod_gearman worker but still showing the same error.

[ssax]

It's likely your workers considering the date said 2000, did you check all of your workers and do the same thing with restarting the service? They would need to send in new results for the dates to change.

If you force a check on one of those services, does it show the proper date then?

[biswajit.banerjee]

Hi Benjamin,

I have checked my two workers and restarted the service on both the workers, but still getting the same error.

When I do the force check on one those services, still it is not showing the proper dates. Please suggest.