Page 1 of 2
Latest Alerts is showing back dates (duration) in services
Posted: Fri Jan 08, 2021 2:43 pm
by biswajit.banerjee
Hi Team,
We are facing issue in Latest Alerts in our Nagios XI, as it is showing back dates duration time (7675d 0h 1m 25s) in some of the devices. For your reference we are attaching a screenshot. Please provide solution as soon as possible.
Re: Latest Alerts is showing back dates (duration) in servic
Posted: Mon Jan 11, 2021 1:54 pm
by benjaminsmith
Hi
@biswajit.banerjee,
Let's check the all the time settings to make sure nothing is out of sync. Please post the output of the following commands to the thread.
Code: Select all
date
php -r 'echo date("D M j G:i:s T Y")."\n";'
echo "SELECT NOW();" | mysql -u root -pnagiosxi
If those are all correct, then send the system profile we'll review the logs for any errors. Thanks, Benjamin
To send us your system profile.
Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Re: Latest Alerts is showing back dates (duration) in servic
Posted: Tue Jan 12, 2021 5:18 pm
by biswajit.banerjee
Hi Benjamin,
Thanks for the reply.
I am sending the output of the following commands.
[root@appserver1 ~]# date
Wed Jan 13 03:50:30 IST 2021
[root@appserver1 ~]# php -r 'echo date("D M j G:i:s T Y")."\n";'
Wed Jan 13 3:50:43 IST 2021
[root@appserver1 ~]# echo "SELECT NOW();" | mysql -u root -pnagiosxi
NOW()
2021-01-13 03:50:50
I am sending my system profile for your reference.
Moderator's Note: The profile has been shared with the support team but has been removed from the public forum.
Re: Latest Alerts is showing back dates (duration) in servic
Posted: Wed Jan 13, 2021 4:34 pm
by benjaminsmith
Hi
@biswajit.banerjee,
The time settings look good and the timestamps in the Nagios log look right. This particular dashlet is pulling from the database, and since your database is offloaded, the log was not in the system profile. Can you retrieve the database log from the remote server and post it to the thread.
The other issue here is the message queues are not getting processed as they should.
------ Message Queues --------
key msqid owner perms used-bytes messages
0x71000002 19300352 nagios 600 126724096 123754
Please follow the steps in the following guide to increasing the message queues on this server and restart Nagios as directed for Cent 6.
NDOUtils - Message Queue Exceeded
Also, I would recommend planning a migration in the near future since Cent 6 is now at the end of life and no longer receiving updates.
Best Regards,
Benjamin
Re: Latest Alerts is showing back dates (duration) in servic
Posted: Mon Jan 18, 2021 4:51 pm
by biswajit.banerjee
Hi Benjamin,
Please find the database logs attached with this post.
Re: Latest Alerts is showing back dates (duration) in servic
Posted: Wed Jan 20, 2021 11:28 am
by biswajit.banerjee
Hi Benjamin,
We are waiting for your response, Please check it on priority.
Re: Latest Alerts is showing back dates (duration) in servic
Posted: Wed Jan 20, 2021 11:32 am
by ssax
Were those mysql logs from the XI server or your offloaded DB server? We need them from the offloaded DB server (192.168.1.50) if they were from the XI server.
You have an offloaded DB server, please run this command against it:
- You may need to change the root mysql password in the command below
Code: Select all
echo "SELECT NOW();" | mysql -h 192.168.1.50 -uroot -pnagiosxi
What does the date show on 192.168.1.50?
Please run these commands as root on the XI server, and let me know if it resolves the issue:
Code: Select all
service httpd stop
service crond stop
service npcd stop
service nagios stop
service ndo2db stop
service mod-gearman2-worker stop || service mod-gearman-worker stop
service gearmand stop
pkill -9 -u nagios
pkill -9 -u apache
for i in $(ipcs -q | grep nagios |awk '{print $2}'); do ipcrm -q $i; done
rm -f /usr/local/nagiosxi/var/dbmaint.lock
rm -f /usr/local/nagiosxi/var/event_handler.lock
rm -f /usr/local/nagiosxi/scripts/reconfigure_nagios.lock
rm -f /usr/local/nagios/var/ndo2db.lock
rm -f /usr/local/nagios/var/ndo2db.pid
rm -f /usr/local/nagios/var/ndo2db.sock
rm -f /usr/local/nagios/var/ndo.sock
rm -f /us/local/nagiosxi/var/subsys/ndo2db
rm -f /var/run/nagios/nagios.lock
rm -f /var/run/nagios.lock
rm -f /usr/local/nagios/var/nagios.lock
rm -f /var/run/httpd/httpd.pid
rm -f /usr/local/nagiosxi/var/subsys/npcd.pid
service gearmand start
service mod-gearman2-worker start || service mod-gearman-worker start
service ndo2db start
service nagios start
service npcd start
service crond start
service httpd start
service snmptt restart
Additionally, please send the output of these commands:
- NOTE: You may need to adjust the -uroot and -pnagiosxi in the command if you've changed the root mysql password
Code: Select all
echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -h 192.168.1.50 -uroot -pnagiosxi --table
You should also SSH into every one of your mod_gearman workers and check the date/restart the worker service to see if that helps.
Re: Latest Alerts is showing back dates (duration) in servic
Posted: Sat Jan 23, 2021 3:40 pm
by biswajit.banerjee
Hi Benjamin,
1. Were those mysql logs from the XI server or your offloaded DB server? We need them from the offloaded DB server (192.168.1.50) if they were from the XI server.
Yes, those mysql logs were from offloaded DB Server.
2. You have an offloaded DB server, please run this command against it:
echo "SELECT NOW();" | mysql -h 192.168.1.50 -uroot -pnagiosxi
Output:
[root@appserver1 ~]# echo "SELECT NOW();" | mysql -h 192.168.1.50 -uroot -p'jaqc31l'
ERROR 1045 (28000): Access denied for user 'root'@'192.168.1.3' (using password: YES)
3. What does the date show on 192.168.1.50?
Sun Jan 24 02:00:41 IST 2021
4. Please run these commands as root on the XI server, and let me know if it resolves the issue:
Run those command which was suggested by you, but it didn't resolve my problem.
5. Additionally, please send the output of these commands:
echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -h 192.168.1.50 -uroot -pnagiosxi --table
Output:
ERROR 1045 (28000): Access denied for user 'root'@'192.168.1.50' (using password: YES)
6. You should also SSH into every one of your mod_gearman workers and check the date/restart the worker service to see if that helps.
I have checked the date on my both mod_gearman workers and found that one worker has date "27 Jan 2000" and after that I have changed the date according to the current year and month and restart the nagios and gearman services on both mod_gearman worker but still showing the same error.
Re: Latest Alerts is showing back dates (duration) in servic
Posted: Mon Jan 25, 2021 5:26 pm
by ssax
It's likely your workers considering the date said 2000, did you check all of your workers and do the same thing with restarting the service? They would need to send in new results for the dates to change.
If you force a check on one of those services, does it show the proper date then?
Re: Latest Alerts is showing back dates (duration) in servic
Posted: Wed Jan 27, 2021 2:55 pm
by biswajit.banerjee
Hi Benjamin,
I have checked my two workers and restarted the service on both the workers, but still getting the same error.
When I do the force check on one those services, still it is not showing the proper dates. Please suggest.