Page 3 of 3
Re: Service recovery emails generated without status change
Posted: Thu Apr 18, 2019 4:44 pm
by npolovenko
Re: Service recovery emails generated without status change
Posted: Thu Apr 18, 2019 5:00 pm
by rferebee
Here's some more info. I found what actually triggers the reboot:
Cronjob -
root@nagiosxi> cat root
1 * * * * date > /var/log/cron
3 * * * * date > /var/log/mailog
5 * * * * date > /var/log/messages
6 * * * * date > /var/log/secure
7 * * * * date > /var/log/httpd/error_log
9 * * * * date > /var/log/httpd/access_log
10 1 * * * date > /var/log/httpd/ssl_request_log
12 1 * * * date > /var/log/httpd/ssl_error_log
14 1 * * * date > /var/log/httpd/ssl_access_log
5 10 * * 1-5 sh /home/nagios/scripts/weekly_bounce
30 14 * * 5 rm -f /store/backups/nagiosxi/restore/*
15 17 * * 4 /usr/local/nagiosxi/scripts/repairmysql.sh nagios
Script -
root@nagiosxi> cd /home/nagios/scripts
root@nagiosxi> ls
failover_event_handler.sh NRPE_failover_event_handler.sh weekly_bounce
root@nagiosxi> cat weekly_bounce
#!/bin/bash
#Weekly reboot of server to refresh memory
/sbin/shutdown -r now
root@nagiosxi>
Re: Service recovery emails generated without status change
Posted: Fri Apr 19, 2019 11:27 am
by npolovenko
@rferebee, It's using soft reboot command. That shouldn't cause any problems. I'm almost certain that you're experiencing the issue from the GitHub bug report.
But let us know if you get approval from your team to delete the retention.dat and status.dat files and take out the reboot script from the cron for 1-2 days.
Re: Service recovery emails generated without status change
Posted: Fri Apr 19, 2019 12:23 pm
by rferebee
Ok, so we suppressed the reboot this morning and no Service Recovery notifications were sent out. It would seem something during the reboot is triggering the messages.
Re: Service recovery emails generated without status change
Posted: Fri Apr 19, 2019 3:01 pm
by npolovenko
@rferebee, Can you manually run the reboot script in a middle of the day and let me know if you'll receive notifications? Or would it cuase too much trouble to keep recieveving recovery emails during the day?
/home/nagios/scripts/weekly_bounce
Re: Service recovery emails generated without status change
Posted: Fri Apr 19, 2019 3:28 pm
by rferebee
Ok, I just bounced XI using the 'weekly_bounce' script and it sent out all the Service Recovery emails like it has been doing.
Re: Service recovery emails generated without status change
Posted: Mon Apr 22, 2019 12:25 pm
by npolovenko
@rferebee, Interesting. I'd like to schedule a remote screen sharing session so I could take a look at your system live while this is happening and gather some more information.
Can you create a new support ticket below and include the link to this thread inside the ticket?
https://support.nagios.com/tickets/
Once you create a ticket I will send you the link to schedule a remote session.
Re: Service recovery emails generated without status change
Posted: Mon Apr 22, 2019 12:34 pm
by rferebee
Support ticket created. Thank you!
Re: Service recovery emails generated without status change
Posted: Mon Apr 22, 2019 1:38 pm
by npolovenko
@rferebee, Alright, I sent you the link in the ticket. I will be closing this thread so that we can focus our efforts in the ticket.
Thank you.