Service recovery emails generated without status change

[npolovenko]

@rferebee, Sounds good!

[rferebee]

Here's some more info. I found what actually triggers the reboot:

Cronjob -

root@nagiosxi> cat root
1 * * * * date > /var/log/cron
3 * * * * date > /var/log/mailog
5 * * * * date > /var/log/messages
6 * * * * date > /var/log/secure
7 * * * * date > /var/log/httpd/error_log
9 * * * * date > /var/log/httpd/access_log
10 1 * * * date > /var/log/httpd/ssl_request_log
12 1 * * * date > /var/log/httpd/ssl_error_log
14 1 * * * date > /var/log/httpd/ssl_access_log
5 10 * * 1-5 sh /home/nagios/scripts/weekly_bounce
30 14 * * 5 rm -f /store/backups/nagiosxi/restore/*
15 17 * * 4 /usr/local/nagiosxi/scripts/repairmysql.sh nagios

Script -

root@nagiosxi> cd /home/nagios/scripts
root@nagiosxi> ls
failover_event_handler.sh NRPE_failover_event_handler.sh weekly_bounce
root@nagiosxi> cat weekly_bounce
#!/bin/bash
#Weekly reboot of server to refresh memory
/sbin/shutdown -r now
root@nagiosxi>

[npolovenko]

@rferebee, It's using soft reboot command. That shouldn't cause any problems. I'm almost certain that you're experiencing the issue from the GitHub bug report.
But let us know if you get approval from your team to delete the retention.dat and status.dat files and take out the reboot script from the cron for 1-2 days.

[rferebee]

Ok, so we suppressed the reboot this morning and no Service Recovery notifications were sent out. It would seem something during the reboot is triggering the messages.

[npolovenko]

@rferebee, Can you manually run the reboot script in a middle of the day and let me know if you'll receive notifications? Or would it cuase too much trouble to keep recieveving recovery emails during the day?

/home/nagios/scripts/weekly_bounce

[rferebee]

Ok, I just bounced XI using the 'weekly_bounce' script and it sent out all the Service Recovery emails like it has been doing.

[npolovenko]

@rferebee, Interesting. I'd like to schedule a remote screen sharing session so I could take a look at your system live while this is happening and gather some more information.
Can you create a new support ticket below and include the link to this thread inside the ticket?
https://support.nagios.com/tickets/
Once you create a ticket I will send you the link to schedule a remote session.

[rferebee]

Support ticket created. Thank you!

[npolovenko]

@rferebee, Alright, I sent you the link in the ticket. I will be closing this thread so that we can focus our efforts in the ticket.
Thank you.