Page 1 of 2

Downtime that won't go away

Posted: Wed Apr 03, 2019 1:40 pm
by efrahm
We have a service that shows as 'in scheduled downtime' and the history tab shows that it keeps rescheduling each hour, but I cannot find any downtime config or recurring downtime to remove it. I have tried turning off retention on that service and restarting, removing all comments, and repairing databases. I cannot seem to find where the downtime is coming from or how to completely wipe it.

Re: Downtime that won't go away

Posted: Wed Apr 03, 2019 3:34 pm
by ssax
What do you have in Home > Scheduled Downtime?

What about for Home > Recurring Downtime?

Please PM me a copy of your profile, you can download it from Admin > System Profile > Download Profile.

Please send this file as well (if you have it):

Code: Select all

/usr/local/nagiosxi/var/recurringdowntime.log

Re: Downtime that won't go away

Posted: Wed Apr 03, 2019 4:11 pm
by efrahm
well.. we have over 1100 objects in the scheduled downtime, but none for that host or that service, which happens to be a BPI process check. And only a few in recurring downtime which again do not include that host or service.

I have sent the recurringdowntime.log, and will PM the profile when it allows.

Neither the service name nor the host name appear in the logfile, as far as i can tell. I am really scratching my head on where it could be coming from.

This service owner often uses the NRDP interface to schedule downtimes from an automated script, but i cannot find any errant posts in the apache access logs that account for this.

Re: Downtime that won't go away

Posted: Wed Apr 03, 2019 4:24 pm
by ssax
Send the profile and the exact hostname/servicedesc that this is occurring with when you can, I will need that info to determine what exists and how it's being added.

Re: Downtime that won't go away

Posted: Wed Apr 03, 2019 4:32 pm
by efrahm
I managed to post the profile.zip in the PM now. I have removed the commend on the service several times, but it keeps recurring hourly and i cannot find the source. Even if i remove the comment, the 'this service is in scheduled downtime' statemen in Service Notes will not go away.

The 'host' is a dummy host called 'Technology Services core services'.. and the service is 'BPI Process:Illinois Compass 2g'.

we have a fairly large system with over 2600 hosts and 9700 services, with many active login accounts having access to post changes to their own objects, but only a few admins and a few users who actively use NRDP for inbound transfers or to create scheduled downtimes.

Re: Downtime that won't go away

Posted: Thu Apr 04, 2019 2:35 pm
by ssax
Please send me the downtime listed in Home > Scheduled Downtime that keeps getting added so that I can look at the start/end, etc.

Thank you

Re: Downtime that won't go away

Posted: Thu Apr 04, 2019 5:31 pm
by efrahm
the downtime in question does not show in that list, at all. there are 1146 items currenly on that list, but none for that host nor for any service by that name. this is coming from some other source. a broken text file, a database entry, or ...?

Re: Downtime that won't go away

Posted: Fri Apr 05, 2019 1:06 pm
by ssax
Please go to Configure > Core Config Manager > Tools > Config File Management:
- Click the Delete Files button (don't worry, it's safe, they will be rewritten)
- Then click the Write Configs button
- Then without doing anything in between this step and the next step (don't try to apply config or anything, we need the files in a bad state and this is the only way to get them):

Run these commands and send me the resulting /tmp/NAGIOSFILES.zip file:
- NOTE: You may need to adjust the -h 127.0.0.1, the -uroot, and -pnagiosxi in the command below if your DB is offloaded to another server and/or you've changed the root mysql password

Code: Select all

mysqldump -h 127.0.0.1 -uroot -pnagiosxi --add-drop-database nagios nagios_downtimehistory nagios_scheduleddowntime  > /tmp/nagios_downtime.dump
zip -r /tmp/NAGIOSFILES.zip /tmp/nagios_downtime.dump /usr/local/nagios/etc
Thank you!

Re: Downtime that won't go away

Posted: Thu May 30, 2019 4:06 pm
by murdock
I'm Rob and I work with Eric and am picking up this topic. Can you refer me to how / where you want me to upload the requested file to you?

Re: Downtime that won't go away

Posted: Thu May 30, 2019 4:23 pm
by npolovenko
Hello, @murdock. You can send these files via private message to me and @ssax. If sending it via private message doesn't work, you may upload files to a google drive or similar cloud storage and share the link with us in this thread.