Nagios Support Forum

Today we had several Nagios daemons segfault randomly (all running RHEL8.10 and Nagios Core 4.4.13). We tried restarting them immediately, but they all continued to segfault. Looking in /var/log/messages, we found the following...

Oct 24 10:45:58 tucnag02 kernel: nagios[1345618]: segfault Oct 24 10:45:58 tucnag02 systemd-coredump[1345671]: Process 1345618 (nagios) of user 100 dumped core.#012#012Stack trace of thread 1345618:#012#0 0x0000000000444cbf query_update_api (nagios)#012#1 0x0000000000444e0d check_for_nagios_updates (nagios)#012#2 0x0000000000413df0 main (nagios)#012#3 0x00007ffff6d2a7e5 __libc_start_main (libc.so.6)#012#4 0x00000000004146be _start (nagios)at 0 ip 0000000000444cbf sp 00007fffffffe550 error 4 in nagios[400000+aa000]

Looking at the stack trace reference in the 2nd line above, I went to the source code and found that function. It looks like it checks for updates at startup and also periodically during run time. It seemed to be failing because of this check. To fix, I set the following values in the retention.dat and status.dat files...

last_update_check=<current epoch time>

Once updated, the daemons started just fine. So I went ahead and created a script and a cron job to periodically set those values to the current time. Is there a way to disable the update check via config files? I don't understand why today all of the instances failed like they did since they had been running with no issue for years before this. Also, I realize that we're fairly behind on the Nagios Core version, so if this has already been fixed in a newer version, I apologize. Any assistance is much appreciated.

I wonder if Nagios inc updated https://www.nagios.org/checkforupdates/ ... nagioscore page.
We getting same issue with version 4.4.6.

we modified nagios.cfg to fix the issue
check_for_updates=1 to check_for_updates=0

Can the nagios host resolve the various nagios addresses?

Hi all,

We can acknowledge an issue with the check_for_nagios_updates functionality. We're looking into both what happened on our end with the call to our update server, and also why Core segfaulted under that condition.

Aaron
Product Development Manager

Thanks for the update and acknowledgement. After the fact, we did end up finding the check_for_updates config option and that seems to be the easiest work around.

I reported the issue here: https://github.com/NagiosEnterprises/na ... ssues/1041 and proposed a fix as well. I believe the issue was triggered by a malformed response from the nagios version check api response during that time and the response parsing logic was missing a null check.

I had not noticed this option until we also ran into this last Friday.
While browsing the docs, at check_for_updates it says:

Data collected by Nagios Enterprises from the update check is processed in accordance with our privacy policy - see http://api.nagios.org for details.

But on that page, the link to the terms of use (http://www.nagios.org/legal/termsofuse/) is dead, the link to the trade mark policy (http://www.nagios.org/legal/trademarkpolicy/) is dead, and the link to the privacy policy (http://www.nagios.org/legal/privacypolicy/) is also dead.

I will be setting check_for_updates (and its companion bare_update_check) to 0.

Thank you @imankurpatel000 for the merge request for this. Even though I can't get an empty api.nagios.org response, I was able to simulate one in reading the response, reproducing the crash and verifying that the MR fixes the issue. It has been merged and will go out in the next Nagios Core release.

Hey, Jason. Thanks for merging the PR. I was also hoping to get some official confirmation from Nagios if api.nagios.org did indeed send an empty response similar to what I mentioned in the GitHub issue during that period. Is there a way to confirm this?

Hi Ankur,

Thanks for your note. We did not capture any return while the API was acting weird, we reverted as soon as we knew something was going wrong. And we are not now able to reproduce the behavior. Your PR does address a path to segfault behavior though, and we know Core can handle no response well enough, so it seems reasonable to conclude that there was something of a response that looked valid but was mangled. Perhaps there was a race condition and the API returned but without the variables filled in for version number. We do know from some post-incident analysis that we had something going on that looked like a resource leak on our api server, with cpu, memory, and network stats all climbing over a couple of hours.

Aaron