Nagios 4 Load issues - OS X 10.9

[jpotter]

We're seeing similar issues as described in this post -- on multiple of our developer's OS X 10.9 boxes. Reverting to 3.5 solves it. When this happens, there are multiple nagios processes running at high CPU, like they're worker processes about to exec a plugin, or worker processes managing a plugin that get into a bad state.

Executable module set to "/usr/local/bin/nagios".
Architecture set to: x86_64-apple-macosx.
(lldb) bt all
* thread #1: tid = 0x1f9045, 0x00007fff8737f94a libsystem_kernel.dylib`poll + 10, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
* frame #0: 0x00007fff8737f94a libsystem_kernel.dylib`poll + 10
frame #1: 0x000000010d840f4b nagios`___lldb_unnamed_function816$$nagios + 126
frame #2: 0x000000010d843a4a nagios`___lldb_unnamed_function894$$nagios + 528
frame #3: 0x000000010d7e9f57 nagios`___lldb_unnamed_function1$$nagios + 4115

sudo dtruss -p 14462
Password:
dtrace: 187845 dynamic variable drops with non-empty dirty list
SYSCALL(args) = return
read(0x12, "\0", 0x1000) = -1 Err#35
read(0x12, "\0", 0x1000) = -1 Err#35
read(0x12, "\0", 0x1000) = -1 Err#35
read(0x12, "\0", 0x1000) = -1 Err#35
read(0x12, "\0", 0x1000) = -1 Err#35
read(0x12, "\0", 0x1000) = -1 Err#35
read(0x12, "\0", 0x1000) = -1 Err#35

[jpotter]

Oh, I forgot one important detail: we almost always see this happening after waking the laptops from sleep.

[sreinhardt]

So this is happening when resuming from sleep? Also where are you seeing these error messages? I can say that nagios is certainly not intended to be put to sleep, and it seems that upon waking, it is possibly missing some system files that have not yet been loaded by the kernel, but I don't think that would be related to a load increase.

[jpotter]

Sorry; just realized I didn't have "notify me when a reply is posted" checked on.

The debugging info I pasted there was from dtruss'ing the process and from attaching lldb debugger to it. There's no error messages to speak off -- the process simply goes to 100% of CPU and burns cycles (and battery) like you would not believe.

This happens when waking the machine from any standard sleep -- nothing exotic, and quite common on a laptop.

[abrist]

There were a number of big fixes for auto-rescheduling of checks in 4.0.8. Could you try upgrading and then testing the sleep issues? I have a suspicion that due to the machine sleeping, all checks were stale and needed to be rescheduled on resume. Due to rescheduling bugs prior to 4.0.8, upgrading *may* solve your issue.

[jpotter]

Hi abrist,

I tried compiling 4.0.8 for OS X, but get the below error (and two warnings nearby) -- it looks like a missing '=' in what presumably should be "== -1"?

runcmd.c:590:50: error: expression is not assignable
if (asprintf(&env_string, "%s=%s", name, value) = -1) return -1;
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^
runcmd.c:592:9: warning: equality comparison result unused [-Wunused-comparison]
errno == ENOMEM;
~~~~~~^~~~~~~~~
runcmd.c:592:9: note: use '=' to turn this equality comparison into an assignment
errno == ENOMEM;

This is using the source code from https://downloads.sourceforge.net/proje ... 0.8.tar.gz, which is the URL in the brew install used for 4.0.6 (I'll submit a patch to brew to use 4.0.8 once this is resolved).

Thanks,
Jeff

[tmcdonald]

Have you tried changing that assignment = to a comparison == and re-compiling?

[jpotter]

Yes, fixing the '='/'==' error enables compiling of Nagios 4.0.8 on OS X. (But not sure if 4.0.8 fixes the load issues on OS X.)

Nagios 4.0.8, as released, has a bug in it (using assignment "=" in place of comparison "==") that prevents it from compiling on OS X. The block of code is in an ifdef, so this block of code isn't reached on some systems:

#ifdef HAVE_SETENV
return setenv(name, value, 1);
#else
/* For Solaris and systems that don't have setenv().
* This will leak memory, but in a "controlled" way, since the memory
* should be freed when the child process exits. */
if (asprintf(&env_string, "%s=%s", name, value) == -1) return -1;
if (!env_string) {
errno == ENOMEM;
return -1;
}
return putenv(env_string);
#endif
}

While I can manually install Nagios 4.0.8 on my machine by fixing this, I can't submit a pull request to Brew to have OS X use 4.0.8 instead of 4.0.6.

It's a simple fix:
lib/runcmd.c
590c590
< if (asprintf(&env_string, "%s=%s", name, value) == -1) return -1;
---
> if (asprintf(&env_string, "%s=%s", name, value) = -1) return -1;


-Jeff

[sreinhardt]

Thanks for the report! I'll get this sent off to the core devs.

[jpotter]

Thanks! I'm happy to send in a bug report / patch, just didn't see where to do it.

-Jeff