Random SIGTERM......

[spaceship]

Hi,

My nagios decides to shut down at random time due to a plugin

Caught SIGTERM, shutting down...

If I take out the plugin in question it works fine but I am puzzled about how a plugin would cause nagios to SIGTERM.

Thanks

Tim

[agriffin]

What plugin is it?

[spaceship]

It's mine, but it's not the first plugin I've written.

[agriffin]

Well, if you think the problem may be with the plugin, you should probably post the plugin. I haven't run into this issue before, so I don't know what may be causing this.

[spaceship]

I suspect the plugin because if I change my command.cfg to point at check_dummy instead of my plugin no SIGTERM's happen. Though I am interested in any other reasons why nagios would randomly SIGTERM as this could be a coincidence.

I've attached my code, it does require that gtk3 and lot of perl modules to get WWW::WebKit installed.

[mguthrie]

This could be an issue related to using embedded perl. I would turn off the two options in the nagios.cfg file related to embedded perl, restart the server, and see if the issue persists. The Core engine can leak memory if used with embedded perl and launching some perl plugins, so it could be SIGTERMing because it runs out of memory somewhere. Check your nagios.log and your system log for any clues.

[spaceship]

I don't use the embedded perl interpreter I have never had much joy with it, so I neither compile it in or have it in the config file.

[spaceship]

My logs just tell me that nagios has sigtermed

[1340893308] Caught SIGTERM, shutting down...
[1340893308] Successfully shutdown... (PID=9475)
[1340893308] npcdmod: If you don't like me, I will go out! Bye.
[1340893308] Event broker module '/usr/local/pnp4nagios/lib/npcdmod.o' deinitialized successfully.
[1340893309] livestatus: Socket thread has terminated
[1340893309] Event broker module '/usr/lib/check_mk/livestatus.o' deinitialized successfully.

I have even been using strace against the nagios daemon

umask(02) = 077
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=3661, ...}) = 0
write(9, "### Active Check Result File ###"..., 291) = 291
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7ff52c34e9d0) = 24644
close(9) = 0
munmap(0x7ff52c358000, 4096) = 0
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=3661, ...}) = 0
nanosleep({0, 500000000}, NULL) = ? ERESTART_RESTARTBLOCK (To be restarted)
--- SIGCHLD (Child exited) @ 0 (0) ---
restart_syscall(<... resuming interrupted call ...>) = ? ERESTART_RESTARTBLOCK (To be restarted)
--- SIGCHLD (Child exited) @ 0 (0) ---
restart_syscall(<... resuming interrupted call ...>) = ? ERESTART_RESTARTBLOCK (To be restarted)
--- SIGCHLD (Child exited) @ 0 (0) ---
restart_syscall(<... resuming interrupted call ...>) = ? ERESTART_RESTARTBLOCK (To be restarted)
--- SIGCHLD (Child exited) @ 0 (0) ---
restart_syscall(<... resuming interrupted call ...>) = ? ERESTART_RESTARTBLOCK (To be restarted)
--- SIGCHLD (Child exited) @ 0 (0) ---
restart_syscall(<... resuming interrupted call ...>) = ? ERESTART_RESTARTBLOCK (To be restarted)
--- SIGCHLD (Child exited) @ 0 (0) ---
restart_syscall(<... resuming interrupted call ...>) = ? ERESTART_RESTARTBLOCK (To be restarted)
--- SIGCHLD (Child exited) @ 0 (0) ---
restart_syscall(<... resuming interrupted call ...>) = ? ERESTART_RESTARTBLOCK (To be restarted)
--- SIGCHLD (Child exited) @ 0 (0) ---
restart_syscall(<... resuming interrupted call ...>) = ? ERESTART_RESTARTBLOCK (To be restarted)
--- SIGCHLD (Child exited) @ 0 (0) ---
restart_syscall(<... resuming interrupted call ...>) = 0
open("/usr/local/nagios/var/nagios.log", O_RDWR|O_CREAT|O_APPEND, 0666) = 9
fstat(9, {st_mode=S_IFREG|0664, st_size=947749, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff52c358000
write(9, "[1340893308] Caught SIGTERM, shu"..., 46) = 46
close(9) = 0
munmap(0x7ff52c358000, 4096) = 0

Still I'm none the wiser.

[agriffin]

Are you running SELinux? Does your plugin require an X server or something that might not be running on the system?

[spaceship]

I'm not running SELinux.

The plugin does require an X server but I am using Xvfb to provide a virtual X session.