nagios stalls unless I use strace

[jssingh]

I am setting up nagios on a new server (RHEL 7) and it was working fine for a couple weeks and then all of a sudden it just stalled. Nagios itself still runs, but nothing is happening. status.dat doesn't update, there is nothing in nagios.log or nagios.debug. The data in spool/checkresults just piles up. We saw a few zombie processes, so we restarted nagios, but then it would run some tests for less than a minute and then just stall.

We can't think of anything that changed on the system to have caused the behavior to change.

Today I tried running nagios with strace to see if we could see what was causing it to stall. But when I run it with strace, nagios works flawlessly. Our theory is that it slows nagios down enough to get around whatever the issue is. But we're not sure what to do from here.

Has anyone else seen anything like this before?

[Box293]

Run this command to see if the nagios user is expired:

Code: Select all

chage -l nagios

If it is, run this command to enable the expired nagios user:

Code: Select all

chage -I -1 -m 0 -M 99999 -E -1 nagios

If that isn't the problem then try setting the debug level on and then restart Nagios.

Code: Select all

sed -i 's/.*debug_level=.*/debug_level=-1/g' /usr/local/nagios/etc/nagios.cfg
service nagios restart

Do you see anything logged in the file /usr/local/nagios/var/nagios.debug

When you are finished this turns debugging off:

Code: Select all

sed -i 's/.*debug_level=.*/debug_level=0/g' /usr/local/nagios/etc/nagios.cfg
service nagios restart

[jssingh]

It turns out that it was set up to write the perflogs to a different filesystem and once I changed them to be on the same filesystem, everything started working. So we'll need to look into what's going on with those filesystems, but that's not a Nagios problem. Thanks!

[Box293]

Awesome, great to hear.