NDO2DB Issue out of the blue

[jfrickson]

No, patch the original. But use this patch instead. I messed up the other one.

[BanditBBS]

No, patch the original. But use this patch instead. I messed up the other one.

Done and looking better

Code: Select all

[nagios@iss-chi-nag09 var]$ ps -ef|grep ndo
nagios    2000     1  0 12:47 ?        00:00:00 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg
nagios    2062  2000  1 12:48 ?        00:00:00 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg
nagios    2063  2062 21 12:48 ?        00:00:04 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg
nagios    2084  1950  0 12:48 pts/2    00:00:00 grep ndo
[nagios@iss-chi-nag09 var]$ ll
total 298656
-rw-r--r-- 1 nagios nagios        76 Aug 26 12:48 debug_2000_.log
-rw-r--r-- 1 nagios nagios   5378622 Aug 26 12:48 debug_2062_.log
-rw-r--r-- 1 nagios nagios  48943887 Aug 26 12:48 debug_2063_.log
-rw-r--r-- 1 nagios nagios   9472224 Aug 26 12:33 ndo2db.debug
-rw-r--r-- 1 nagios nagios  20002925 Aug 26 12:33 ndo2db.debug.old
-rw-r--r-- 1 nagios nagios         5 Aug 26 12:47 ndo2db.lock
-rw-r--r-- 1 root   root   222007026 Aug 21 13:44 ndo2db.log

I'll update when it crashes/hangs

p.s. I liked the "Bleh...sorry" unedited message, lol

EDIT: It still seems to be only the same pids so the files are growing rather rapidly, the 2063 one is 500MB already

[jfrickson]

...
EDIT: It still seems to be only the same pids so the files are growing rather rapidly, the 2063 one is 500MB already

I was thinking they would turn over faster.
This one should keep less than 1,000 lines in the current file, and keep a previous log for the previous 1,000 lines.

[BanditBBS]

...
EDIT: It still seems to be only the same pids so the files are growing rather rapidly, the 2063 one is 500MB already

I was thinking they would turn over faster.
This one should keep less than 1,000 lines in the current file, and keep a previous log for the previous 1,000 lines.

Ok, this one is now in place, we'll see what happens! With this I can also keep my event handler running too since the pids will change and the old logs will remain

[hsmith]

...
EDIT: It still seems to be only the same pids so the files are growing rather rapidly, the 2063 one is 500MB already

I was thinking they would turn over faster.
This one should keep less than 1,000 lines in the current file, and keep a previous log for the previous 1,000 lines.

Ok, this one is now in place, we'll see what happens! With this I can also keep my event handler running too since the pids will change and the old logs will remain

You know the drill, let us know what you come up with!

[BanditBBS]

PM'ing Trevor and jfrickson these files:

Code: Select all

-rw-r--r-- 1 nagios nagios        76 Aug 26 13:15 debug_6938_.log
-rw-r--r-- 1 nagios nagios     72366 Aug 26 17:46 debug_6948_.log
-rw-r--r-- 1 nagios nagios    204854 Aug 26 17:46 debug_6948_.log.prev
-rw-r--r-- 1 nagios nagios  12624549 Aug 26 17:46 debug_6949_.log
-rw-r--r-- 1 nagios nagios   9673526 Aug 26 17:41 debug_6949_.log.prev

[BanditBBS]

I have 2 more sets of files I can send if needed, just let me know!

[jfrickson]

I have 2 more sets of files I can send if needed, just let me know!

I don't think I will need them. I should have a fix for you in a little while, probably under an hour.

[jfrickson]

Apply the attached patch to the original ndo2db.c and run it. I'm pretty sure this will solve the problem.

[BanditBBS]

Apply the attached patch to the original ndo2db.c and run it. I'm pretty sure this will solve the problem.

Well, I have this in, we'll see what happens. I must mention this though as well....based on the tail end of some of the debug logs generated when it got in its infinite loop I fixed the 2 checks that were causing a ton of longoutput and : from being sent, so, if that was the cause, we may not see it again anyway. But I had to do it! I'll let you know if it seems resolved(I'll call it resolved after it doesn't happen again in 3 days)