NDO2DB Issue out of the blue

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
Locked
User avatar
BanditBBS
Posts: 2474
Joined: Tue May 31, 2011 12:57 pm
Location: Scio, OH
Contact:

Re: NDO2DB Issue out of the blue

Post by BanditBBS »

It hasn't been an issue in 27 hours. Not since I did the make and make install on the debug version. Once I stopped that and put the original in and restarted its been running great every since. I'll of course update this on Monday again.
2 of XI5.6.14 Prod/DR/DEV - Nagios LogServer 2 Nodes
See my projects on the Exchange at BanditBBS - Also check out my Nagios stuff on my personal page at Bandit's Home and at github
User avatar
BanditBBS
Posts: 2474
Joined: Tue May 31, 2011 12:57 pm
Location: Scio, OH
Contact:

Re: NDO2DB Issue out of the blue

Post by BanditBBS »

45 hour mark and if failed...was just a good Saturday apparently. Nice thing though, I used the action component for the first time and create a script to fix(restart ndo and kill extra queues) it upon a click.

So, patiently waiting any feedback from the log, beta/alpha of a patch or any other idea.

EDIT: It has crashed 3 times this morning(Monday).
2 of XI5.6.14 Prod/DR/DEV - Nagios LogServer 2 Nodes
See my projects on the Exchange at BanditBBS - Also check out my Nagios stuff on my personal page at Bandit's Home and at github
jfrickson

Re: NDO2DB Issue out of the blue

Post by jfrickson »

I looked over the log over the weekend and couldn't find anything suspicious.

Let's try something. Perform the same process as before, but instead of uncommenting all of the defines starting at line 72, just uncomment the first line from:

Code: Select all

/*#define DEBUG_NDO2DB 1*/                        /* don't daemonize */
to:

Code: Select all

#define DEBUG_NDO2DB 1                        /* don't daemonize */
Go to line 218. There you should see the following:

Code: Select all

	signal(SIGQUIT,ndo2db_parent_sighandler);
	signal(SIGTERM,ndo2db_parent_sighandler);
	signal(SIGINT,ndo2db_parent_sighandler);
	signal(SIGSEGV,ndo2db_parent_sighandler);
	signal(SIGFPE,ndo2db_parent_sighandler);
	signal(SIGCHLD,ndo2db_parent_sighandler);
at the end of that list, add the following lines:

Code: Select all

	signal(SIGHUP,ndo2db_parent_sighandler);
	signal(SIGILL,ndo2db_parent_sighandler);
	signal(SIGABRT,ndo2db_parent_sighandler);
	signal(SIGPIPE,ndo2db_parent_sighandler);
	signal(SIGALRM,ndo2db_parent_sighandler);
	signal(SIGUSR1,ndo2db_parent_sighandler);
	signal(SIGUSR2,ndo2db_parent_sighandler);
	signal(SIGBUS,ndo2db_parent_sighandler);
	signal(SIGPOLL,ndo2db_parent_sighandler);
	signal(SIGPROF,ndo2db_parent_sighandler);
	signal(SIGSYS,ndo2db_parent_sighandler);
	signal(SIGTRAP,ndo2db_parent_sighandler);
	signal(SIGVTALRM,ndo2db_parent_sighandler);
	signal(SIGXCPU,ndo2db_parent_sighandler);
	signal(SIGXFSZ,ndo2db_parent_sighandler);
	signal(SIGIO,ndo2db_parent_sighandler);
	signal(SIGPWR,ndo2db_parent_sighandler);
These are other signals that will cause the process to terminate.

Run ndo2db stand-alone as before, but you don't need to redirect stdout to a file. When the process dies, you should get a message saying:

Code: Select all

Caught the Signal 99 but don't care about this.
with the 99 replaced with some number. Let me know what (if anything) gets printed.
User avatar
BanditBBS
Posts: 2474
Joined: Tue May 31, 2011 12:57 pm
Location: Scio, OH
Contact:

Re: NDO2DB Issue out of the blue

Post by BanditBBS »

Sure, thanks, I'll get right on this! (just need to figure out how to keep my terminal session active as it times out after inactivity - I'm sure there is a setting, lol).
2 of XI5.6.14 Prod/DR/DEV - Nagios LogServer 2 Nodes
See my projects on the Exchange at BanditBBS - Also check out my Nagios stuff on my personal page at Bandit's Home and at github
tmcdonald
Posts: 9117
Joined: Mon Sep 23, 2013 8:40 am

Re: NDO2DB Issue out of the blue

Post by tmcdonald »

BanditBBS wrote:Sure, thanks, I'll get right on this! (just need to figure out how to keep my terminal session active as it times out after inactivity - I'm sure there is a setting, lol).
tmux or screen will at least keep the session active on the server so you can reconnect and not lose any work.
Former Nagios employee
User avatar
rseiwert
Posts: 196
Joined: Wed Jun 22, 2011 10:33 pm
Location: Somewhere between Here and Now

Re: NDO2DB Issue out of the blue

Post by rseiwert »

I'm wondering, if you set NDO2DEBUG directive and it does not daemonize, doesn't that stop it from using IPC and cause everything to run in one process?
Grumpy Olde IT Guy
jfrickson

Re: NDO2DB Issue out of the blue

Post by jfrickson »

rseiwert wrote:I'm wondering, if you set NDO2DEBUG directive and it does not daemonize, doesn't that stop it from using IPC and cause everything to run in one process?
It does run in just one process, but it still uses IPC.
User avatar
BanditBBS
Posts: 2474
Joined: Tue May 31, 2011 12:57 pm
Location: Scio, OH
Contact:

Re: NDO2DB Issue out of the blue

Post by BanditBBS »

Quick end of day update: Been running with modified ndo2db since 11:30ish(Central) and still running strong. Will update whenever it crashes.
2 of XI5.6.14 Prod/DR/DEV - Nagios LogServer 2 Nodes
See my projects on the Exchange at BanditBBS - Also check out my Nagios stuff on my personal page at Bandit's Home and at github
jfrickson

Re: NDO2DB Issue out of the blue

Post by jfrickson »

These things never crash when you want them to. Thanks for keeping on top of it!
User avatar
BanditBBS
Posts: 2474
Joined: Tue May 31, 2011 12:57 pm
Location: Scio, OH
Contact:

Re: NDO2DB Issue out of the blue

Post by BanditBBS »

jfrickson wrote:These things never crash when you want them to. Thanks for keeping on top of it!
Nothing useful. It was at 100% and not doing its job so I can to close it manually, so its not crashing to give any code

EDIT: Went back to original binary. Its not crashing to give us anything, it just spikes to 100% and stops doing anything. Its crashed 2 more times since last night
2 of XI5.6.14 Prod/DR/DEV - Nagios LogServer 2 Nodes
See my projects on the Exchange at BanditBBS - Also check out my Nagios stuff on my personal page at Bandit's Home and at github
Locked