NDO2DB Issue out of the blue

Post by **BanditBBS** » Fri Aug 21, 2015 12:19 pm

Awesome instructions

I'll get to this ASAP. Just need to find the source file(may need to extract from nagiosxi tar). My setup is just a tad different since I have MySQL and NDO2DB offloaded, so no clue where I put that source. Hopefully you'll have a log file this afternoon sometime.

Post by **BanditBBS** » Fri Aug 21, 2015 12:39 pm

Ok, its running. That log is growing quite rapidly, so I may need to come up with a method to auto rotate it and only keep 100MB or something, after running for just a minute its at 100MB already. So that's 1GB in 10 minutes and I only have 30 GB available space.

jfrickson · Post by **jfrickson** » Fri Aug 21, 2015 12:45 pm

I figured it would be big, but wow! I hate to say it, but I hope it crashes soon!

jfrickson · Post by **jfrickson** » Fri Aug 21, 2015 12:53 pm

You could probably do something like ndo2db -c /usr/local/nagios/etc/ndo2db.cfg | gzip -c > ../var/ndo2db.log and cut the size down by 80-90%

Post by **BanditBBS** » Fri Aug 21, 2015 1:14 pm

jfrickson wrote:You could probably do something like ndo2db -c /usr/local/nagios/etc/ndo2db.cfg | gzip -c > ../var/ndo2db.log and cut the size down by 80-90%

Made that change, much much smaller now.

Also, my lines I had to change in src were 72-81, not 82-91.

Post by **BanditBBS** » Fri Aug 21, 2015 1:51 pm

Here are the last 100 lines...doesnt look like anything bad to me

Code: Select all

[root@iss-chi-nag09 tmp]# tail -n100 ndo2db.log
4=1440182644.882980
53=rbns-chi-ftp1
114=Zombie Processes
95=PROCS OK: 0 processes with STATE = Z
125=
99=procs=0;5;10;0;
27=0
51=1
115=1
25=1
76=3
61=1440182644
83=1440182944
12=0
63=1439127565
57=1439127565
56=0
66=1440182644
70=0
67=0
64=1439098977
121=1
62=0
84=0
85=1
88=1
101=0
7=0
26=0
97=1
38=1
9=1
47=1
54=0
98=0.00000
71=0.00000
42=0.04949
113=0
45=0
103=1
93=1
80=0
37=
11=check_by_nrpe_procs!5!10!-s Z!!!!!
86=5.00000
109=2.00000
209=xi_timeperiod_24x7
262=XIWIZARD:0:sshproxy
9
  USED1: 511, BYTES: 0, LINES: 0
BYTESREAD: 4
RAWBUF: 99


  USED1: 4, BYTES: 0, LINES: 0
BYTESREAD: 511
RAWBUF:
213:
1=1202
2=0
3=0
4=1440182644.883085
53=rbns-chi-ftp1
114=Zombie Processes
95=PROCS OK: 0 processes with STATE = Z
125=
99=procs=0;5;10;0;
27=0
51=1
115=1
25=1
76=3
61=1440182644
83=1440182944
12=0
63=1439127565
57=1439127565
56=0
66=1440182644
70=0
67=0
64=1439098977
121=1
62=0
84=0
85=1
88=1
101=0
7=0
26=0
97=1
38=1
9=1
47=1
54=0
98=0.00000
71=0.00000
42=0.04949
113=0
45=0

Edit: Here are the last 500 lines, still nothing looking odd to me at least:

Code: Select all

[root@iss-chi-nag09 tmp]# tail -n500 ndo2db.log
56=0
66=1440182644
70=0
67=0
64=1439877724
121=1
62=0
84=0
85=1
88=1
101=0
7=0
26=0
97=1
38=1
9=1
47=1
54=0
98=0.00000
71=0.00000
42=0.36368
113=0
45=0
103=1
93=1
80=0
37=
11=check_by_nrpe_rofs!!!!!!!!
86=7.00000
109=2.00000
209=xi_timeperiod_24x7
262=XIWIZARD:0:sshproxy
999


  USED1: 494, BYTES: 0, LINES: 0
BYTESREAD: 511
RAWBUF:
213:
1=1202
2=0
3=0
4=1440182644.765904
53=ctfg-365-pap01
114=Log File
95=OK - No matches found.
125=
99=lines=0
27=0
51=1
115=1
25=1
76=1
61=1440182644
83=1440183064
12=0
63=1437802224
57=1437802224
56=0
66=1440182644
70=1431872057
67=1422387186
64=1437773636
121=1
62=0
84=0
85=1
88=1
101=0
7=0
26=0
97=1
38=1
9=1
47=1
54=0
98=0.00000
71=0.00000
42=0.36777
113=0
45=0
103=1
93=1
80=0
37=
11=check_by_nrpe_log!/var/log/messages!Error!most_recent!!!!!
86=7.00000
109=2.00000
209=xi_timeperiod_24x7
262=XIWIZARD
  USED1: 511, BYTES: 0, LINES: 0
BYTESREAD: 17
RAWBUF: :0:sshproxy
999


  USED1: 17, BYTES: 0, LINES: 0
BYTESREAD: 511
RAWBUF:
213:
1=1202
2=0
3=0
4=1440182644.765976
53=ctfg-365-pap01
114=Log File
95=OK - No matches found.
125=
99=lines=0
27=0
51=1
115=1
25=1
76=1
61=1440182644
83=1440183064
12=0
63=1437802224
57=1437802224
56=0
66=1440182644
70=1431872057
67=1422387186
64=1437773636
121=1
62=0
84=0
85=1
88=1
101=0
7=0
26=0
97=1
38=1
9=1
47=1
54=0
98=0.00000
71=0.00000
42=0.36777
113=0
45=0
103=1
93=1
80=0
37=
11=check_by_nrpe_log!/var/log/messages!Error!most_recent!!!!!
86=7.00000
109=2.00000
209=xi_timeperiod_24x7
262=XIWIZARD
  USED1: 511, BYTES: 0, LINES: 0
BYTESREAD: 17
RAWBUF: :0:sshproxy
999


  USED1: 17, BYTES: 0, LINES: 0
BYTESREAD: 511
RAWBUF:
213:
1=1202
2=0
3=0
4=1440182644.830575
53=wltr-365-dap14
114=Swap
95=SWAP OK - 100% free (16383 MB out of 16383 MB)
125=
99=swap=16383MB;13107;11468;0;16383
27=0
51=1
115=1
25=1
76=3
61=1440182644
83=1440182944
12=0
63=1432780484
57=1432780484
56=0
66=1440182644
70=0
67=0
64=1432751993
121=1
62=0
84=0
85=1
88=1
101=0
7=0
26=0
97=1
38=1
9=1
47=1
54=0
98=0.00000
71=0.00000
42=0.31646
113=0
45=0
103=1
93=1
80=0
37=
11=check_by_nrpe_swap!80%!70%!!!!!!
86=5.00000
109=2.00000
209=xi_timeperiod_24x7
262=XIWIZAR
  USED1: 511, BYTES: 0, LINES: 0
BYTESREAD: 18
RAWBUF: D:0:sshproxy
999


  USED1: 18, BYTES: 0, LINES: 0
BYTESREAD: 511
RAWBUF:
213:
1=1202
2=0
3=0
4=1440182644.830657
53=wltr-365-dap14
114=Swap
95=SWAP OK - 100% free (16383 MB out of 16383 MB)
125=
99=swap=16383MB;13107;11468;0;16383
27=0
51=1
115=1
25=1
76=3
61=1440182644
83=1440182944
12=0
63=1432780484
57=1432780484
56=0
66=1440182644
70=0
67=0
64=1432751993
121=1
62=0
84=0
85=1
88=1
101=0
7=0
26=0
97=1
38=1
9=1
47=1
54=0
98=0.00000
71=0.00000
42=0.31646
113=0
45=0
103=1
93=1
80=0
37=
11=check_by_nrpe_swap!80%!70%!!!!!!
86=5.00000
109=2.00000
209=xi_timeperiod_24x7
262=XIWIZAR
  USED1: 511, BYTES: 0, LINES: 0
BYTESREAD: 511
RAWBUF: D:0:sshproxy
999


213:
1=1202
2=0
3=0
4=1440182644.832927
53=virt-chi-pdb01
114=PROD - Concurrent Manager
95=Concurrent Manager status ok
125=
99=
27=0
51=1
115=1
25=1
76=3
61=1440182644
83=1440182944
12=0
63=1438097652
57=1438097652
56=0
66=1440182644
70=0
67=1438067276
64=1438069064
121=1
62=0
84=0
85=1
88=1
101=0
7=0
26=0
97=1
38=1
9=1
47=1
54=0
98=0.00000
71=0.00000
42=0.10836
113=0
45=0
103=1
93=1
80=0
37=
11=check_ccm!PROD!1527!itcnimbus!itcnimmon!!!!
86=5.00000
109=2.00000
209=xi_timeperiod_24x7
99
  USED1: 511, BYTES: 0, LINES: 0
BYTESREAD: 426
RAWBUF: 9


213:
1=1202
2=0
3=0
4=1440182644.832972
53=virt-chi-pdb01
114=PROD - Concurrent Manager
95=Concurrent Manager status ok
125=
99=
27=0
51=1
115=1
25=1
76=3
61=1440182644
83=1440182944
12=0
63=1438097652
57=1438097652
56=0
66=1440182644
70=0
67=1438067276
64=1438069064
121=1
62=0
84=0
85=1
88=1
101=0
7=0
26=0
97=1
38=1
9=1
47=1
54=0
98=0.00000
71=0.00000
42=0.10836
113=0
45=0
103=1
93=1
80=0
37=
11=check_ccm!PROD!1527!it
  USED1: 426, BYTES: 0, LINES: 0
BYTESREAD: 73
RAWBUF: cnimbus!itcnimmon!!!!
86=5.00000
109=2.00000
209=xi_timeperiod_24x7
999


  USED1: 73, BYTES: 0, LINES: 0
BYTESREAD: 511
RAWBUF:
213:
1=1202
2=0
3=0
4=1440182644.882980
53=rbns-chi-ftp1
114=Zombie Processes
95=PROCS OK: 0 processes with STATE = Z
125=
99=procs=0;5;10;0;
27=0
51=1
115=1
25=1
76=3
61=1440182644
83=1440182944
12=0
63=1439127565
57=1439127565
56=0
66=1440182644
70=0
67=0
64=1439098977
121=1
62=0
84=0
85=1
88=1
101=0
7=0
26=0
97=1
38=1
9=1
47=1
54=0
98=0.00000
71=0.00000
42=0.04949
113=0
45=0
103=1
93=1
80=0
37=
11=check_by_nrpe_procs!5!10!-s Z!!!!!
86=5.00000
109=2.00000
209=xi_timeperiod_24x7
262=XIWIZARD:0:sshproxy
9
  USED1: 511, BYTES: 0, LINES: 0
BYTESREAD: 4
RAWBUF: 99


  USED1: 4, BYTES: 0, LINES: 0
BYTESREAD: 511
RAWBUF:
213:
1=1202
2=0
3=0
4=1440182644.883085
53=rbns-chi-ftp1
114=Zombie Processes
95=PROCS OK: 0 processes with STATE = Z
125=
99=procs=0;5;10;0;
27=0
51=1
115=1
25=1
76=3
61=1440182644
83=1440182944
12=0
63=1439127565
57=1439127565
56=0
66=1440182644
70=0
67=0
64=1439098977
121=1
62=0
84=0
85=1
88=1
101=0
7=0
26=0
97=1
38=1
9=1
47=1
54=0
98=0.00000
71=0.00000
42=0.04949
113=0
45=0

jfrickson · Post by **jfrickson** » Fri Aug 21, 2015 1:54 pm

Hmm, guess I'll need more than that. If the file is less than 20MB compressed, just upload the whole thing. Otherwise, grab a good-sized chunk of it and send that.

I take that back. There should be enough info in there. I'll take a closer look and let you know if I find anything.

Post by **BanditBBS** » Fri Aug 21, 2015 1:57 pm

PM me a email address(although I could guess it) and I'll share the 200MB compressed log out to you from my OneDrive

tmcdonald · Post by **tmcdonald** » Fri Aug 21, 2015 2:25 pm

Bandit, since you're able to replicate the issue, would you be willing/able to run any beta or possible fixes we get?

Post by **BanditBBS** » Fri Aug 21, 2015 2:28 pm

tmcdonald wrote:Bandit, since you're able to replicate the issue, would you be willing/able to run any beta or possible fixes we get?

Instantly will put into prod.

Nagios Support Forum

NDO2DB Issue out of the blue

Re: NDO2DB Issue out of the blue

Re: NDO2DB Issue out of the blue

Re: NDO2DB Issue out of the blue

Re: NDO2DB Issue out of the blue

Re: NDO2DB Issue out of the blue

Re: NDO2DB Issue out of the blue

Re: NDO2DB Issue out of the blue

Re: NDO2DB Issue out of the blue

Re: NDO2DB Issue out of the blue

Re: NDO2DB Issue out of the blue