MariaDB displaying special characters in Nagios DB

[ssax]

What is the output of these commands on the server?

Code: Select all

ps aux | grep nagios.cfg
ipcs -q
ps aux | grep ndo

Additionally, are you seeing any errors in your /var/log/mysqld.log or /var/log/mariadb/mariadb.log that could be related?

If you not seeing anything, you may want to enable debug logging for ndo2db by editing your /usr/local/nagios/etc/ndo2db.cfg file and set:

Code: Select all

debug_level=-1
debug_verbosity=2

Then restart the services:

Code: Select all

service nagios stop
service ndo2db restart
service nagios start

Then you will see the debug output logged in /usr/local/nagios/var/ndo2db.debug

It has to be ndo2db that is having issues, on the nagios restart it rebuilds the data in nagios_objects table so it has to come from there.

[DonGately]

Hello there and thank you for your reply.

Here are the outputs you requested

ps aux | grep nagios.cfg

Code: Select all

nagios   10063  3.3  0.2  57576 26384 ?        Ss   06:28   3:10 /usr/sbin/nagios -d /etc/nagios/nagios.cfg
nagios   10197  0.0  0.2  57308 24564 ?        S    06:28   0:00 /usr/sbin/nagios -d /etc/nagios/nagios.cfg


ipcs -q

Code: Select all

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages
0x7c000040 5898240    nagios     600        757760       740


ps aux | grep ndo

Code: Select all

nagios    9838  0.0  0.0  56148   924 ?        Ss   06:28   0:00 /usr/sbin/ndo2db -c /etc/nagios/ndo2db.cfg
nagios   10076  0.3  0.0 129880  2052 ?        S    06:28   0:17 /usr/sbin/ndo2db -c /etc/nagios/ndo2db.cfg
nagios   10077 10.5  0.0 138004 10324 ?        S    06:28  10:05 /usr/sbin/ndo2db -c /etc/nagios/ndo2db.cfg


I can't see anything in the mariadb.log File that would reference to the behaviour I'm experiencing. I have also set debug_level to 1 and debug_verbosity to 2. I looked through the logfile (ndo2db.log) but I understand too little to see anything related to the problems I'm having. I don't see any strange characters like in the databse though.

Here's a snippet of the ndo2db.log file.

Code: Select all

206:
1=701
2=0
3=0
4=1544172522.497826
53=servername
114=Ser
[1544172522.497900] [001.2] [pid=22073] Handling:
[1544172522.497915] [001.2] [pid=22073] Handling: 213:
[1544172522.497928] [001.2] [pid=22073] Handling: 1=1202
[1544172522.497940] [001.2] [pid=22073] Handling: 2=0
[1544172522.497951] [001.2] [pid=22073] Handling: 3=0
[1544172522.497961] [001.2] [pid=22073] Handling: 4=1544172522.497789
[1544172522.497974] [001.2] [pid=22073] Handling: 53=servername
[1544172522.497984] [001.2] [pid=22073] Handling: 114=Server Sitzungen
[1544172522.497995] [001.2] [pid=22073] Handling: 95=Terminal Server Sitzungen - 37
[1544172522.498006] [001.2] [pid=22073] Handling: 125=
[1544172522.498017] [001.2] [pid=22073] Handling: 99=Sitzungen=37
[1544172522.498028] [001.2] [pid=22073] Handling: 27=0
[1544172522.498038] [001.2] [pid=22073] Handling: 51=1
[1544172522.498049] [001.2] [pid=22073] Handling: 115=1
[1544172522.498059] [001.2] [pid=22073] Handling: 25=1
[1544172522.498070] [001.2] [pid=22073] Handling: 76=3
[1544172522.498080] [001.2] [pid=22073] Handling: 61=1544172522
[1544172522.498091] [001.2] [pid=22073] Handling: 83=1544172582
[1544172522.498101] [001.2] [pid=22073] Handling: 12=0
[1544172522.498112] [001.2] [pid=22073] Handling: 63=1543759204
[1544172522.498122] [001.2] [pid=22073] Handling: 57=1543759204
[1544172522.498133] [001.2] [pid=22073] Handling: 56=0
[1544172522.498143] [001.2] [pid=22073] Handling: 66=1544172522
[1544172522.498154] [001.2] [pid=22073] Handling: 70=1543759204
[1544172522.498164] [001.2] [pid=22073] Handling: 67=0
[1544172522.498175] [001.2] [pid=22073] Handling: 64=1538197358
[1544172522.498185] [001.2] [pid=22073] Handling: 121=1
[1544172522.498196] [001.2] [pid=22073] Handling: 62=0
[1544172522.498209] [001.2] [pid=22073] Handling: 84=0
[1544172522.498219] [001.2] [pid=22073] Handling: 85=0
[1544172522.498232] [001.2] [pid=22073] Handling: 88=1
[1544172522.498269] [001.2] [pid=22073] Handling: 101=0
[1544172522.498286] [001.2] [pid=22073] Handling: 7=0
[1544172522.498297] [001.2] [pid=22073] Handling: 26=0
[1544172522.498308] [001.2] [pid=22073] Handling: 97=1
[1544172522.498318] [001.2] [pid=22073] Handling: 38=1
[1544172522.498329] [001.2] [pid=22073] Handling: 9=1
[1544172522.498339] [001.2] [pid=22073] Handling: 47=1
[1544172522.498350] [001.2] [pid=22073] Handling: 54=0
[1544172522.498360] [001.2] [pid=22073] Handling: 98=0.00000
[1544172522.498371] [001.2] [pid=22073] Handling: 71=0.00000
[1544172522.498381] [001.2] [pid=22073] Handling: 42=0.29980
[1544172522.498391] [001.2] [pid=22073] Handling: 113=0
[1544172522.498402] [001.2] [pid=22073] Handling: 45=0
[1544172522.498412] [001.2] [pid=22073] Handling: 103=1
[1544172522.498422] [001.2] [pid=22073] Handling: 93=1
[1544172522.498433] [001.2] [pid=22073] Handling: 80=0
[1544172522.498443] [001.2] [pid=22073] Handling: 37=
[1544172522.498454] [001.2] [pid=22073] Handling: 11=check_TS_users
[1544172522.498464] [001.2] [pid=22073] Handling: 86=1.00000
[1544172522.498475] [001.2] [pid=22073] Handling: 109=1.00000
[1544172522.498502] [001.2] [pid=22073] Handling: 209=24x7
[1544172522.498514] [001.2] [pid=22073] Handling: 999
[1544172522.499364] [001.2] [pid=22073] Handling:
[1544172522.499380] [001.2] [pid=22073] Handling:
[1544172522.499391] [001.2] [pid=22073] Handling: 206:
[1544172522.499404] [001.2] [pid=22073] Handling: 1=701
[1544172522.499415] [001.2] [pid=22073] Handling: 2=0
[1544172522.499425] [001.2] [pid=22073] Handling: 3=0
[1544172522.499436] [001.2] [pid=22073] Handling: 4=1544172522.497826
[1544172522.499446] [001.2] [pid=22073] Handling: 53=servername
[1544172522.499460] [001.2] [pid=22073] Queue Message: ver Sitzungen
12=0
25=1
76=3
121=1
118=0
123=360
127=
13=
14=
117=1544172522.197882
33=1544172522.497682
32=0
42=0.29980
71=0.00000
110=0
95=Terminal Server Sitzungen - 37
125=
99=Sitzungen=37
999


213:
1=1202
2=0
3=0
4=1544172522.497840
53=servername
114=Server Sitzungen
95=Terminal Server Sitzungen - 37
125=
99=Sitzungen=37
27=0
51=1
115=1
25=1
76=3
61=1544172522
83=1544172582
12=0
63=1543759204
57=1543759204
56=0
66=1544172522
70=1543759204
67=0
64=1538197358
121=1
62=0
84=0
85=0
88=1
101=0
7=0
26=0
97=1
[1544172522.499469] [001.2] [pid=22073] Handling: 114=Server Sitzungen
[1544172522.499495] [001.2] [pid=22073] Handling: 12=0
[1544172522.499507] [001.2] [pid=22073] Handling: 25=1
[1544172522.499518] [001.2] [pid=22073] Handling: 76=3
[1544172522.499529] [001.2] [pid=22073] Handling: 121=1
[1544172522.499542] [001.2] [pid=22073] Handling: 118=0
[1544172522.499555] [001.2] [pid=22073] Handling: 123=360
[1544172522.499565] [001.2] [pid=22073] Handling: 127=
[1544172522.499576] [001.2] [pid=22073] Handling: 13=
[1544172522.499587] [001.2] [pid=22073] Handling: 14=
[1544172522.499602] [001.2] [pid=22073] Handling: 117=1544172522.197882
[1544172522.499613] [001.2] [pid=22073] Handling: 33=1544172522.497682
[1544172522.499623] [001.2] [pid=22073] Handling: 32=0
[1544172522.499634] [001.2] [pid=22073] Handling: 42=0.29980
[1544172522.499644] [001.2] [pid=22073] Handling: 71=0.00000
[1544172522.499655] [001.2] [pid=22073] Handling: 110=0
[1544172522.499665] [001.2] [pid=22073] Handling: 95=Terminal Server Sitzungen - 37
[1544172522.499676] [001.2] [pid=22073] Handling: 125=
[1544172522.499686] [001.2] [pid=22073] Handling: 99=Sitzungen=37
[1544172522.499697] [001.2] [pid=22073] Handling: 999
[1544172522.500271] [001.2] [pid=22073] Handling:
[1544172522.500287] [001.2] [pid=22073] Handling:
[1544172522.500297] [001.2] [pid=22073] Handling: 213:
[1544172522.500309] [001.2] [pid=22073] Handling: 1=1202
[1544172522.500320] [001.2] [pid=22073] Handling: 2=0
[1544172522.500330] [001.2] [pid=22073] Handling: 3=0
[1544172522.500341] [001.2] [pid=22073] Handling: 4=1544172522.497840
[1544172522.500352] [001.2] [pid=22073] Handling: 53=servername
[1544172522.500362] [001.2] [pid=22073] Handling: 114=Server Sitzungen
[1544172522.500373] [001.2] [pid=22073] Handling: 95=Terminal Server Sitzungen - 37
[1544172522.500384] [001.2] [pid=22073] Handling: 125=
[1544172522.500398] [001.2] [pid=22073] Handling: 99=Sitzungen=37
[1544172522.500410] [001.2] [pid=22073] Handling: 27=0
[1544172522.500420] [001.2] [pid=22073] Handling: 51=1
[1544172522.500430] [001.2] [pid=22073] Handling: 115=1
[1544172522.500441] [001.2] [pid=22073] Handling: 25=1
[1544172522.500451] [001.2] [pid=22073] Handling: 76=3
[1544172522.500462] [001.2] [pid=22073] Handling: 61=1544172522
[1544172522.500472] [001.2] [pid=22073] Handling: 83=1544172582
[1544172522.500500] [001.2] [pid=22073] Handling: 12=0
[1544172522.500513] [001.2] [pid=22073] Handling: 63=1543759204
[1544172522.500524] [001.2] [pid=22073] Handling: 57=1543759204
[1544172522.500534] [001.2] [pid=22073] Handling: 56=0
[1544172522.500545] [001.2] [pid=22073] Handling: 66=1544172522
[1544172522.500555] [001.2] [pid=22073] Handling: 70=1543759204
[1544172522.500566] [001.2] [pid=22073] Handling: 67=0
[1544172522.500576] [001.2] [pid=22073] Handling: 64=1538197358
[1544172522.500586] [001.2] [pid=22073] Handling: 121=1
[1544172522.500597] [001.2] [pid=22073] Handling: 62=0
[1544172522.500607] [001.2] [pid=22073] Handling: 84=0
[1544172522.500617] [001.2] [pid=22073] Handling: 85=0
[1544172522.500627] [001.2] [pid=22073] Handling: 88=1
[1544172522.500638] [001.2] [pid=22073] Handling: 101=0
[1544172522.500648] [001.2] [pid=22073] Handling: 7=0
[1544172522.500659] [001.2] [pid=22073] Handling: 26=0
[1544172522.500672] [001.2] [pid=22073] Queue Message:
38=1
9=1
47=1
54=0
98=0.00000
71=0.00000
42=0.29980
113=0
45=0
103=1
93=1
80=0
37=
11=check_TS_users
86=1.00000
109=1.00000
209=24x7
999


[1544172522.500680] [001.2] [pid=22073] Handling: 97=1
[1544172522.500690] [001.2] [pid=22073] Handling: 38=1
[1544172522.500700] [001.2] [pid=22073] Handling: 9=1
[1544172522.500711] [001.2] [pid=22073] Handling: 47=1
[1544172522.500721] [001.2] [pid=22073] Handling: 54=0
[1544172522.500732] [001.2] [pid=22073] Handling: 98=0.00000
[1544172522.500742] [001.2] [pid=22073] Handling: 71=0.00000
[1544172522.500752] [001.2] [pid=22073] Handling: 42=0.29980
[1544172522.500763] [001.2] [pid=22073] Handling: 113=0
[1544172522.500773] [001.2] [pid=22073] Handling: 45=0
[1544172522.500783] [001.2] [pid=22073] Handling: 103=1
[1544172522.500794] [001.2] [pid=22073] Handling: 93=1
[1544172522.500804] [001.2] [pid=22073] Handling: 80=0
[1544172522.500814] [001.2] [pid=22073] Handling: 37=
[1544172522.500828] [001.2] [pid=22073] Handling: 11=check_TS_users
[1544172522.500838] [001.2] [pid=22073] Handling: 86=1.00000
[1544172522.500863] [001.2] [pid=22073] Handling: 109=1.00000
[1544172522.500873] [001.2] [pid=22073] Handling: 209=24x7
[1544172522.500884] [001.2] [pid=22073] Handling: 999
[1544172522.501435] [001.2] [pid=22073] Handling:
[1544172522.511207] [001.2] [pid=22073] Queue Message:
213:
1=1202
2=0
3=0
4=1544172522.511114
53=servername1
114=Diskspace /
95=DISK OK - free space: / 6006 MB (58% inode=99%):
125=
99=/=4223MB;9718;9923;0;10230
27=0
51=1
115=1
25=1
76=5
61=1544172522
83=1544173122
12=0
63=1540129654
57=1540129654
56=0
66=1544172522
70=0
67=1540129654
64=0
121=1
62=0
84=0
85=0
88=0
101=0
7=0
26=0
97=1
38=1
9=1
47=1
54=0
98=0.00000
71=0.00047
42=0.03428
113=0
45=0
103=0
93=1
80=1
37=
11=check_nrpe1!check_disk!5% 3% /
86=10.00000
109=2.00000
209=24x7
999


[1544172522.511225] [001.2] [pid=22073] Handling:
[1544172522.511261] [001.2] [pid=22073] Handling: 213:
[1544172522.511275] [001.2] [pid=22073] Handling: 1=1202
[1544172522.511287] [001.2] [pid=22073] Handling: 2=0
[1544172522.511297] [001.2] [pid=22073] Handling: 3=0
[1544172522.511308] [001.2] [pid=22073] Handling: 4=1544172522.511114
[1544172522.511319] [001.2] [pid=22073] Handling: 53=servername1
[1544172522.511330] [001.2] [pid=22073] Handling: 114=Diskspace /
[1544172522.511340] [001.2] [pid=22073] Handling: 95=DISK OK - free space: / 6006 MB (58% inode=99%):
[1544172522.511351] [001.2] [pid=22073] Handling: 125=
[1544172522.511362] [001.2] [pid=22073] Handling: 99=/=4223MB;9718;9923;0;10230
[1544172522.511373] [001.2] [pid=22073] Handling: 27=0
[1544172522.511383] [001.2] [pid=22073] Handling: 51=1
[1544172522.511398] [001.2] [pid=22073] Handling: 115=1
[1544172522.511410] [001.2] [pid=22073] Handling: 25=1
[1544172522.511420] [001.2] [pid=22073] Handling: 76=5
[1544172522.511430] [001.2] [pid=22073] Handling: 61=1544172522
[1544172522.511441] [001.2] [pid=22073] Handling: 83=1544173122
[1544172522.511451] [001.2] [pid=22073] Handling: 12=0
[1544172522.511462] [001.2] [pid=22073] Handling: 63=1540129654
[1544172522.511472] [001.2] [pid=22073] Handling: 57=1540129654
[1544172522.511499] [001.2] [pid=22073] Handling: 56=0
[1544172522.511513] [001.2] [pid=22073] Handling: 66=1544172522
[1544172522.511523] [001.2] [pid=22073] Handling: 70=0
[1544172522.511534] [001.2] [pid=22073] Handling: 67=1540129654
[1544172522.511547] [001.2] [pid=22073] Handling: 64=0
[1544172522.511558] [001.2] [pid=22073] Handling: 121=1
[1544172522.511568] [001.2] [pid=22073] Handling: 62=0
[1544172522.511579] [001.2] [pid=22073] Handling: 84=0
[1544172522.511589] [001.2] [pid=22073] Handling: 85=0
[1544172522.511599] [001.2] [pid=22073] Handling: 88=0
[1544172522.511610] [001.2] [pid=22073] Handling: 101=0
[1544172522.511620] [001.2] [pid=22073] Handling: 7=0
[1544172522.511631] [001.2] [pid=22073] Handling: 26=0
[1544172522.511641] [001.2] [pid=22073] Handling: 97=1
[1544172522.511651] [001.2] [pid=22073] Handling: 38=1
[1544172522.511662] [001.2] [pid=22073] Handling: 9=1
[1544172522.511672] [001.2] [pid=22073] Handling: 47=1
[1544172522.511682] [001.2] [pid=22073] Handling: 54=0
[1544172522.511693] [001.2] [pid=22073] Handling: 98=0.00000
[1544172522.511703] [001.2] [pid=22073] Handling: 71=0.00047
[1544172522.511713] [001.2] [pid=22073] Handling: 42=0.03428
[1544172522.511724] [001.2] [pid=22073] Handling: 113=0
[1544172522.511734] [001.2] [pid=22073] Handling: 45=0
[1544172522.511745] [001.2] [pid=22073] Handling: 103=0
[1544172522.511755] [001.2] [pid=22073] Handling: 93=1
[1544172522.511765] [001.2] [pid=22073] Handling: 80=1
[1544172522.511775] [001.2] [pid=22073] Handling: 37=
[1544172522.511786] [001.2] [pid=22073] Handling: 11=check_nrpe1!check_disk!5% 3% /
[1544172522.511796] [001.2] [pid=22073] Handling: 86=10.00000
[1544172522.511807] [001.2] [pid=22073] Handling: 109=2.00000
[1544172522.511817] [001.2] [pid=22073] Handling: 209=24x7
[1544172522.511827] [001.2] [pid=22073] Handling: 999
[1544172522.513069] [001.2] [pid=22073] Handling:
[1544172522.513096] [001.2] [pid=22073] Queue Message:
206:
1=701
2=0
3=0
4=1544172522.511154
53=servername1
114=Diskspace /
12=0
25=1
76=5
121=1
118=0
123=360
127=
13=
14=
117=1544172522.476715
33=1544172522.510996
32=0
42=0.03428
71=0.00047
110=0
95=DISK OK - free space: / 6006 MB (58% inode=99%):
125=
99=/=4223MB;9718;9923;0;10230
999


[1544172522.513106] [001.2] [pid=22073] Handling:
[1544172522.513116] [001.2] [pid=22073] Handling: 206:
[1544172522.513128] [001.2] [pid=22073] Handling: 1=701
[1544172522.513140] [001.2] [pid=22073] Handling: 2=0
[1544172522.513150] [001.2] [pid=22073] Handling: 3=0
[1544172522.513161] [001.2] [pid=22073] Handling: 4=1544172522.511154
[1544172522.513172] [001.2] [pid=22073] Handling: 53=servername1
[1544172522.513182] [001.2] [pid=22073] Handling: 114=Diskspace /
[1544172522.513193] [001.2] [pid=22073] Handling: 12=0
[1544172522.513203] [001.2] [pid=22073] Handling: 25=1
[1544172522.513214] [001.2] [pid=22073] Handling: 76=5
[1544172522.513224] [001.2] [pid=22073] Handling: 121=1
[1544172522.513271] [001.2] [pid=22073] Handling: 118=0
[1544172522.513285] [001.2] [pid=22073] Handling: 123=360
[1544172522.513298] [001.2] [pid=22073] Handling: 127=
[1544172522.513306] [001.2] [pid=22073] Handling: 13=
[1544172522.513316] [001.2] [pid=22073] Handling: 14=
[1544172522.513327] [001.2] [pid=22073] Handling: 117=1544172522.476715
[1544172522.513337] [001.2] [pid=22073] Handling: 33=1544172522.510996
[1544172522.513348] [001.2] [pid=22073] Handling: 32=0
[1544172522.513358] [001.2] [pid=22073] Handling: 42=0.03428
[1544172522.513369] [001.2] [pid=22073] Handling: 71=0.00047
[1544172522.513379] [001.2] [pid=22073] Handling: 110=0
[1544172522.513389] [001.2] [pid=22073] Handling: 95=DISK OK - free space: / 6006 MB (58% inode=99%):
[1544172522.513406] [001.2] [pid=22073] Handling: 125=
[1544172522.513417] [001.2] [pid=22073] Handling: 99=/=4223MB;9718;9923;0;10230
[1544172522.513428] [001.2] [pid=22073] Handling: 999
[1544172522.514425] [001.2] [pid=22073] Handling:
[1544172522.514470] [001.2] [pid=22073] Queue Message:
213:
1=1202
2=0
3=0
4=1544172522.511168
53=servername1
114=Diskspace /
95=DISK OK - free space: / 6006 MB (58% inode=99%):
125=
99=/=4223MB;9718;9923;0;10230
27=0
51=1
115=1
25=1
76=5
61=1544172522
83=1544173122
12=0
63=1540129654
57=1540129654
56=0
66=1544172522
70=0
67=1540129654
64=0
121=1
62=0
84=0
85=0
88=0
101=0
7=0
26=0
97=1
38=1
9=1
47=1
54=0
98=0.00000
71=0.00047
42=0.03428
113=0
45=0
103=0
93=1
80=1
37=
11=check_nrpe1!check_disk!5% 3% /
86=10.00000
109=2.00000
209=24x7
999


[1544172522.514507] [001.2] [pid=22073] Handling:
[1544172522.514524] [001.2] [pid=22073] Handling: 213:
[1544172522.514548] [001.2] [pid=22073] Handling: 1=1202
[1544172522.514569] [001.2] [pid=22073] Handling: 2=0
[1544172522.514601] [001.2] [pid=22073] Handling: 3=0
[1544172522.514636] [001.2] [pid=22073] Handling: 4=1544172522.511168
[1544172522.514654] [001.2] [pid=22073] Handling: 53=servername1
[1544172522.514670] [001.2] [pid=22073] Handling: 114=Diskspace /
[1544172522.514705] [001.2] [pid=22073] Handling: 95=DISK OK - free space: / 6006 MB (58% inode=99%):
[1544172522.514720] [001.2] [pid=22073] Handling: 125=
[1544172522.514731] [001.2] [pid=22073] Handling: 99=/=4223MB;9718;9923;0;10230
[1544172522.514742] [001.2] [pid=22073] Handling: 27=0
[1544172522.514753] [001.2] [pid=22073] Handling: 51=1
[1544172522.514789] [001.2] [pid=22073] Handling: 115=1
[1544172522.514802] [001.2] [pid=22073] Handling: 25=1
[1544172522.514813] [001.2] [pid=22073] Handling: 76=5
[1544172522.514823] [001.2] [pid=22073] Handling: 61=1544172522
[1544172522.514834] [001.2] [pid=22073] Handling: 83=1544173122
[1544172522.514845] [001.2] [pid=22073] Handling: 12=0
[1544172522.514855] [001.2] [pid=22073] Handling: 63=1540129654
[1544172522.514870] [001.2] [pid=22073] Handling: 57=1540129654
[1544172522.514881] [001.2] [pid=22073] Handling: 56=0
[1544172522.514891] [001.2] [pid=22073] Handling: 66=1544172522
[1544172522.514902] [001.2] [pid=22073] Handling: 70=0
[1544172522.514965] [001.2] [pid=22073] Handling: 67=1540129654
[1544172522.514984] [001.2] [pid=22073] Handling: 64=0
[1544172522.514998] [001.2] [pid=22073] Handling: 121=1
[1544172522.515010] [001.2] [pid=22073] Handling: 62=0
[1544172522.515041] [001.2] [pid=22073] Handling: 84=0
[1544172522.515061] [001.2] [pid=22073] Handling: 85=0
[1544172522.515079] [001.2] [pid=22073] Handling: 88=0
[1544172522.515096] [001.2] [pid=22073] Handling: 101=0
[1544172522.515115] [001.2] [pid=22073] Handling: 7=0
[1544172522.515135] [001.2] [pid=22073] Handling: 26=0
[1544172522.515151] [001.2] [pid=22073] Handling: 97=1
[1544172522.515165] [001.2] [pid=22073] Handling: 38=1
[1544172522.515175] [001.2] [pid=22073] Handling: 9=1
[1544172522.515209] [001.2] [pid=22073] Handling: 47=1
[1544172522.515223] [001.2] [pid=22073] Handling: 54=0
[1544172522.515234] [001.2] [pid=22073] Handling: 98=0.00000
[1544172522.515270] [001.2] [pid=22073] Handling: 71=0.00047
[1544172522.515282] [001.2] [pid=22073] Handling: 42=0.03428
[1544172522.515305] [001.2] [pid=22073] Handling: 113=0
[1544172522.515317] [001.2] [pid=22073] Handling: 45=0
[1544172522.515327] [001.2] [pid=22073] Handling: 103=0
[1544172522.515338] [001.2] [pid=22073] Handling: 93=1
[1544172522.515348] [001.2] [pid=22073] Handling: 80=1
[1544172522.515359] [001.2] [pid=22073] Handling: 37=
[1544172522.515369] [001.2] [pid=22073] Handling: 11=check_nrpe1!check_disk!5% 3% /
[1544172522.515380] [001.2] [pid=22073] Handling: 86=10.00000
[1544172522.515391] [001.2] [pid=22073] Handling: 109=2.00000
[1544172522.515401] [001.2] [pid=22073] Handling: 209=24x7
[1544172522.515412] [001.2] [pid=22073] H


I have also attached the mariadb logfile.

Edit:
I accidentally set debug_level to 1, not -1 as you suggested. Now it is on -1 and here's some of the output of ndo2db.log

Code: Select all

213:
1=1202
2=0
3=0
4=1544189928.512939
53=servername2
114=Memory Usage
95=Memory usage: total:7167.61 MB - used: 2977.62 MB (42%) - f
[1544189928.513850] [001.2] [pid=30772] Handling: 86=5.00000
[1544189928.513860] [001.2] [pid=30772] Handling: 109=1.00000
[1544189928.513871] [001.2] [pid=30772] Handling: 209=24x7
[1544189928.513885] [001.2] [pid=30772] Handling: 999
[1544189928.513929] [002.0] [pid=30772] INSERT INTO nagios_servicestatus SET instance_id='1', service_object_id='37818', status_update_time=FROM_UNIXTIME(1544189928), output='Memory usage: total:7167\.61 MB - used: 2977\.62 MB \(42%\) - free: 4189\.99 MB \(58%\)', long_output='', perfdata='\'Memory usage\'=2977\.62MB;6737\.56;6880\.91;0\.00;7167\.61', current_state='0', has_been_checked='1', should_be_scheduled='1', current_check_attempt='1', max_check_attempts='3', last_check=FROM_UNIXTIME(1544189928), next_check=FROM_UNIXTIME(1544190228), check_type='0', last_state_change=FROM_UNIXTIME(1543231283), last_hard_state_change=FROM_UNIXTIME(1542968726), last_hard_state='0', last_time_ok=FROM_UNIXTIME(1544189928), last_time_warning=FROM_UNIXTIME(0), last_time_unknown=FROM_UNIXTIME(0), last_time_critical=FROM_UNIXTIME(1543231283), state_type='1', last_notification=FROM_UNIXTIME(0), next_notification=FROM_UNIXTIME(0), no_more_notifications='0', notifications_enabled='1', problem_has_been_acknowledged='0', acknowledgement_type='0', current_notification_number='0', passive_checks_enabled='1', active_checks_enabled='1', event_handler_enabled='1', flap_detection_enabled='1', is_flapping='0', percent_state_change='0.000000', latency='0.000000', execution_time='0.004050', scheduled_downtime_depth='0', failure_prediction_enabled='0', process_performance_data='1', obsess_over_service='1', modified_service_attributes='0', event_handler='', check_command='check_nt!MEMUSE!-w 94 -c 96', normal_check_interval='5.000000', retry_check_interval='1.000000', check_timeperiod_object_id='2' ON DUPLICATE KEY UPDATE instance_id='1', service_object_id='37818', status_update_time=FROM_UNIXTIME(1544189928), output='Memory usage: total:7167\.61 MB - used: 2977\.62 MB \(42%\) - free: 4189\.99 MB \(58%\)', long_output='', perfdata='\'Memory usage\'=2977\.62MB;6737\.56;6880\.91;0\.00;7167\.61', current_state='0', has_been_checked='1', should_be_scheduled='1', current_check_attempt='1', max_check_attempts='3', last_check=FROM_UNIXTIME(1544189928), next_check=FROM_UNIXTIME(1544190228), check_type='0', last_state_change=FROM_UNIXTIME(1543231283), last_hard_state_change=FROM_UNIXTIME(1542968726), last_hard_state='0', last_time_ok=FROM_UNIXTIME(1544189928), last_time_warning=FROM_UNIXTIME(0), last_time_unknown=FROM_UNIXTIME(0), last_time_critical=FROM_UNIXTIME(1543231283), state_type='1', last_notification=FROM_UNIXTIME(0), next_notification=FROM_UNIXTIME(0), no_more_notifications='0', notifications_enabled='1', problem_has_been_acknowledged='0', acknowledgement_type='0', current_notification_number='0', passive_checks_enabled='1', active_checks_enabled='1', event_handler_enabled='1', flap_detection_enabled='1', is_flapping='0', percent_state_change='0.000000', latency='0.000000', execution_time='0.004050', scheduled_downtime_depth='0', failure_prediction_enabled='0', process_performance_data='1', obsess_over_service='1', modified_service_attributes='0', event_handler='', check_command='check_nt!MEMUSE!-w 94 -c 96', normal_check_interval='5.000000', retry_check_interval='1.000000', check_timeperiod_object_id='2'
[1544189928.514905] [001.2] [pid=30772] Handling:
[1544189928.514925] [001.2] [pid=30772] Handling:
[1544189928.514937] [001.2] [pid=30772] Handling: 206:
[1544189928.514949] [001.2] [pid=30772] Handling: 1=701
[1544189928.514960] [001.2] [pid=30772] Handling: 2=0
[1544189928.514972] [001.2] [pid=30772] Handling: 3=0
[1544189928.514983] [001.2] [pid=30772] Handling: 4=1544189928.512926
[1544189928.515016] [001.2] [pid=30772] Handling: 53=servername2
[1544189928.515031] [001.2] [pid=30772] Handling: 114=Memory Usage
[1544189928.515043] [001.2] [pid=30772] Handling: 12=0
[1544189928.515054] [001.2] [pid=30772] Handling: 25=1
[1544189928.515065] [001.2] [pid=30772] Handling: 76=3
[1544189928.515076] [001.2] [pid=30772] Handling: 121=1
[1544189928.515087] [001.2] [pid=30772] Handling: 118=0
[1544189928.515098] [001.2] [pid=30772] Handling: 123=360
[1544189928.515109] [001.2] [pid=30772] Handling: 127=
[1544189928.515120] [001.2] [pid=30772] Handling: 13=
[1544189928.515156] [001.2] [pid=30772] Handling: 14=
[1544189928.515168] [001.2] [pid=30772] Handling: 117=1544189928.508743
[1544189928.515180] [001.2] [pid=30772] Handling: 33=1544189928.512794
[1544189928.515191] [001.2] [pid=30772] Handling: 32=0
[1544189928.515207] [001.2] [pid=30772] Handling: 42=0.00405
[1544189928.515224] [001.2] [pid=30772] Handling: 71=0.00000
[1544189928.515235] [001.2] [pid=30772] Handling: 110=0
[1544189928.515246] [001.2] [pid=30772] Handling: 95=Memory usage: total:7167.61 MB - used: 2977.62 MB (42%) - free: 4189.99 MB (58%)
[1544189928.515258] [001.2] [pid=30772] Handling: 125=
[1544189928.515269] [001.2] [pid=30772] Handling: 99='Memory usage'=2977.62MB;6737.56;6880.91;0.00;7167.61
[1544189928.515280] [001.2] [pid=30772] Handling: 999
[1544189928.515305] [002.0] [pid=30772] INSERT INTO nagios_servicechecks SET instance_id='1', service_object_id='37818', check_type='0', current_check_attempt='1', max_check_attempts='3', state='0', state_type='1', start_time=FROM_UNIXTIME(1544189928), start_time_usec='508743', end_time=FROM_UNIXTIME(1544189928), end_time_usec='512794', timeout='360', early_timeout='0', execution_time='0.004050', latency='0.000000', return_code='0', output='Memory usage: total:7167\.61 MB - used: 2977\.62 MB \(42%\) - free: 4189\.99 MB \(58%\)', long_output='', perfdata='\'Memory usage\'=2977\.62MB;6737\.56;6880\.91;0\.00;7167\.61', command_object_id='0', command_args='', command_line='' ON DUPLICATE KEY UPDATE instance_id='1', service_object_id='37818', check_type='0', current_check_attempt='1', max_check_attempts='3', state='0', state_type='1', start_time=FROM_UNIXTIME(1544189928), start_time_usec='508743', end_time=FROM_UNIXTIME(1544189928), end_time_usec='512794', timeout='360', early_timeout='0', execution_time='0.004050', latency='0.000000', return_code='0', output='Memory usage: total:7167\.61 MB - used: 2977\.62 MB \(42%\) - free: 4189\.99 MB \(58%\)', long_output='', perfdata='\'Memory usage\'=2977\.62MB;6737\.56;6880\.91;0\.00;7167\.61'
[1544189928.515913] [001.2] [pid=30772] Handling:
[1544189928.515937] [001.2] [pid=30772] Handling:
[1544189928.515948] [001.2] [pid=30772] Handling: 213:
[1544189928.515961] [001.2] [pid=30772] Handling: 1=1202
[1544189928.515972] [001.2] [pid=30772] Handling: 2=0
[1544189928.515984] [001.2] [pid=30772] Handling: 3=0
[1544189928.515995] [001.2] [pid=30772] Handling: 4=1544189928.512939
[1544189928.516006] [001.2] [pid=30772] Handling: 53=servername2
[1544189928.516043] [001.2] [pid=30772] Handling: 114=Memory Usage
[1544189928.516060] [001.2] [pid=30772] Queue Message: ree: 4189.99 MB (58%)
125=
99='Memory usage'=2977.62MB;6737.56;6880.91;0.00;7167.61
27=0
51=1
115=1
25=1
76=3
61=1544189928
83=1544190228
12=0
63=1543231283
57=1542968726
56=0
66=1544189928
70=0
67=0
64=1543231283
121=1
62=0
84=0
85=0
88=1
101=0
7=0
26=0
97=1
38=1
9=1
47=1
54=0
98=0.00000
71=0.00000
42=0.00405
113=0
45=0
103=1
93=1
80=0
37=
11=check_nt!MEMUSE!-w 94 -c 96
86=5.00000
109=1.00000
209=24x7
999


[1544189928.516070] [001.2] [pid=30772] Handling: 95=Memory usage: total:7167.61 MB - used: 2977.62 MB (42%) - free: 4189.99 MB (58%)
[1544189928.516081] [001.2] [pid=30772] Handling: 125=
[1544189928.516092] [001.2] [pid=30772] Handling: 99='Memory usage'=2977.62MB;6737.56;6880.91;0.00;7167.61
[1544189928.516104] [001.2] [pid=30772] Handling: 27=0
[1544189928.516115] [001.2] [pid=30772] Handling: 51=1
[1544189928.516125] [001.2] [pid=30772] Handling: 115=1
[1544189928.516136] [001.2] [pid=30772] Handling: 25=1
[1544189928.516147] [001.2] [pid=30772] Handling: 76=3
[1544189928.516158] [001.2] [pid=30772] Handling: 61=1544189928
[1544189928.516169] [001.2] [pid=30772] Handling: 83=1544190228
[1544189928.516180] [001.2] [pid=30772] Handling: 12=0
[1544189928.516190] [001.2] [pid=30772] Handling: 63=1543231283
[1544189928.516201] [001.2] [pid=30772] Handling: 57=1542968726
[1544189928.516212] [001.2] [pid=30772] Handling: 56=0
[1544189928.516223] [001.2] [pid=30772] Handling: 66=1544189928
[1544189928.516256] [001.2] [pid=30772] Handling: 70=0
[1544189928.516269] [001.2] [pid=30772] Handling: 67=0
[1544189928.516280] [001.2] [pid=30772] Handling: 64=1543231283
[1544189928.516291] [001.2] [pid=30772] Handling: 121=1
[1544189928.516302] [001.2] [pid=30772] Handling: 62=0
[1544189928.516320] [001.2] [pid=30772] Handling: 84=0
[1544189928.516332] [001.2] [pid=30772] Handling: 85=0
[1544189928.516344] [001.2] [pid=30772] Handling: 88=1
[1544189928.516355] [001.2] [pid=30772] Handling: 101=0
[1544189928.516365] [001.2] [pid=30772] Handling: 7=0
[1544189928.516377] [001.2] [pid=30772] Handling: 26=0
[1544189928.516403] [001.2] [pid=30772] Handling: 97=1
[1544189928.516424] [001.2] [pid=30772] Handling: 38=1
[1544189928.516435] [001.2] [pid=30772] Handling: 9=1
[1544189928.516447] [001.2] [pid=30772] Handling: 47=1
[1544189928.516457] [001.2] [pid=30772] Handling: 54=0
[1544189928.516522] [001.2] [pid=30772] Handling: 98=0.00000
[1544189928.516536] [001.2] [pid=30772] Handling: 71=0.00000
[1544189928.516547] [001.2] [pid=30772] Handling: 42=0.00405
[1544189928.516558] [001.2] [pid=30772] Handling: 113=0
[1544189928.516568] [001.2] [pid=30772] Handling: 45=0
[1544189928.516579] [001.2] [pid=30772] Handling: 103=1
[1544189928.516590] [001.2] [pid=30772] Handling: 93=1
[1544189928.516601] [001.2] [pid=30772] Handling: 80=0
[1544189928.516611] [001.2] [pid=30772] Handling: 37=
[1544189928.516639] [001.2] [pid=30772] Handling: 11=check_nt!MEMUSE!-w 94 -c 96
[1544189928.516654] [001.2] [pid=30772] Handling: 86=5.00000
[1544189928.516673] [001.2] [pid=30772] Handling: 109=1.00000
[1544189928.516685] [001.2] [pid=30772] Handling: 209=24x7
[1544189928.516696] [001.2] [pid=30772] Handling: 999
[1544189928.516729] [002.0] [pid=30772] INSERT INTO nagios_servicestatus SET instance_id='1', service_object_id='37818', status_update_time=FROM_UNIXTIME(1544189928), output='Memory usage: total:7167\.61 MB - used: 2977\.62 MB \(42%\) - free: 4189\.99 MB \(58%\)', long_output='', perfdata='\'Memory usage\'=2977\.62MB;6737\.56;6880\.91;0\.00;7167\.61', current_state='0', has_been_checked='1', should_be_scheduled='1', current_check_attempt='1', max_check_attempts='3', last_check=FROM_UNIXTIME(1544189928), next_check=FROM_UNIXTIME(1544190228), check_type='0', last_state_change=FROM_UNIXTIME(1543231283), last_hard_state_change=FROM_UNIXTIME(1542968726), last_hard_state='0', last_time_ok=FROM_UNIXTIME(1544189928), last_time_warning=FROM_UNIXTIME(0), last_time_unknown=FROM_UNIXTIME(0), last_time_critical=FROM_UNIXTIME(1543231283), state_type='1', last_notification=FROM_UNIXTIME(0), next_notification=FROM_UNIXTIME(0), no_more_notifications='0', notifications_enabled='1', problem_has_been_acknowledged='0', acknowledgement_type='0', current_notification_number='0', passive_checks_enabled='1', active_checks_enabled='1', event_handler_enabled='1', flap_detection_enabled='1', is_flapping='0', percent_state_change='0.000000', latency='0.000000', execution_time='0.004050', scheduled_downtime_depth='0', failure_prediction_enabled='0', process_performance_data='1', obsess_over_service='1', modified_service_attributes='0', event_handler='', check_command='check_nt!MEMUSE!-w 94 -c 96', normal_check_interval='5.000000', retry_check_interval='1.000000', check_timeperiod_object_id='2' ON DUPLICATE KEY UPDATE instance_id='1', service_object_id='37818', status_update_time=FROM_UNIXTIME(1544189928), output='Memory usage: total:7167\.61 MB - used: 2977\.62 MB \(42%\) - free: 4189\.99 MB \(58%\)', long_output='', perfdata='\'Memory usage\'=2977\.62MB;6737\.56;6880\.91;0\.00;7167\.61', current_state='0', has_been_checked='1', should_be_scheduled='1', current_check_attempt='1', max_check_attempts='3', last_check=FROM_UNIXTIME(1544189928), next_check=FROM_UNIXTIME(1544190228), check_type='0', last_state_change=FROM_UNIXTIME(1543231283), last_hard_state_change=FROM_UNIXTIME(1542968726), last_hard_state='0', last_time_ok=FROM_UNIXTIME(1544189928), last_time_warning=FROM_UNIXTIME(0), last_time_unknown=FROM_UNIXTIME(0), last_time_critical=FROM_UNIXTIME(1543231283), state_type='1', last_notification=FROM_UNIXTIME(0), next_notification=FROM_UNIXTIME(0), no_more_notifications='0', notifications_enabled='1', problem_has_been_acknowledged='0', acknowledgement_type='0', current_notification_number='0', passive_checks_enabled='1', active_checks_enabled='1', event_handler_enabled='1', flap_detection_enabled='1', is_flapping='0', percent_state_change='0.000000', latency='0.000000', execution_time='0.004050', scheduled_downtime_depth='0', failure_prediction_enabled='0', process_performance_data='1', obsess_over_service='1', modified_service_attributes='0', event_handler='', check_command='check_nt!MEMUSE!-w 94 -c 96', normal_check_interval='5.000000', retry_check_interval='1.000000', check_timeperiod_object_id='2'
[1544189928.517366] [001.2] [pid=30772] Handling:



[ssax]

Nothing is standing out as an issue.

Please run this command and PM me the resulting /tmp/GARBLEDDUMP.sql file:

Code: Select all

mysqldump -uroot -pROOTMYSQLPASS --add-drop-database nagios nagios_objects > /tmp/GARBLEDDUMP.sql

[DonGately]

Nothing is standing out as an issue.

Please run this command and PM me the resulting /tmp/GARBLEDDUMP.sql file:

Code: Select all

mysqldump -uroot -pROOTMYSQLPASS --add-drop-database nagios nagios_objects > /tmp/GARBLEDDUMP.sql

I've sent you a pm

[ssax]

Please PM me these files:

Code: Select all

/usr/local/nagios/var/objects.cache
/usr/local/nagios/var/status.dat

[DonGately]

Please PM me these files:

Code: Select all

/usr/local/nagios/var/objects.cache
/usr/local/nagios/var/status.dat

Sorry for the delay. I've sent you a pm.

[ssax]

Everything looks proper.

Did you try recompiling NDO2DB? This really looks like a memory issue or a some type of corruption issue.

Can you try upgrading to NDO2DB version 2.1.3?

[DonGately]

Everything looks proper.

Did you try recompiling NDO2DB? This really looks like a memory issue or a some type of corruption issue.

Can you try upgrading to NDO2DB version 2.1.3?

I can't try recompiling it, because I have installed it via yum on Red Hat Enterprise Linux. That's also the newest version that's in the repositories right now (ndoutils 2.1.2-1).

I just looked into our Testenvironment and there's the same problem. The MariaDB I have installed on a Test-Server has also a corrupted nagios_objects table. At first everything looks fine, but at around object_id 11849 it becomes garbled. In the Testenvironment it starts aprubtly, i.e. from one object_id to another it becomes "gibberish" and continues on without exception as it seems. In the production environment it started off with just a few garbled entries and gets worse and worse until all of them are garbled. Sadly I don't have the Nagios Testserver anymore (only the Testserver where MariaDB is installed, that's why I was able to look into it).

What could possible other solutions be if it's a memory or corruption issue?

[tgriep]

One question, you said that the test environment is showing the same issue, are both the Production server and the test server pointing to the same MYSQL server and the same tables?
Can you verify that they are not?

[DonGately]

One question, you said that the test environment is showing the same issue, are both the Production server and the test server pointing to the same MYSQL server and the same tables?
Can you verify that they are not?

I can verify for shure, that they are not writing to the same MariaDB server / tables.

Everything looks proper.

Did you try recompiling NDO2DB? This really looks like a memory issue or a some type of corruption issue.

Can you try upgrading to NDO2DB version 2.1.3?

I have installed Nagios Core on a testserver again and configured it so that it writes data into MariaDB on another testserver. More precisely I choose the old MariaDB testserver and dropped the corrupted Nagios database. I removed version 2.1.2 of ndoutils from the Nagios testserver and manually compiled and installed ndoutils 2.1.3 (I installed version 2.1.2 via yum first so that I later on can reiterate the same procedure of reinstalling/upgrading in the production environment). I then ran the installdb script on the MariaDB testserver to initalize the Nagios database. Then I ran upgradescript to upgrade the database version. After that I copied the Nagios configuration directory from the Nagios production environment onto the test environment and started the Nagios service on the testserver, after first starting the ndo2db service. The nagios_objects table was filled with clean and uncorrupted data. I will be running the Nagios testserver together with the MariaDB testserver for a while, before I reproduce the same procedure in the production environment.