XI 2014RC3 BETA DB issue

[snapon_admin]

So I just logged into my test server with XI 2014R1.0RC3 on it and was greeted with a bunch of DB errors and instructions to run the repair_database script. This has never happened on our prod server so it was a little strange, especially since there is almost nothing on our test server atm, but I ran the script and everything appeared to repair normally, with one exception. I have a few users created on this test server, all of them Admins except one. After this DB hiccup, I used the "Masquerade as" feature to view nagios as all the users. Everything looks normal for all of the admins, but the one account created with just user privileges sees this:

test server sql issue.png

[lmiltchev]

Did you try running both commands?

Code: Select all

./repairmysql.sh nagios
./repairmysql.sh nagiosql

Run the following commands and show the output:

Code: Select all

service mysqld restart
tail -25 /var/log/mysqld.log

[snapon_admin]

This was the command the error told me to run:

Code: Select all

/usr/local/nagiosxi/scripts/repair_databases.sh

Which I believe includes both of those correct?

The output your requested:

Code: Select all

[root@keno-ngos-01-pv scripts]# service mysqld restart
Stopping mysqld:                                           [  OK  ]
Starting mysqld:                                           [  OK  ]
[root@keno-ngos-01-pv scripts]# tail -25 /var/log/mysqld.log
140430 16:12:16 [Note] /usr/libexec/mysqld: Shutdown complete

140430 16:12:16 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
140430 16:12:17 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140430 16:12:17  InnoDB: Initializing buffer pool, size = 8.0M
140430 16:12:17  InnoDB: Completed initialization of buffer pool
140430 16:12:17  InnoDB: Started; log sequence number 0 44233
140430 16:12:17 [Note] Event Scheduler: Loaded 0 events
140430 16:12:17 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.71'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
140430 16:12:27 [Note] /usr/libexec/mysqld: Normal shutdown

140430 16:12:27 [Note] Event Scheduler: Purging the queue. 0 events
140430 16:12:29  InnoDB: Starting shutdown...
140430 16:12:32  InnoDB: Shutdown completed; log sequence number 0 44233
140430 16:12:32 [Note] /usr/libexec/mysqld: Shutdown complete

140430 16:12:32 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
140430 16:12:33 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140430 16:12:33  InnoDB: Initializing buffer pool, size = 8.0M
140430 16:12:33  InnoDB: Completed initialization of buffer pool
140430 16:12:33  InnoDB: Started; log sequence number 0 44233
140430 16:12:33 [Note] Event Scheduler: Loaded 0 events
140430 16:12:33 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.71'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
[root@keno-ngos-01-pv scripts]#

[snapon_admin]

Update: Setting this user to Admin corrects the issue and everything under Service Status Detail looks normal. Setting the user back to User with limited privileges causes the issue to return.

[slansing]

Those displayed errors are ndoutils errors, not a crashed SQL tables error. Where else did you see these? And where did you see the crashed tables error? Do you recall what tables were erroring? I was not able to reproduce this on a system that was originally 2012 r2.9, upgraded to 2014 Beta RC1, 2, then 3. Has ndo been running?:

Code: Select all

service ndo2db status

Is this user's contact assigned to any hosts/services or is it blank?

[lmiltchev]

This is interesting. Can you tell us which of the checkboxes do you have selected under the "Security Settings" for this user (Admin->Manage Users->Edit)? I would like to be able to recreate the issue in house.

[snapon_admin]

Ok, so I didn't change anything after my last post yesterday. I left this user the way it was and the issue was still present as of 5pm CST yesterday. Today I come in and everything is fine....so now I'm really baffled. To answer your questions:

Where else did you see these?

Only on this one user, but this is the only UID on our test box that isn't an admin.

And where did you see the crashed tables error?

I logged into the web UI as nagiosadmin and there was a bunch of errors talking about the mysql database. There was also instructions to run /usr/local/nagiosxi/scripts/repair_databases.sh, which I did. I'm sorry, I didn't get a screenshot of the error. I should have, but it was pre-coffee and I didn't even think about it.

I was not able to reproduce this on a system that was originally 2012 r2.9, upgraded to 2014 Beta RC1, 2, then 3

Not sure if it matters but this server went from 2012 r2.9 to 2014 beta RC1 and then straight to RC3, no RC2.

Has ndo been running?

Yep.

Is this user's contact assigned to any hosts/services or is it blank?

Yes, this user's contact is assigned to one host and specifically, 10 services on that host as well.

Can you tell us which of the checkboxes do you have selected under the "Security Settings" for this user

User settings.png

[abrist]

This sounds like a possible caching/crashed db issue. Has the issue returned, or is the server and non-admin user working correctly?

[snapon_admin]

As of when I left the office at 4 today, everything was working fine for all users. Not sure how it suddenly started working again, but I'll keep an eye on it and let you know if anything crops up again.

[tmcdonald]

We'll leave this open for a bit.