Nagios Event Log Empty

[ahoward12]

Hey Guys, I have a previous post about this: https://support.nagios.com/forum/viewto ... 16&t=44288

I have no problem running the repair databases script every so often, however I am having to run it every time I want to look at my event log. This is getting a bit ridiculous, as I lose all of the prior events running the script and fixing the databases.. I have attached a profile as well. Any ideas would be great..

CentOS 6.8
VMware Virtual Machine
Nagios XI 5.4.4

Edit: Attached is the SSH output from running the script if it helps.

[tgriep]

The corruption could be caused by the kernel message queue is not large enough to process the data that XI writes to the MYSQL database.
There are some settings that can be changed on the server to increase the queue, give it a try and see if this resolves the MYSQL database corruption.
https://support.nagios.com/kb/article/n ... eeded.html

[ahoward12]

Edit** I am leaving below for reference Issue is resolved by permissions:

Code: Select all

chmod 777 -R /tmp

I did 777 to eliminate problems. What are the permissions actually supposed to be?[/b]


Ok, I can give that a try but the problem developed more. I went to log into Nagios XI yesterday and everything is blank. I can see the hosts in CCM but if I go to my "Hostgoup Overview" tab they are all blank, this message is all over the screen:

Code: Select all

SQL Error [ndoutils] : Can't create/write to file '/tmp/#sql_15da_0.MYI' (Errcode: 13)

I restored it from a backup of a week ago, when I know everything was working perfect. It said it was restored but the same problem exists. It ends with this:

Code: Select all

Checking misc settings...
        Error: Unable to write to temp_path ('/tmp') - Permission denied

Total Warnings: 0
Total Errors:   1

***> One or more problems was encountered while running the pre-flight check...

     Check your configuration file(s) to ensure that they contain valid
     directives and data defintions.  If you are upgrading from a previous
     version of Nagios, you should be aware that some variables/definitions
     may have been removed or modified in this version.  Make sure to read
     the HTML documentation regarding the config files, as well as the
     'Whats New' section to find out what has changed.


=======================
nagios database repair succeeded
nagiosql database repair succeeded
nagiosxi database repair succeeded

[root@NAGIOS scripts]#

Here is the mysqld.log

Code: Select all

[root@NAGIOS /]# tail -n 20 /var/log/mysqld.log
170713  9:00:02 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_logentries' is marked as crashed and last (automatic                                                                                                                     ?) repair failed
170713  9:04:08 [Warning] Warning: Enabling keys got errno 13 on nagios.nagios_logentries, retrying
170713  9:04:12 [Note] /usr/libexec/mysqld: Normal shutdown

170713  9:04:12 [Note] Event Scheduler: Purging the queue. 0 events
170713  9:04:14 [Note] /usr/libexec/mysqld: Shutdown complete

170713 09:04:14 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
170713 09:04:14 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
/usr/libexec/mysqld: Can't create/write to file '/tmp/ibrAmCPr' (Errcode: 13)
170713  9:04:14  InnoDB: Error: unable to create temporary file; errno: 13
170713  9:04:14 [ERROR] Plugin 'InnoDB' init function returned error.
170713  9:04:14 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
170713  9:04:14 [Note] Event Scheduler: Loaded 0 events
170713  9:04:14 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
170713  9:05:02 [Warning] Warning: Optimize table got errno 13 on nagios.nagios_logentries, retrying
170713  9:10:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_logentries' is marked as crashed and last (automatic                                                                                                                     ?) repair failed
170713  9:15:02 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_logentries' is marked as crashed and last (automatic                                                                                                                     ?) repair failed
170713  9:20:02 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_logentries' is marked as crashed and last (automatic                                                                                                                     ?) repair failed
[root@NAGIOS /]#

In that error above, the permissions for /tmp is the same as everything else.

[tgriep]

Yes, it looks like the cause of the problem is the permissions of the /tmp folder is causing the dbmaint script to fail and also the ndoutils and that is causing the corruption.

The permissions for the /tmp folder should be read / write / executable for the user, grout and other as well as the stick bit set for other.
Do set the permissions for the folder, run the following as root

Code: Select all

chmod 777 /tmp
chmod o+t /tmp

Those permission need to be set that way so any applications and daemons that use the folder for temporary files would have full access to do so.
If something is resetting or removing the permissions of the /tmp folder, that would have to stop.

Also, the free space of the hard drive could be increased so in the future that will not cause an issue.
If the space is available, the doubling or tripling the drive by following this document.
https://assets.nagios.com/downloads/nag ... M-Disk.pdf

[ahoward12]

Appreciate the quick responses as always. I extended the drive shortly after the initial post, then followed the other guide for the database. Hopefully this will fix my issues.

Can we keep this thread open for the time being? I'll put a reminder on my calender to close it on Monday if I have no futher issues.

[bolson]

We'll go ahead and leave this open for you, let us know Monday and we'll go from there.

[ahoward12]

I apologize very late on this. I needed to check my event log today and same issue as always its empty, I haven't ran the script to repair it yet. I'm also not sure what information you need to troubleshoot this. Please let me know, thank you.

[tacolover101]

email your profile to bolson, or tgriep. that will be a good start. there could be corruption within SQL, depending where it'd be interesting to see what the tables are populated with.

[dwhitfield]

email your profile to bolson, or tgriep.

PM is a better option (you can also PM it to me)...or email [email protected]

[ahoward12]

Attached is a system profile. No need to PM it, my security is on point, only local addresses.

I forgot to subscribe to the thread so my responses have been slow. They will be quicker from now on. Thanks for the help