Page 1 of 3
Nagios Event Log Empty
Posted: Wed Jul 12, 2017 7:45 am
by ahoward12
Hey Guys, I have a previous post about this:
https://support.nagios.com/forum/viewto ... 16&t=44288
I have no problem running the repair databases script every so often, however I am having to run it every time I want to look at my event log. This is getting a bit ridiculous, as I lose all of the prior events running the script and fixing the databases.. I have attached a profile as well. Any ideas would be great..
CentOS 6.8
VMware Virtual Machine
Nagios XI 5.4.4
Edit: Attached is the SSH output from running the script if it helps.
Re: Nagios Event Log Empty
Posted: Wed Jul 12, 2017 12:17 pm
by tgriep
The corruption could be caused by the kernel message queue is not large enough to process the data that XI writes to the MYSQL database.
There are some settings that can be changed on the server to increase the queue, give it a try and see if this resolves the MYSQL database corruption.
https://support.nagios.com/kb/article/n ... eeded.html
Re: Nagios Event Log Empty
Posted: Thu Jul 13, 2017 7:56 am
by ahoward12
Edit** I am leaving below for reference Issue is resolved by permissions:
I did 777 to eliminate problems. What are the permissions actually supposed to be?[/b]
Ok, I can give that a try but the problem developed more. I went to log into Nagios XI yesterday and everything is blank. I can see the hosts in CCM but if I go to my "Hostgoup Overview" tab they are all blank, this message is all over the screen:
Code: Select all
SQL Error [ndoutils] : Can't create/write to file '/tmp/#sql_15da_0.MYI' (Errcode: 13)
I restored it from a backup of a week ago, when I know everything was working perfect. It said it was restored but the same problem exists. It ends with this:
Code: Select all
Checking misc settings...
Error: Unable to write to temp_path ('/tmp') - Permission denied
Total Warnings: 0
Total Errors: 1
***> One or more problems was encountered while running the pre-flight check...
Check your configuration file(s) to ensure that they contain valid
directives and data defintions. If you are upgrading from a previous
version of Nagios, you should be aware that some variables/definitions
may have been removed or modified in this version. Make sure to read
the HTML documentation regarding the config files, as well as the
'Whats New' section to find out what has changed.
=======================
nagios database repair succeeded
nagiosql database repair succeeded
nagiosxi database repair succeeded
[root@NAGIOS scripts]#
Here is the mysqld.log
Code: Select all
[root@NAGIOS /]# tail -n 20 /var/log/mysqld.log
170713 9:00:02 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_logentries' is marked as crashed and last (automatic ?) repair failed
170713 9:04:08 [Warning] Warning: Enabling keys got errno 13 on nagios.nagios_logentries, retrying
170713 9:04:12 [Note] /usr/libexec/mysqld: Normal shutdown
170713 9:04:12 [Note] Event Scheduler: Purging the queue. 0 events
170713 9:04:14 [Note] /usr/libexec/mysqld: Shutdown complete
170713 09:04:14 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
170713 09:04:14 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
/usr/libexec/mysqld: Can't create/write to file '/tmp/ibrAmCPr' (Errcode: 13)
170713 9:04:14 InnoDB: Error: unable to create temporary file; errno: 13
170713 9:04:14 [ERROR] Plugin 'InnoDB' init function returned error.
170713 9:04:14 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
170713 9:04:14 [Note] Event Scheduler: Loaded 0 events
170713 9:04:14 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
170713 9:05:02 [Warning] Warning: Optimize table got errno 13 on nagios.nagios_logentries, retrying
170713 9:10:01 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_logentries' is marked as crashed and last (automatic ?) repair failed
170713 9:15:02 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_logentries' is marked as crashed and last (automatic ?) repair failed
170713 9:20:02 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_logentries' is marked as crashed and last (automatic ?) repair failed
[root@NAGIOS /]#
In that error above, the permissions for /tmp is the same as everything else.
Re: Nagios Event Log Empty
Posted: Thu Jul 13, 2017 9:53 am
by tgriep
Yes, it looks like the cause of the problem is the permissions of the /tmp folder is causing the dbmaint script to fail and also the ndoutils and that is causing the corruption.
The permissions for the /tmp folder should be read / write / executable for the user, grout and other as well as the stick bit set for other.
Do set the permissions for the folder, run the following as root
Those permission need to be set that way so any applications and daemons that use the folder for temporary files would have full access to do so.
If something is resetting or removing the permissions of the /tmp folder, that would have to stop.
Also, the free space of the hard drive could be increased so in the future that will not cause an issue.
If the space is available, the doubling or tripling the drive by following this document.
https://assets.nagios.com/downloads/nag ... M-Disk.pdf
Re: Nagios Event Log Empty
Posted: Thu Jul 13, 2017 12:04 pm
by ahoward12
Appreciate the quick responses as always. I extended the drive shortly after the initial post, then followed the other guide for the database. Hopefully this will fix my issues.
Can we keep this thread open for the time being? I'll put a reminder on my calender to close it on Monday if I have no futher issues.
Re: Nagios Event Log Empty
Posted: Thu Jul 13, 2017 1:43 pm
by bolson
We'll go ahead and leave this open for you, let us know Monday and we'll go from there.
Re: Nagios Event Log Empty
Posted: Sat Jul 22, 2017 8:42 am
by ahoward12
I apologize very late on this. I needed to check my event log today and same issue as always its empty, I haven't ran the script to repair it yet. I'm also not sure what information you need to troubleshoot this. Please let me know, thank you.
Re: Nagios Event Log Empty
Posted: Sun Jul 23, 2017 11:04 pm
by tacolover101
email your profile to bolson, or tgriep. that will be a good start. there could be corruption within SQL, depending where it'd be interesting to see what the tables are populated with.
Re: Nagios Event Log Empty
Posted: Mon Jul 24, 2017 10:32 am
by dwhitfield
tacolover101 wrote:email your profile to bolson, or tgriep.
PM is a better option (you can also PM it to me)...or email
[email protected]
Re: Nagios Event Log Empty
Posted: Wed Jul 26, 2017 10:10 am
by ahoward12
Attached is a system profile. No need to PM it, my security is on point, only local addresses.
I forgot to subscribe to the thread so my responses have been slow. They will be quicker from now on. Thanks for the help
