Empty Event Log

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
Locked
adminfun
Posts: 28
Joined: Mon May 11, 2015 9:06 am
Location: Bruges

Empty Event Log

Post by adminfun »

Dear members

Last night, the Nagios went haywire.
We received a massive amount of faulty e-mails. (Service checks that timed out)
I couldn't connect through SSH nor the web-interface so I had to force restart the virtual machine.

I wanted to check the event log, but it is completely empty which is a bit odd.
Could somebody help me troublehsoot? I posted some sreenshots in attachement.
You do not have the required permissions to view the files attached to this post.
Top
User avatar
tgriep
Madmin
Posts: 9190
Joined: Thu Oct 30, 2014 9:02 am

Re: Empty Event Log

Post by tgriep »

That could be caused by the MYSQL database corruption. Run the following to see if there are any errors.

Code: Select all

tail -50 /var/log/mysqld.log
If there are errors, you can run the following to repair the database.

Code: Select all

cd /usr/local/nagiosxi/scripts
./repair_databases.sh
Be sure to check out our Knowledgebase for helpful articles and solutions!
Top
adminfun
Posts: 28
Joined: Mon May 11, 2015 9:06 am
Location: Bruges

Re: Empty Event Log

Post by adminfun »

There were about a dozen errors.
I ran the repair command and will keep you guys posted.
Top
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:
Contact scottwilkerson

Re: Empty Event Log

Post by scottwilkerson »

Sounds good, let us know if you have further problems
Former Nagios employee
Creator:
Human Design Website
Get Your Human Design Chart
Top
adminfun
Posts: 28
Joined: Mon May 11, 2015 9:06 am
Location: Bruges

Re: Empty Event Log

Post by adminfun »

We received a bunch of faulty emails again this night.
All emails were about Service checks that timed out.
So my guess is that the Nagios lost connection with those.
In attachement a screenshot of the Nagios Performance.

What could be the cause of this?
You do not have the required permissions to view the files attached to this post.
Top
User avatar
lmiltchev
Bugs find me
Posts: 13589
Joined: Mon May 23, 2011 12:15 pm

Re: Empty Event Log

Post by lmiltchev »

Can you run the following commands and show the output in code wraps?

Code: Select all

df -h
df -i
tail -50 /var/log/mysqld.log
tail -50 /var/log/httpd/error_log
Be sure to check out our Knowledgebase for helpful articles and solutions!
Top
adminfun
Posts: 28
Joined: Mon May 11, 2015 9:06 am
Location: Bruges

Re: Empty Event Log

Post by adminfun »

Here you are.
In the mean time I increased the amount of memory from 2GB to 4GB and increased the CPU cores from 1 to 2.

df -h

Code: Select all

[root@localhost ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
                      7.3G  5.3G  1.7G  77% /
tmpfs                 1.9G     0  1.9G   0% /dev/shm
/dev/sda1             477M   66M  386M  15% /boot
df -i

Code: Select all

[root@localhost ~]# df -i
Filesystem           Inodes IUsed  IFree IUse% Mounted on
/dev/mapper/VolGroup-lv_root
                     494832 78867 415965   16% /
tmpfs                490556     1 490555    1% /dev/shm
/dev/sda1            128016    50 127966    1% /boot
tail -50 /var/log/mysqld.log

Code: Select all

[root@localhost ~]# tail -50 /var/log/mysqld.log
150929  9:09:12  InnoDB: Completed initialization of buffer pool
150929  9:09:13  InnoDB: Started; log sequence number 0 44233
150929  9:09:13 [Note] Event Scheduler: Loaded 0 events
150929  9:09:13 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distr                                   ibution
150929  9:09:14 [Note] /usr/libexec/mysqld: Normal shutdown

150929  9:09:14 [Note] Event Scheduler: Purging the queue. 0 events
150929  9:09:14  InnoDB: Starting shutdown...
150929  9:09:18  InnoDB: Shutdown completed; log sequence number 0 44233
150929  9:09:18 [Note] /usr/libexec/mysqld: Shutdown complete

150929 09:09:18 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ende                                   d
150929 09:09:19 mysqld_safe Starting mysqld daemon with databases from /var/lib/                                   mysql
150929  9:09:19  InnoDB: Initializing buffer pool, size = 8.0M
150929  9:09:19  InnoDB: Completed initialization of buffer pool
150929  9:09:19  InnoDB: Started; log sequence number 0 44233
150929  9:09:19 [Note] Event Scheduler: Loaded 0 events
150929  9:09:19 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distr                                   ibution
150929 13:19:38 [Note] /usr/libexec/mysqld: Normal shutdown

150929 13:19:38 [Note] Event Scheduler: Purging the queue. 0 events
150929 13:19:38  InnoDB: Starting shutdown...
150929 13:19:40  InnoDB: Shutdown completed; log sequence number 0 44233
150929 13:19:40 [Note] /usr/libexec/mysqld: Shutdown complete

150929 13:19:40 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ende                                   d
150929 13:20:17 mysqld_safe Starting mysqld daemon with databases from /var/lib/                                   mysql
150929 13:20:17  InnoDB: Initializing buffer pool, size = 8.0M
150929 13:20:17  InnoDB: Completed initialization of buffer pool
150929 13:20:18  InnoDB: Started; log sequence number 0 44233
150929 13:20:18 [Note] Event Scheduler: Loaded 0 events
150929 13:20:18 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distr                                   ibution
150929 16:27:54 [Note] /usr/libexec/mysqld: Normal shutdown

150929 16:27:54 [Note] Event Scheduler: Purging the queue. 0 events
150929 16:27:54  InnoDB: Starting shutdown...
150929 16:27:59  InnoDB: Shutdown completed; log sequence number 0 44233
150929 16:27:59 [Note] /usr/libexec/mysqld: Shutdown complete

150929 16:27:59 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ende                                   d
150929 16:31:45 mysqld_safe Starting mysqld daemon with databases from /var/lib/                                   mysql
150929 16:31:46  InnoDB: Initializing buffer pool, size = 8.0M
150929 16:31:46  InnoDB: Completed initialization of buffer pool
150929 16:31:46  InnoDB: Started; log sequence number 0 44233
150929 16:31:46 [Note] Event Scheduler: Loaded 0 events
150929 16:31:46 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distr                                   ibution
tail -50 /var/log/httpd/error_log

Code: Select all

[root@localhost ~]# tail -50 /var/log/httpd/error_log
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice:  Undefined index: avg_latency in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 110, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice:  Undefined index: min_execution_time in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 111, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice:  Undefined index: max_execution_time in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 112, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice:  Undefined index: avg_execution_time in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 113, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice:  Undefined index: min_latency in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 118, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice:  Undefined index: max_latency in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 119, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice:  Undefined index: avg_latency in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 120, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice:  Undefined index: min_execution_time in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 121, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice:  Undefined index: max_execution_time in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 122, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice:  Undefined index: avg_execution_time in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 123, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice:  Undefined index: min_latency in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 108, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice:  Undefined index: max_latency in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 109, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice:  Undefined index: avg_latency in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 110, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice:  Undefined index: min_execution_time in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 111, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice:  Undefined index: max_execution_time in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 112, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice:  Undefined index: avg_execution_time in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 113, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice:  Undefined index: min_latency in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 118, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice:  Undefined index: max_latency in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 119, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice:  Undefined index: avg_latency in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 120, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice:  Undefined index: min_execution_time in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 121, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice:  Undefined index: max_execution_time in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 122, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice:  Undefined index: avg_execution_time in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 123, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:06 2015] [error] [client 192.168.133.153] PHP Notice:  Undefined index: min_latency in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 108, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:06 2015] [error] [client 192.168.133.153] PHP Notice:  Undefined index: max_latency in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 109, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:06 2015] [error] [client 192.168.133.153] PHP Notice:  Undefined index: avg_latency in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 110, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:06 2015] [error] [client 192.168.133.153] PHP Notice:  Undefined index: min_execution_time in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 111, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:06 2015] [error] [client 192.168.133.153] PHP Notice:  Undefined index: max_execution_time in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 112, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:06 2015] [error] [client 192.168.133.153] PHP Notice:  Undefined index: avg_execution_time in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 113, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:06 2015] [error] [client 192.168.133.153] PHP Notice:  Undefined index: min_latency in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 118, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:06 2015] [error] [client 192.168.133.153] PHP Notice:  Undefined index: max_latency in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 119, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:06 2015] [error] [client 192.168.133.153] PHP Notice:  Undefined index: avg_latency in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 120, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:06 2015] [error] [client 192.168.133.153] PHP Notice:  Undefined index: min_execution_time in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 121, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:06 2015] [error] [client 192.168.133.153] PHP Notice:  Undefined index: max_execution_time in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 122, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:06 2015] [error] [client 192.168.133.153] PHP Notice:  Undefined index: avg_execution_time in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 123, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 12:20:06 2015] [error] [client 192.168.133.157] PHP Notice:  Undefined index: undefined in /usr/local/nagiosxi/html/includes/components/graphexplorer/visFunctions.inc.php on line 167, referer: http://10.20.1.175/nagiosxi/includes/components/graphexplorer/
[Tue Sep 29 12:20:06 2015] [error] [client 192.168.133.157] PHP Notice:  Undefined index: undefined in /usr/local/nagiosxi/html/includes/components/graphexplorer/visFunctions.inc.php on line 184, referer: http://10.20.1.175/nagiosxi/includes/components/graphexplorer/
[Tue Sep 29 12:20:06 2015] [error] [client 192.168.133.157] PHP Notice:  Undefined index: undefined in /usr/local/nagiosxi/html/includes/components/graphexplorer/visFunctions.inc.php on line 185, referer: http://10.20.1.175/nagiosxi/includes/components/graphexplorer/
[Tue Sep 29 12:20:06 2015] [error] [client 192.168.133.157] PHP Warning:  implode(): Invalid arguments passed in /usr/local/nagiosxi/html/includes/components/graphexplorer/templates/timeline.inc.php on line 176, referer: http://10.20.1.175/nagiosxi/includes/components/graphexplorer/
[Tue Sep 29 12:24:55 2015] [error] [client 192.168.133.157] PHP Warning:  simplexml_load_file(): I/O warning : failed to load external entity "/usr/local/nagios/share/perfdata//_HOST_.xml" in /usr/local/nagiosxi/html/includes/components/graphexplorer/visFunctions.inc.php on line 637, referer: http://10.20.1.175/nagiosxi/includes/components/graphexplorer/
[Tue Sep 29 12:24:55 2015] [error] [client 192.168.133.157] PHP Fatal error:  Call to a member function xpath() on a non-object in /usr/local/nagiosxi/html/includes/components/graphexplorer/visFunctions.inc.php on line 638, referer: http://10.20.1.175/nagiosxi/includes/components/graphexplorer/
[Tue Sep 29 13:19:35 2015] [notice] caught SIGTERM, shutting down
[Tue Sep 29 13:20:24 2015] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Tue Sep 29 13:20:24 2015] [notice] Digest: generating secret for digest authentication ...
[Tue Sep 29 13:20:24 2015] [notice] Digest: done
[Tue Sep 29 13:20:27 2015] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.3 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips configured -- resuming normal operations
[Tue Sep 29 16:27:53 2015] [notice] caught SIGTERM, shutting down
[Tue Sep 29 16:31:51 2015] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Tue Sep 29 16:31:51 2015] [notice] Digest: generating secret for digest authentication ...
[Tue Sep 29 16:31:51 2015] [notice] Digest: done
[Tue Sep 29 16:31:53 2015] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.3 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips configured -- resuming normal operations
Top
jdalrymple
Skynet Drone
Posts: 2620
Joined: Wed Feb 11, 2015 1:56 pm

Re: Empty Event Log

Post by jdalrymple »

Is this a VM or a physical box? Either way can you have the storage administrator of the system take a look at IO latency during the period of time that things went south? I have a hunch you'll see some anomalous readings.
Top
adminfun
Posts: 28
Joined: Mon May 11, 2015 9:06 am
Location: Bruges

Re: Empty Event Log

Post by adminfun »

It's a virtual machine. Installed with an OVF template.
There is a lot going on at night. Backups via the network and such, could it be that the storage box or the network is saturated?
Top
jdalrymple
Skynet Drone
Posts: 2620
Joined: Wed Feb 11, 2015 1:56 pm

Re: Empty Event Log

Post by jdalrymple »

adminfun wrote:could it be that the storage box or the network is saturated
Absolutely - and that's kind of what I was expecting. Maybe correlate the timeline with your backup admins and see if it lines up? The solution if that's the case is our ramdisk modification and to also schedule your XI backups off those hours.

https://assets.nagios.com/downloads/nag ... giosXI.pdf
Top
Locked

Return to “Nagios XI”