Page 1 of 1
Empty Event Log
Posted: Fri Sep 25, 2015 2:35 am
by adminfun
Dear members
Last night, the Nagios went haywire.
We received a massive amount of faulty e-mails. (Service checks that timed out)
I couldn't connect through SSH nor the web-interface so I had to force restart the virtual machine.
I wanted to check the event log, but it is completely empty which is a bit odd.
Could somebody help me troublehsoot? I posted some sreenshots in attachement.
Re: Empty Event Log
Posted: Fri Sep 25, 2015 10:17 am
by tgriep
That could be caused by the MYSQL database corruption. Run the following to see if there are any errors.
If there are errors, you can run the following to repair the database.
Code: Select all
cd /usr/local/nagiosxi/scripts
./repair_databases.sh
Re: Empty Event Log
Posted: Mon Sep 28, 2015 9:27 am
by adminfun
There were about a dozen errors.
I ran the repair command and will keep you guys posted.
Re: Empty Event Log
Posted: Mon Sep 28, 2015 10:04 am
by scottwilkerson
Sounds good, let us know if you have further problems
Re: Empty Event Log
Posted: Tue Sep 29, 2015 6:59 am
by adminfun
We received a bunch of faulty emails again this night.
All emails were about Service checks that timed out.
So my guess is that the Nagios lost connection with those.
In attachement a screenshot of the Nagios Performance.
What could be the cause of this?
Re: Empty Event Log
Posted: Tue Sep 29, 2015 9:35 am
by lmiltchev
Can you run the following commands and show the output in code wraps?
Code: Select all
df -h
df -i
tail -50 /var/log/mysqld.log
tail -50 /var/log/httpd/error_log
Re: Empty Event Log
Posted: Tue Sep 29, 2015 9:46 am
by adminfun
Here you are.
In the mean time I increased the amount of memory from 2GB to 4GB and increased the CPU cores from 1 to 2.
df -h
Code: Select all
[root@localhost ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
7.3G 5.3G 1.7G 77% /
tmpfs 1.9G 0 1.9G 0% /dev/shm
/dev/sda1 477M 66M 386M 15% /boot
Code: Select all
[root@localhost ~]# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/VolGroup-lv_root
494832 78867 415965 16% /
tmpfs 490556 1 490555 1% /dev/shm
/dev/sda1 128016 50 127966 1% /boot
Code: Select all
[root@localhost ~]# tail -50 /var/log/mysqld.log
150929 9:09:12 InnoDB: Completed initialization of buffer pool
150929 9:09:13 InnoDB: Started; log sequence number 0 44233
150929 9:09:13 [Note] Event Scheduler: Loaded 0 events
150929 9:09:13 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distr ibution
150929 9:09:14 [Note] /usr/libexec/mysqld: Normal shutdown
150929 9:09:14 [Note] Event Scheduler: Purging the queue. 0 events
150929 9:09:14 InnoDB: Starting shutdown...
150929 9:09:18 InnoDB: Shutdown completed; log sequence number 0 44233
150929 9:09:18 [Note] /usr/libexec/mysqld: Shutdown complete
150929 09:09:18 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ende d
150929 09:09:19 mysqld_safe Starting mysqld daemon with databases from /var/lib/ mysql
150929 9:09:19 InnoDB: Initializing buffer pool, size = 8.0M
150929 9:09:19 InnoDB: Completed initialization of buffer pool
150929 9:09:19 InnoDB: Started; log sequence number 0 44233
150929 9:09:19 [Note] Event Scheduler: Loaded 0 events
150929 9:09:19 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distr ibution
150929 13:19:38 [Note] /usr/libexec/mysqld: Normal shutdown
150929 13:19:38 [Note] Event Scheduler: Purging the queue. 0 events
150929 13:19:38 InnoDB: Starting shutdown...
150929 13:19:40 InnoDB: Shutdown completed; log sequence number 0 44233
150929 13:19:40 [Note] /usr/libexec/mysqld: Shutdown complete
150929 13:19:40 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ende d
150929 13:20:17 mysqld_safe Starting mysqld daemon with databases from /var/lib/ mysql
150929 13:20:17 InnoDB: Initializing buffer pool, size = 8.0M
150929 13:20:17 InnoDB: Completed initialization of buffer pool
150929 13:20:18 InnoDB: Started; log sequence number 0 44233
150929 13:20:18 [Note] Event Scheduler: Loaded 0 events
150929 13:20:18 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distr ibution
150929 16:27:54 [Note] /usr/libexec/mysqld: Normal shutdown
150929 16:27:54 [Note] Event Scheduler: Purging the queue. 0 events
150929 16:27:54 InnoDB: Starting shutdown...
150929 16:27:59 InnoDB: Shutdown completed; log sequence number 0 44233
150929 16:27:59 [Note] /usr/libexec/mysqld: Shutdown complete
150929 16:27:59 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ende d
150929 16:31:45 mysqld_safe Starting mysqld daemon with databases from /var/lib/ mysql
150929 16:31:46 InnoDB: Initializing buffer pool, size = 8.0M
150929 16:31:46 InnoDB: Completed initialization of buffer pool
150929 16:31:46 InnoDB: Started; log sequence number 0 44233
150929 16:31:46 [Note] Event Scheduler: Loaded 0 events
150929 16:31:46 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distr ibution
Code: Select all
[root@localhost ~]# tail -50 /var/log/httpd/error_log
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice: Undefined index: avg_latency in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 110, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice: Undefined index: min_execution_time in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 111, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice: Undefined index: max_execution_time in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 112, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice: Undefined index: avg_execution_time in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 113, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice: Undefined index: min_latency in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 118, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice: Undefined index: max_latency in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 119, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice: Undefined index: avg_latency in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 120, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice: Undefined index: min_execution_time in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 121, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice: Undefined index: max_execution_time in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 122, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice: Undefined index: avg_execution_time in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 123, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice: Undefined index: min_latency in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 108, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice: Undefined index: max_latency in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 109, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice: Undefined index: avg_latency in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 110, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice: Undefined index: min_execution_time in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 111, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice: Undefined index: max_execution_time in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 112, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice: Undefined index: avg_execution_time in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 113, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice: Undefined index: min_latency in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 118, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice: Undefined index: max_latency in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 119, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice: Undefined index: avg_latency in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 120, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice: Undefined index: min_execution_time in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 121, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice: Undefined index: max_execution_time in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 122, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:05 2015] [error] [client 192.168.133.153] PHP Notice: Undefined index: avg_execution_time in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 123, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:06 2015] [error] [client 192.168.133.153] PHP Notice: Undefined index: min_latency in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 108, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:06 2015] [error] [client 192.168.133.153] PHP Notice: Undefined index: max_latency in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 109, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:06 2015] [error] [client 192.168.133.153] PHP Notice: Undefined index: avg_latency in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 110, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:06 2015] [error] [client 192.168.133.153] PHP Notice: Undefined index: min_execution_time in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 111, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:06 2015] [error] [client 192.168.133.153] PHP Notice: Undefined index: max_execution_time in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 112, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:06 2015] [error] [client 192.168.133.153] PHP Notice: Undefined index: avg_execution_time in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 113, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:06 2015] [error] [client 192.168.133.153] PHP Notice: Undefined index: min_latency in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 118, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:06 2015] [error] [client 192.168.133.153] PHP Notice: Undefined index: max_latency in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 119, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:06 2015] [error] [client 192.168.133.153] PHP Notice: Undefined index: avg_latency in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 120, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:06 2015] [error] [client 192.168.133.153] PHP Notice: Undefined index: min_execution_time in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 121, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:06 2015] [error] [client 192.168.133.153] PHP Notice: Undefined index: max_execution_time in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 122, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 09:10:06 2015] [error] [client 192.168.133.153] PHP Notice: Undefined index: avg_execution_time in /usr/local/nagiosxi/html/includes/utils-xmlsysstat.inc.php on line 123, referer: http://10.20.1.175/nagiosxi/dashboards/dashboard.php?id=t7trethp
[Tue Sep 29 12:20:06 2015] [error] [client 192.168.133.157] PHP Notice: Undefined index: undefined in /usr/local/nagiosxi/html/includes/components/graphexplorer/visFunctions.inc.php on line 167, referer: http://10.20.1.175/nagiosxi/includes/components/graphexplorer/
[Tue Sep 29 12:20:06 2015] [error] [client 192.168.133.157] PHP Notice: Undefined index: undefined in /usr/local/nagiosxi/html/includes/components/graphexplorer/visFunctions.inc.php on line 184, referer: http://10.20.1.175/nagiosxi/includes/components/graphexplorer/
[Tue Sep 29 12:20:06 2015] [error] [client 192.168.133.157] PHP Notice: Undefined index: undefined in /usr/local/nagiosxi/html/includes/components/graphexplorer/visFunctions.inc.php on line 185, referer: http://10.20.1.175/nagiosxi/includes/components/graphexplorer/
[Tue Sep 29 12:20:06 2015] [error] [client 192.168.133.157] PHP Warning: implode(): Invalid arguments passed in /usr/local/nagiosxi/html/includes/components/graphexplorer/templates/timeline.inc.php on line 176, referer: http://10.20.1.175/nagiosxi/includes/components/graphexplorer/
[Tue Sep 29 12:24:55 2015] [error] [client 192.168.133.157] PHP Warning: simplexml_load_file(): I/O warning : failed to load external entity "/usr/local/nagios/share/perfdata//_HOST_.xml" in /usr/local/nagiosxi/html/includes/components/graphexplorer/visFunctions.inc.php on line 637, referer: http://10.20.1.175/nagiosxi/includes/components/graphexplorer/
[Tue Sep 29 12:24:55 2015] [error] [client 192.168.133.157] PHP Fatal error: Call to a member function xpath() on a non-object in /usr/local/nagiosxi/html/includes/components/graphexplorer/visFunctions.inc.php on line 638, referer: http://10.20.1.175/nagiosxi/includes/components/graphexplorer/
[Tue Sep 29 13:19:35 2015] [notice] caught SIGTERM, shutting down
[Tue Sep 29 13:20:24 2015] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Tue Sep 29 13:20:24 2015] [notice] Digest: generating secret for digest authentication ...
[Tue Sep 29 13:20:24 2015] [notice] Digest: done
[Tue Sep 29 13:20:27 2015] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.3 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips configured -- resuming normal operations
[Tue Sep 29 16:27:53 2015] [notice] caught SIGTERM, shutting down
[Tue Sep 29 16:31:51 2015] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Tue Sep 29 16:31:51 2015] [notice] Digest: generating secret for digest authentication ...
[Tue Sep 29 16:31:51 2015] [notice] Digest: done
[Tue Sep 29 16:31:53 2015] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.3 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips configured -- resuming normal operations
Re: Empty Event Log
Posted: Tue Sep 29, 2015 1:34 pm
by jdalrymple
Is this a VM or a physical box? Either way can you have the storage administrator of the system take a look at IO latency during the period of time that things went south? I have a hunch you'll see some anomalous readings.
Re: Empty Event Log
Posted: Wed Sep 30, 2015 2:58 am
by adminfun
It's a virtual machine. Installed with an OVF template.
There is a lot going on at night. Backups via the network and such, could it be that the storage box or the network is saturated?
Re: Empty Event Log
Posted: Wed Sep 30, 2015 9:18 am
by jdalrymple
adminfun wrote:could it be that the storage box or the network is saturated
Absolutely - and that's kind of what I was expecting. Maybe correlate the timeline with your backup admins and see if it lines up? The solution if that's the case is our ramdisk modification and to also schedule your XI backups off those hours.
https://assets.nagios.com/downloads/nag ... giosXI.pdf