Filesystem filled up, now Nagios XI is not fully functional

[sgd]

The last 50 lines of /var/log/msyqld.log:

Code: Select all

150312 10:01:35 [Note] Event Scheduler: Purging the queue. 0 events
150312 10:01:37  InnoDB: Starting shutdown...
150312 10:01:39  InnoDB: Shutdown completed; log sequence number 0 44233
150312 10:01:39 [Note] /usr/libexec/mysqld: Shutdown complete

150312 10:01:39 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150312 10:01:41 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150312 10:01:41  InnoDB: Started; log sequence number 0 44233
150312 10:01:41 [Note] Event Scheduler: Loaded 0 events
150312 10:01:41 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.52'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
150312 11:30:16 [Note] /usr/libexec/mysqld: Normal shutdown

150312 11:30:16 [Note] Event Scheduler: Purging the queue. 0 events
150312 11:30:18  InnoDB: Starting shutdown...
150312 11:30:22  InnoDB: Shutdown completed; log sequence number 0 44233
150312 11:30:22 [Note] /usr/libexec/mysqld: Shutdown complete

150312 11:30:22 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150312 11:30:22 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150312 11:30:23  InnoDB: Started; log sequence number 0 44233
150312 11:30:23 [Note] Event Scheduler: Loaded 0 events
150312 11:30:23 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.52'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
150312 12:57:53 [Note] /usr/libexec/mysqld: Normal shutdown

150312 12:57:53 [Note] Event Scheduler: Purging the queue. 0 events
150312 12:57:55  InnoDB: Starting shutdown...
150312 12:57:58  InnoDB: Shutdown completed; log sequence number 0 44233
150312 12:57:58 [Note] /usr/libexec/mysqld: Shutdown complete

150312 12:57:58 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150312 12:58:32 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150312 12:58:32  InnoDB: Started; log sequence number 0 44233
150312 12:58:32 [Note] Event Scheduler: Loaded 0 events
150312 12:58:32 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.52'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
150312 13:17:57 [Note] /usr/libexec/mysqld: Normal shutdown

150312 13:17:57 [Note] Event Scheduler: Purging the queue. 0 events
150312 13:17:59  InnoDB: Starting shutdown...
150312 13:18:04  InnoDB: Shutdown completed; log sequence number 0 44233
150312 13:18:04 [Note] /usr/libexec/mysqld: Shutdown complete

150312 13:18:04 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150312 13:21:57 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150312 13:21:57  InnoDB: Started; log sequence number 0 44233
150312 13:21:57 [Note] Event Scheduler: Loaded 0 events
150312 13:21:57 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.52'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution

The result of reconfiguring Nagios:

Code: Select all

Script started on Thu 12 Mar 2015 02:47:35 PM PDT
[root@nagios scripts]# ./reconfigure_nagios.sh 
URL: http://localhost/nagiosxi/includes/components/ccm/
CMDLINE
/usr/bin/wget --save-cookies nagiosql.cookies --keep-session-cookies htlhost/nagiosxi/includes/components/ccm/ --no-check-certificate --post-dit=Login&hidelog=true&loginSubmitted=true&username=nagiosxi&password=m3nagiosql.login--2015-03-12 14:47:52--  http://localhost/nagiosxi/includents/ccm/
Resolving localhost... 127.0.0.1
Connecting to localhost|127.0.0.1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ânagiosql.loginânagiosql.loginânagiosql.loginânagiosql.login


    [<=>                                    ] 0           --.-K/s                 
 [ <=>                                   ] 8,263       --.-K/s                  [ 
 <=>                                  ] 8,263       --.-K/s                  [   <
=>                                 ] 8,263       --.-K/s                  [    <=>
                                ] 8,263       --.-K/s                  [     <=>  
                             ] 13,685      2.84K/s                  [      <=>    
                          ] 13,685      2.84K/s   in 4.7s    

2015-03-12 14:48:02 (2.84 KB/s) - ânagiosql.export.monitoringâ

WRITE CONFIGS SUCCESSFUL!
OUTPUT: 
Nagios Core 4.0.8
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 08-12-2014
License: GPL

Website: http://www.nagios.org
Reading configuration data...
   Read main config file okay...
Warning: Duplicate definition found for service 'HTTP' on host 'cgwifi.net' (confi
g file '/usr/local/nagios/etc/services/cgwifi.net.cfg', starting on line 29)
Warning: Duplicate definition found for service 'HTTP' on host 'www.cgwifi.net' (c
onfig file '/usr/local/nagios/etc/services/magnetite-services.cfg', starting on li
ne 14)
   Read object config files okay...

Running pre-flight check on configuration data...

Checking objects...
Warning: Service 'hermes-zombies' on host 'hermes.earthclick.net' has no default contacts or contactgroups defined!
Warning: Service 'nephele-zombies' on host 'nephele.earthclick.net' has no default contacts or contactgroups defined!
        Checked 240 services.
        Checked 66 hosts.
        Checked 9 host groups.
        Checked 3 service groups.
        Checked 9 contacts.
        Checked 3 contact groups.
        Checked 119 commands.
        Checked 14 time periods.
        Checked 0 host escalations.
        Checked 0 service escalations.
Checking for circular paths...
        Checked 66 hosts
        Checked 0 service dependencies
        Checked 0 host dependencies
        Checked 14 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 2
Total Errors:   0

Things look okay - No serious problems were detected during the pre-flight check
RET: 0
Running configuration check...
Stopping nagios:. done.
Starting nagios: done.
[root@nagios scripts]# exit

Still no status change.

[cmerchant]

I've run the fixperms.sh script.

I think you should be running:

Code: Select all

cd /usr/local/nagiosxi/scripts
./reset_config_perms

and show us the output.

[sgd]

Code: Select all

[root@nagios scripts]# ./reset_config_perms
SETUID ROOT OK
RESETTING PERMS

[sgd]

When I filled up /usr/local (including /usr/local/store) might something critical have been truncated or zeroed out? That is when the problem began.

[sgd]

Hello,

We need to resolve this issue as soon as possible - it's severely impacting our business now. When I did the database rebuilds yesterday several previously monitored objects were dropped, and I still cannot provision new monitored services. We're a small company with 8x5 live coverage, and depend on Nagios XI to do off-hours monitoring. Until we get this fixed I have to have live 24/7 manual monitoring of services, which is very expensive.

We're at a point where restoring from an older backup would be acceptable if we can't fix the issue. Please advise.

[tmcdonald]

We just had Mike from Sales come down and inform us that you're a current customer, so I am going to move this thread to the Customer forum where our SLA will apply. However, it might be time to move this to a ticket so a remote session can be performed. If you want to open a ticket, you can email [email protected] with a link to this thread, a descriptive subject, and a short summary of the problem in the body.

In the future if you would like to start an issue on the forums it would be best to post in the Customer section: http://support.nagios.com/forum/viewforum.php?f=3

Customers usually post in General when it's either not a critical issue or if they want to share their info/resolution with the world.

[sgd]

Hi, Thanks for escalating this.

I'm confused, though, because I started this thread in Customer Support -> Nagios XI, which I thought was the customer support forum. If there is somewhere else this should go, please let me know.

[tmcdonald]

This section of the forum is the correct place for customers to post, but looking through the logs on this thread all I can see action-wise was that it was moved from General to Customer 30 minutes ago by myself. There is no listing of it having been moved in the opposite direction.

At any rate, will you be opening a ticket or would you like to continue here? If we will be moving this to a ticket I will close the thread once you submit it.

[sgd]

Hi, I just emailed you to open a support case that way, as it seems you're better equipped to respond quickly to emails than the support forum. Please feel free to close this thread and we can proceed with the email case.
Thanks!

[lmiltchev]

We will continue communicating via emails. I am locking this topic.