Nagios Support Forum

I'm getting the following error message when trying to access the Nagios interface

DB Connect Error [nagiosxi]: Database connection failed DB Connect Error [nagiosxi]: Database connection failed DB Connect Error [nagiosxi]: Database connection failed

I've logged into the server and found the issue is because of low disk space. My question is with Nagios, what's using up the space? I would assume logs? I have our Nagios XI installed on CentOS on a 750GB volume. But I don't see a particular directory with a large amount of data. Here are a few from my ncdu view.

3.4GiB /usr
339.9MiB /var
224.4MiB /lib
153.3MiB /store
108.8MiB /home

The rest of the directories are rather small, under 14MiB in size. So my question is, should I be doing routine maintenance on the Nagios box to clear our certain logs?

can you run

Here's the output of the commands you requested scottwilkerson



Uploaded with ImageShack.us

Lets drill down the space issues:

Alright, here's what I got from those commands:

[root@nagios /]# du -hsx * | sort -rh | head -10
du: cannot access `proc/25186/task/25186/fd/4': No such file or directory
du: cannot access `proc/25186/task/25186/fdinfo/4': No such file or directory
du: cannot access `proc/25186/fd/4': No such file or directory
du: cannot access `proc/25186/fdinfo/4': No such file or directory
40G var
3.5G usr
225M lib
190M store
130M home
54M boot
35M etc
27M lib64
14M sbin
7.6M bin
[root@nagios /]# find . -type f -print0 | xargs -0 du -s | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}
find: `./proc/27659/task/27659/fd/5': No such file or directory
find: `./proc/27659/task/27659/fdinfo/5': No such file or directory
find: `./proc/27659/fd/5': No such file or directory
find: `./proc/27659/fdinfo/5': No such file or directory
701M ./media/440C-7268/Suits.2011.Complete.Season 1/Suits.S01E01.Pilot.HDTV.XviD-FQM.[VTV].avi
1.4G ./media/440C-7268/Arbitrage.2012.HDRiP.AC3-2.0.XviD-AXED/Arbitrage.2012.HDRiP.AC3-2.0.XviD-AXED.avi
1.9G ./var/log/httpd/ssl_error_log-20130331
2.0G ./var/log/httpd/error_log-20130331
2.5G ./var/log/httpd/error_log-20130317
2.5G ./var/log/httpd/error_log-20130324
3.5G ./var/log/httpd/error_log-20130310
6.7G ./var/log/httpd/ssl_error_log-20130310
8.4G ./var/log/httpd/ssl_error_log-20130317
11G ./var/log/httpd/ssl_error_log-20130324
[root@nagios /]# find . -type d -print0 | xargs -0 du -s | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}
find: `./proc/27846/task/27846/fd/5': No such file or directory
find: `./proc/27846/task/27846/fdinfo/5': No such file or directory
find: `./proc/27846/fd/5': No such file or directory
find: `./proc/27846/fdinfo/5': No such file or directory
du: cannot access `./proc/27852/task/27852/fd/4': No such file or directory
du: cannot access `./proc/27852/task/27852/fdinfo/4': No such file or directory
du: cannot access `./proc/27852/fd/4': No such file or directory
du: cannot access `./proc/27852/fdinfo/4': No such file or directory
du: cannot access `./proc/27852/task/27852/fd/4': No such file or directory
du: cannot access `./proc/27852/task/27852/fdinfo/4': No such file or directory
du: cannot access `./proc/27852/fd/4': No such file or directory
du: cannot access `./proc/27852/fdinfo/4': No such file or directory
1.4G ./usr/share
1.4G ./media/440C-7268/Arbitrage.2012.HDRiP.AC3-2.0.XviD-AXED
3.5G ./usr
4.5G ./media/440C-7268/Suits.2011.Complete.Season 1
5.8G ./media/440C-7268
5.8G ./media
40G ./var/log/httpd
40G ./var/log
40G ./var
du: cannot access `./proc/28206': No such file or directory
du: cannot access `./proc/28223/task/28223/fd/4': No such file or directory
du: cannot access `./proc/28223/task/28223/fdinfo/4': No such file or directory
du: cannot access `./proc/28223/fd/4': No such file or directory
du: cannot access `./proc/28223/fdinfo/4': No such file or directory
50G .

sheld0r wrote:1.9G ./var/log/httpd/ssl_error_log-20130331
2.0G ./var/log/httpd/error_log-20130331
2.5G ./var/log/httpd/error_log-20130317
2.5G ./var/log/httpd/error_log-20130324
3.5G ./var/log/httpd/error_log-20130310
6.7G ./var/log/httpd/ssl_error_log-20130310
8.4G ./var/log/httpd/ssl_error_log-20130317
11G ./var/log/httpd/ssl_error_log-20130324

/var/log/httpd is your problem here as it contains around 40gb of logs. If you are unconcerned with retaining the historical log information, you could remove these files open up a ton of space.

As of right now, I'm not concerned. This is the very early stages of our Nagios deployment. I'm still a little confused as to what else is using up all the space. I have a 750GB volume. Granted, freeing up 40GBs is a good start. I'm thinking the logs filled up quickly because the Nagios server isn't fine tuned yet.

While you say you have a 750GB volume, only 50GB is allocated to /

The bulk of it is allocated to /home which an XI install barely uses. XI will grow the most over time in
/usr
/store
/var

To see what is repeating in the logs (as it would be a symptomatic of something you may need to fix), try:

Ah I was thinking of the install of CentOS, where I specified to use the entire disk, but of course for Nagios it's differnet. I believe I installed the defaults for Nagios via the command line. Is 50GB default for / ?

It sounds like I didn't allocate the correct amount of space for my Nagios build. Am I able to modify this, or am I stuck now and require a rebuild?