DB Connect Error

[sheld0r]

I'm getting the following error message when trying to access the Nagios interface

DB Connect Error [nagiosxi]: Database connection failed DB Connect Error [nagiosxi]: Database connection failed DB Connect Error [nagiosxi]: Database connection failed

I've logged into the server and found the issue is because of low disk space. My question is with Nagios, what's using up the space? I would assume logs? I have our Nagios XI installed on CentOS on a 750GB volume. But I don't see a particular directory with a large amount of data. Here are a few from my ncdu view.

3.4GiB /usr
339.9MiB /var
224.4MiB /lib
153.3MiB /store
108.8MiB /home

The rest of the directories are rather small, under 14MiB in size. So my question is, should I be doing routine maintenance on the Nagios box to clear our certain logs?

[scottwilkerson]

can you run

Code: Select all

df -h
df -i

[sheld0r]

Here's the output of the commands you requested scottwilkerson



Uploaded with ImageShack.us

[abrist]

Lets drill down the space issues:

Code: Select all

cd /
du -hsx * | sort -rh | head -10
find . -type f -print0 | xargs -0 du -s | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}
find . -type d -print0 | xargs -0 du -s | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}

[sheld0r]

Alright, here's what I got from those commands:

[root@nagios /]# du -hsx * | sort -rh | head -10
du: cannot access `proc/25186/task/25186/fd/4': No such file or directory
du: cannot access `proc/25186/task/25186/fdinfo/4': No such file or directory
du: cannot access `proc/25186/fd/4': No such file or directory
du: cannot access `proc/25186/fdinfo/4': No such file or directory
40G var
3.5G usr
225M lib
190M store
130M home
54M boot
35M etc
27M lib64
14M sbin
7.6M bin
[root@nagios /]# find . -type f -print0 | xargs -0 du -s | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}
find: `./proc/27659/task/27659/fd/5': No such file or directory
find: `./proc/27659/task/27659/fdinfo/5': No such file or directory
find: `./proc/27659/fd/5': No such file or directory
find: `./proc/27659/fdinfo/5': No such file or directory
701M ./media/440C-7268/Suits.2011.Complete.Season 1/Suits.S01E01.Pilot.HDTV.XviD-FQM.[VTV].avi
1.4G ./media/440C-7268/Arbitrage.2012.HDRiP.AC3-2.0.XviD-AXED/Arbitrage.2012.HDRiP.AC3-2.0.XviD-AXED.avi
1.9G ./var/log/httpd/ssl_error_log-20130331
2.0G ./var/log/httpd/error_log-20130331
2.5G ./var/log/httpd/error_log-20130317
2.5G ./var/log/httpd/error_log-20130324
3.5G ./var/log/httpd/error_log-20130310
6.7G ./var/log/httpd/ssl_error_log-20130310
8.4G ./var/log/httpd/ssl_error_log-20130317
11G ./var/log/httpd/ssl_error_log-20130324
[root@nagios /]# find . -type d -print0 | xargs -0 du -s | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}
find: `./proc/27846/task/27846/fd/5': No such file or directory
find: `./proc/27846/task/27846/fdinfo/5': No such file or directory
find: `./proc/27846/fd/5': No such file or directory
find: `./proc/27846/fdinfo/5': No such file or directory
du: cannot access `./proc/27852/task/27852/fd/4': No such file or directory
du: cannot access `./proc/27852/task/27852/fdinfo/4': No such file or directory
du: cannot access `./proc/27852/fd/4': No such file or directory
du: cannot access `./proc/27852/fdinfo/4': No such file or directory
du: cannot access `./proc/27852/task/27852/fd/4': No such file or directory
du: cannot access `./proc/27852/task/27852/fdinfo/4': No such file or directory
du: cannot access `./proc/27852/fd/4': No such file or directory
du: cannot access `./proc/27852/fdinfo/4': No such file or directory
1.4G ./usr/share
1.4G ./media/440C-7268/Arbitrage.2012.HDRiP.AC3-2.0.XviD-AXED
3.5G ./usr
4.5G ./media/440C-7268/Suits.2011.Complete.Season 1
5.8G ./media/440C-7268
5.8G ./media
40G ./var/log/httpd
40G ./var/log
40G ./var
du: cannot access `./proc/28206': No such file or directory
du: cannot access `./proc/28223/task/28223/fd/4': No such file or directory
du: cannot access `./proc/28223/task/28223/fdinfo/4': No such file or directory
du: cannot access `./proc/28223/fd/4': No such file or directory
du: cannot access `./proc/28223/fdinfo/4': No such file or directory
50G .

[abrist]

1.9G ./var/log/httpd/ssl_error_log-20130331
2.0G ./var/log/httpd/error_log-20130331
2.5G ./var/log/httpd/error_log-20130317
2.5G ./var/log/httpd/error_log-20130324
3.5G ./var/log/httpd/error_log-20130310
6.7G ./var/log/httpd/ssl_error_log-20130310
8.4G ./var/log/httpd/ssl_error_log-20130317
11G ./var/log/httpd/ssl_error_log-20130324

/var/log/httpd is your problem here as it contains around 40gb of logs. If you are unconcerned with retaining the historical log information, you could remove these files open up a ton of space.

Code: Select all

rm /var/log/httpd/error_log-20130331 /var/log/httpd/error_log-20130317 /var/log/httpd/error_log-20130324 /var/log/httpd/error_log-20130310 /var/log/httpd/ssl_error_log-20130310 /var/log/httpd/ssl_error_log-20130317 /var/log/httpd/ssl_error_log-20130324

[sheld0r]

As of right now, I'm not concerned. This is the very early stages of our Nagios deployment. I'm still a little confused as to what else is using up all the space. I have a 750GB volume. Granted, freeing up 40GBs is a good start. I'm thinking the logs filled up quickly because the Nagios server isn't fine tuned yet.

[scottwilkerson]

While you say you have a 750GB volume, only 50GB is allocated to /

The bulk of it is allocated to /home which an XI install barely uses. XI will grow the most over time in
/usr
/store
/var

[abrist]

To see what is repeating in the logs (as it would be a symptomatic of something you may need to fix), try:

Code: Select all

sort /var/log/httpd/ssl_error_log-20130324 | uniq -d

[sheld0r]

Ah I was thinking of the install of CentOS, where I specified to use the entire disk, but of course for Nagios it's differnet. I believe I installed the defaults for Nagios via the command line. Is 50GB default for / ?

It sounds like I didn't allocate the correct amount of space for my Nagios build. Am I able to modify this, or am I stuck now and require a rebuild?