Page 1 of 1

Nagios xi down need help ASAP database down

Posted: Wed Mar 19, 2014 7:05 am
by charle.bs
Hi, ok im quite new to nagios and learning as I go. I have downloaded the trial of XI and plane to set it up and then activate it and use it for out live monitoring server. I have downloaded the VM and all was working perfect with 5 hosts running on it but today I am getting this error message from the web browser.

DB Connect Error [nagiosxi]: Database connection failed DB Connect Error [nagiosxi]: Database connection failed DB Connect Error [nagiosxi]: Database connection failed.
Here are the results of the df -h command

[root@localhost ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
7.5G 7.2G 0 100% / <-----------------------------ran out of space
tmpfs 935M 0 935M 0% /dev/shm
/dev/sda1 485M 50M 410M 11% /boot

I also cannot start the postgresql service, below are my results

[root@bby1nms01 ~]# service postgresql start
Starting postgresql service: [FAILED]
[root@bby1nms01 ~]#

I have also looked at the PG startup log and get the following

"The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale en_US.UTF-8.
The default database encoding has accordingly been set to UTF8.
The default text search configuration will be set to "english".

fixing permissions on existing directory /var/lib/pgsql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 32MB
creating configuration files ... ok
creating template1 database in /var/lib/pgsql/data/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok

Success. You can now start the database server using:

/usr/bin/postgres -D /var/lib/pgsql/data
or
/usr/bin/pg_ctl -D /var/lib/pgsql/data -l logfile start

could not write to log file: No space left on device
FATAL: could not write lock file "postmaster.pid": No space left on device
FATAL: could not write lock file "postmaster.pid": No space left on device
FATAL: could not write lock file "postmaster.pid": No space left on device
"

Lookes like I have ran our of diskspace but how can I find where and clear some space.

Thank you for your help in advanced

Re: Nagios xi down need help ASAP database down

Posted: Wed Mar 19, 2014 1:10 pm
by abrist
My apologies, support has been crazy today. Lets start by tracking down where the diskspace went:

Code: Select all

cd /
find . -type f -print0 | xargs -0 du | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}
find . -type d -print0 | xargs -0 du | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}
ls /usr/local/nagios/var/spool/xidpe | wc -l
ls /usr/local/nagios/var/spool/perfdata | wc -l
ls /usr/local/nagios/var/spool/checkresults | wc -l
You database will not be able to start as you do not have enough space for it to create a lock file. So once space is cleared, we can bring up the required services.