Database Repair has no effect

[matt.lilek]

Hello Team,

Happy New Year? Had a bit of issues today database maintenance event manager not running. Tried stopping and starting services, postgresql would not start. Rebooting didnt help, i tried running a cleanup, did a repair, lost the web gui and cannot get it back now. Please help! Thank you in advance.

[scottwilkerson]

One of the biggest causes of postgresql not starting is being really low or out of disk space

Can you run the following and show the results

Code: Select all

df -h
df -i

[matt.lilek]

Hello Scott, Happy New Year!

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
96G 79G 13G 87% /
tmpfs 6.8G 0 6.8G 0% /dev/shm
/dev/sda1 477M 137M 315M 31% /boot
tmpfs 500M 500M 0 100% /var/nagiosramdisk

Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/VolGroup-lv_root
6335472 219242 6116230 4% /
tmpfs 1782240 1 1782239 1% /dev/shm
/dev/sda1 128016 62 127954 1% /boot
tmpfs 1798752 6941 1791811 1% /var/nagiosramdisk

[scottwilkerson]

Can you see any errors in the pgstartup log?

Code: Select all

tail -50 /var/lib/pgsql/pgstartup.log

[matt.lilek]

[root@att1-nag1 mlilek]# tail -50 /var/lib/pgsql/pgstartup.log
could not write to log file: No space left on device
FATAL: could not write lock file "postmaster.pid": No space left on device
FATAL: lock file "postmaster.pid" already exists
HINT: Is another postmaster (PID 63382) running in data directory "/var/lib/pgs ql/data"?
FATAL: lock file "postmaster.pid" already exists
HINT: Is another postmaster (PID 63382) running in data directory "/var/lib/pgs ql/data"?
FATAL: lock file "postmaster.pid" already exists
HINT: Is another postmaster (PID 65438) running in data directory "/var/lib/pgs ql/data"?
FATAL: lock file "postmaster.pid" already exists
HINT: Is another postmaster (PID 1683) running in data directory "/var/lib/pgsq l/data"?
FATAL: lock file "postmaster.pid" already exists
HINT: Is another postmaster (PID 1683) running in data directory "/var/lib/pgsq l/data"?
FATAL: lock file "postmaster.pid" already exists
HINT: Is another postmaster (PID 1678) running in data directory "/var/lib/pgsq l/data"?
FATAL: lock file "postmaster.pid" already exists
HINT: Is another postmaster (PID 47220) running in data directory "/var/lib/pgs ql/data"?
FATAL: lock file "postmaster.pid" already exists
HINT: Is another postmaster (PID 64328) running in data directory "/var/lib/pgs ql/data"?
FATAL: lock file "postmaster.pid" already exists
HINT: Is another postmaster (PID 64328) running in data directory "/var/lib/pgs ql/data"?
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device

[scottwilkerson]

What error do you get when you restart postgresql?

[matt.lilek]

No issues restarting

[scottwilkerson]

is it running?

Code: Select all

service postgresql status

If not lets restart and then look at the system log

Code: Select all

service postgresql restart
tail -50 /var/log/messages

[matt.lilek]

its already running

[matt.lilek]

Jan 8 12:25:47 att1-nag1 ndo2db: Trimming hostchecks.
Jan 8 12:25:47 att1-nag1 ndo2db: Trimming eventhandlers.
Jan 8 12:26:31 att1-nag1 check_nrpe: Remote 10.41.76.178 does not support Version 3 Packets
Jan 8 12:26:48 att1-nag1 ndo2db: Trimming timedevents.
Jan 8 12:26:48 att1-nag1 ndo2db: Trimming systemcommands.
Jan 8 12:26:48 att1-nag1 ndo2db: Trimming servicechecks.
Jan 8 12:26:48 att1-nag1 ndo2db: Trimming hostchecks.
Jan 8 12:26:48 att1-nag1 ndo2db: Trimming eventhandlers.
Jan 8 12:26:54 att1-nag1 check_nrpe: Remote 10.41.76.178 does not support Version 3 Packets
Jan 8 12:27:10 att1-nag1 check_nrpe: Remote 10.41.76.178 does not support Version 3 Packets
Jan 8 12:27:18 att1-nag1 check_nrpe: Remote 10.41.76.178 does not support Version 3 Packets
Jan 8 12:27:23 att1-nag1 check_nrpe: Remote 10.41.76.178 does not support Version 3 Packets
Jan 8 12:27:49 att1-nag1 ndo2db: Trimming timedevents.
Jan 8 12:27:49 att1-nag1 ndo2db: Trimming systemcommands.
Jan 8 12:27:49 att1-nag1 ndo2db: Trimming servicechecks.
Jan 8 12:27:49 att1-nag1 ndo2db: Trimming hostchecks.
Jan 8 12:27:49 att1-nag1 ndo2db: Trimming eventhandlers.
Jan 8 12:27:54 att1-nag1 check_nrpe: Remote 10.41.76.178 does not support Version 3 Packets
Jan 8 12:28:19 att1-nag1 check_nrpe: Remote 10.41.76.178 does not support Version 3 Packets
Jan 8 12:28:50 att1-nag1 ndo2db: Trimming timedevents.
Jan 8 12:28:50 att1-nag1 ndo2db: Trimming systemcommands.
Jan 8 12:28:50 att1-nag1 ndo2db: Trimming servicechecks.
Jan 8 12:28:50 att1-nag1 ndo2db: Trimming hostchecks.
Jan 8 12:28:50 att1-nag1 ndo2db: Trimming eventhandlers.
Jan 8 12:28:56 att1-nag1 check_nrpe: Remote 10.41.76.178 does not support Version 3 Packets
Jan 8 12:29:10 att1-nag1 check_nrpe: Remote 10.41.76.178 does not support Version 3 Packets
Jan 8 12:29:23 att1-nag1 check_nrpe: Remote 10.41.76.178 does not support Version 3 Packets
Jan 8 12:29:51 att1-nag1 ndo2db: Trimming timedevents.
Jan 8 12:29:51 att1-nag1 ndo2db: Trimming systemcommands.
Jan 8 12:29:51 att1-nag1 ndo2db: Trimming servicechecks.
Jan 8 12:29:51 att1-nag1 ndo2db: Trimming hostchecks.
Jan 8 12:29:51 att1-nag1 ndo2db: Trimming eventhandlers.
Jan 8 12:30:05 att1-nag1 check_nrpe: Remote 10.41.76.178 does not support Version 3 Packets
Jan 8 12:30:16 att1-nag1 check_nrpe: Remote 10.41.76.178 does not support Version 3 Packets
Jan 8 12:31:25 att1-nag1 ndo2db: Trimming timedevents.
Jan 8 12:31:25 att1-nag1 ndo2db: Trimming systemcommands.
Jan 8 12:31:25 att1-nag1 ndo2db: Trimming servicechecks.
Jan 8 12:31:25 att1-nag1 ndo2db: Trimming hostchecks.
Jan 8 12:31:25 att1-nag1 ndo2db: Trimming eventhandlers.
Jan 8 12:31:30 att1-nag1 check_nrpe: Remote 10.41.76.178 does not support Version 3 Packets
Jan 8 12:31:53 att1-nag1 check_nrpe: Remote 10.41.76.178 does not support Version 3 Packets
Jan 8 12:32:08 att1-nag1 check_nrpe: Remote 10.41.76.178 does not support Version 3 Packets
Jan 8 12:32:17 att1-nag1 check_nrpe: Remote 10.41.76.178 does not support Version 3 Packets
Jan 8 12:32:21 att1-nag1 check_nrpe: Remote 10.41.76.178 does not support Version 3 Packets
Jan 8 12:32:42 att1-nag1 ndo2db: Trimming timedevents.
Jan 8 12:32:42 att1-nag1 ndo2db: Trimming systemcommands.
Jan 8 12:32:42 att1-nag1 ndo2db: Trimming servicechecks.
Jan 8 12:32:42 att1-nag1 ndo2db: Trimming hostchecks.
Jan 8 12:32:42 att1-nag1 ndo2db: Trimming eventhandlers.
Jan 8 12:32:53 att1-nag1 check_nrpe: Remote 10.41.76.178 does not support Version 3 Packets