Page 1 of 3
Database Repair has no effect
Posted: Mon Jan 07, 2019 7:16 pm
by matt.lilek
Hello Team,
Happy New Year? Had a bit of issues today database maintenance event manager not running. Tried stopping and starting services, postgresql would not start. Rebooting didnt help, i tried running a cleanup, did a repair, lost the web gui and cannot get it back now. Please help! Thank you in advance.
Re: Database Repair has no effect
Posted: Tue Jan 08, 2019 9:14 am
by scottwilkerson
One of the biggest causes of postgresql not starting is being really low or out of disk space
Can you run the following and show the results
Re: Database Repair has no effect
Posted: Tue Jan 08, 2019 9:43 am
by matt.lilek
Hello Scott, Happy New Year!
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
96G 79G 13G 87% /
tmpfs 6.8G 0 6.8G 0% /dev/shm
/dev/sda1 477M 137M 315M 31% /boot
tmpfs 500M 500M 0 100% /var/nagiosramdisk
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/VolGroup-lv_root
6335472 219242 6116230 4% /
tmpfs 1782240 1 1782239 1% /dev/shm
/dev/sda1 128016 62 127954 1% /boot
tmpfs 1798752 6941 1791811 1% /var/nagiosramdisk
Re: Database Repair has no effect
Posted: Tue Jan 08, 2019 9:54 am
by scottwilkerson
Can you see any errors in the pgstartup log?
Code: Select all
tail -50 /var/lib/pgsql/pgstartup.log
Re: Database Repair has no effect
Posted: Tue Jan 08, 2019 10:17 am
by matt.lilek
[root@att1-nag1 mlilek]# tail -50 /var/lib/pgsql/pgstartup.log
could not write to log file: No space left on device
FATAL: could not write lock file "postmaster.pid": No space left on device
FATAL: lock file "postmaster.pid" already exists
HINT: Is another postmaster (PID 63382) running in data directory "/var/lib/pgs ql/data"?
FATAL: lock file "postmaster.pid" already exists
HINT: Is another postmaster (PID 63382) running in data directory "/var/lib/pgs ql/data"?
FATAL: lock file "postmaster.pid" already exists
HINT: Is another postmaster (PID 65438) running in data directory "/var/lib/pgs ql/data"?
FATAL: lock file "postmaster.pid" already exists
HINT: Is another postmaster (PID 1683) running in data directory "/var/lib/pgsq l/data"?
FATAL: lock file "postmaster.pid" already exists
HINT: Is another postmaster (PID 1683) running in data directory "/var/lib/pgsq l/data"?
FATAL: lock file "postmaster.pid" already exists
HINT: Is another postmaster (PID 1678) running in data directory "/var/lib/pgsq l/data"?
FATAL: lock file "postmaster.pid" already exists
HINT: Is another postmaster (PID 47220) running in data directory "/var/lib/pgs ql/data"?
FATAL: lock file "postmaster.pid" already exists
HINT: Is another postmaster (PID 64328) running in data directory "/var/lib/pgs ql/data"?
FATAL: lock file "postmaster.pid" already exists
HINT: Is another postmaster (PID 64328) running in data directory "/var/lib/pgs ql/data"?
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
Re: Database Repair has no effect
Posted: Tue Jan 08, 2019 10:43 am
by scottwilkerson
What error do you get when you restart postgresql?
Re: Database Repair has no effect
Posted: Tue Jan 08, 2019 11:18 am
by matt.lilek
No issues restarting
Re: Database Repair has no effect
Posted: Tue Jan 08, 2019 12:15 pm
by scottwilkerson
is it running?
If not lets restart and then look at the system log
Code: Select all
service postgresql restart
tail -50 /var/log/messages
Re: Database Repair has no effect
Posted: Tue Jan 08, 2019 12:32 pm
by matt.lilek
its already running
Re: Database Repair has no effect
Posted: Tue Jan 08, 2019 12:34 pm
by matt.lilek
Jan 8 12:25:47 att1-nag1 ndo2db: Trimming hostchecks.
Jan 8 12:25:47 att1-nag1 ndo2db: Trimming eventhandlers.
Jan 8 12:26:31 att1-nag1 check_nrpe: Remote 10.41.76.178 does not support Version 3 Packets
Jan 8 12:26:48 att1-nag1 ndo2db: Trimming timedevents.
Jan 8 12:26:48 att1-nag1 ndo2db: Trimming systemcommands.
Jan 8 12:26:48 att1-nag1 ndo2db: Trimming servicechecks.
Jan 8 12:26:48 att1-nag1 ndo2db: Trimming hostchecks.
Jan 8 12:26:48 att1-nag1 ndo2db: Trimming eventhandlers.
Jan 8 12:26:54 att1-nag1 check_nrpe: Remote 10.41.76.178 does not support Version 3 Packets
Jan 8 12:27:10 att1-nag1 check_nrpe: Remote 10.41.76.178 does not support Version 3 Packets
Jan 8 12:27:18 att1-nag1 check_nrpe: Remote 10.41.76.178 does not support Version 3 Packets
Jan 8 12:27:23 att1-nag1 check_nrpe: Remote 10.41.76.178 does not support Version 3 Packets
Jan 8 12:27:49 att1-nag1 ndo2db: Trimming timedevents.
Jan 8 12:27:49 att1-nag1 ndo2db: Trimming systemcommands.
Jan 8 12:27:49 att1-nag1 ndo2db: Trimming servicechecks.
Jan 8 12:27:49 att1-nag1 ndo2db: Trimming hostchecks.
Jan 8 12:27:49 att1-nag1 ndo2db: Trimming eventhandlers.
Jan 8 12:27:54 att1-nag1 check_nrpe: Remote 10.41.76.178 does not support Version 3 Packets
Jan 8 12:28:19 att1-nag1 check_nrpe: Remote 10.41.76.178 does not support Version 3 Packets
Jan 8 12:28:50 att1-nag1 ndo2db: Trimming timedevents.
Jan 8 12:28:50 att1-nag1 ndo2db: Trimming systemcommands.
Jan 8 12:28:50 att1-nag1 ndo2db: Trimming servicechecks.
Jan 8 12:28:50 att1-nag1 ndo2db: Trimming hostchecks.
Jan 8 12:28:50 att1-nag1 ndo2db: Trimming eventhandlers.
Jan 8 12:28:56 att1-nag1 check_nrpe: Remote 10.41.76.178 does not support Version 3 Packets
Jan 8 12:29:10 att1-nag1 check_nrpe: Remote 10.41.76.178 does not support Version 3 Packets
Jan 8 12:29:23 att1-nag1 check_nrpe: Remote 10.41.76.178 does not support Version 3 Packets
Jan 8 12:29:51 att1-nag1 ndo2db: Trimming timedevents.
Jan 8 12:29:51 att1-nag1 ndo2db: Trimming systemcommands.
Jan 8 12:29:51 att1-nag1 ndo2db: Trimming servicechecks.
Jan 8 12:29:51 att1-nag1 ndo2db: Trimming hostchecks.
Jan 8 12:29:51 att1-nag1 ndo2db: Trimming eventhandlers.
Jan 8 12:30:05 att1-nag1 check_nrpe: Remote 10.41.76.178 does not support Version 3 Packets
Jan 8 12:30:16 att1-nag1 check_nrpe: Remote 10.41.76.178 does not support Version 3 Packets
Jan 8 12:31:25 att1-nag1 ndo2db: Trimming timedevents.
Jan 8 12:31:25 att1-nag1 ndo2db: Trimming systemcommands.
Jan 8 12:31:25 att1-nag1 ndo2db: Trimming servicechecks.
Jan 8 12:31:25 att1-nag1 ndo2db: Trimming hostchecks.
Jan 8 12:31:25 att1-nag1 ndo2db: Trimming eventhandlers.
Jan 8 12:31:30 att1-nag1 check_nrpe: Remote 10.41.76.178 does not support Version 3 Packets
Jan 8 12:31:53 att1-nag1 check_nrpe: Remote 10.41.76.178 does not support Version 3 Packets
Jan 8 12:32:08 att1-nag1 check_nrpe: Remote 10.41.76.178 does not support Version 3 Packets
Jan 8 12:32:17 att1-nag1 check_nrpe: Remote 10.41.76.178 does not support Version 3 Packets
Jan 8 12:32:21 att1-nag1 check_nrpe: Remote 10.41.76.178 does not support Version 3 Packets
Jan 8 12:32:42 att1-nag1 ndo2db: Trimming timedevents.
Jan 8 12:32:42 att1-nag1 ndo2db: Trimming systemcommands.
Jan 8 12:32:42 att1-nag1 ndo2db: Trimming servicechecks.
Jan 8 12:32:42 att1-nag1 ndo2db: Trimming hostchecks.
Jan 8 12:32:42 att1-nag1 ndo2db: Trimming eventhandlers.
Jan 8 12:32:53 att1-nag1 check_nrpe: Remote 10.41.76.178 does not support Version 3 Packets