@#$%^ postgresql ....

[benhank]

hey all this is odd

Code: Select all

[root@lkennagiost01 ~]# service postgresql restart
Stopping postgresql service:                               [FAILED]
Starting postgresql service:                               [  OK  ]
[root@lkennagiost01 ~]# service postgresql restart
Stopping postgresql service:                               [  OK  ]
Starting postgresql service:                               [  OK  ]
[root@lkennagiost01 ~]# service postgresql restart
Stopping postgresql service:                               [  OK  ]
Starting postgresql service:                               [  OK  ]
[root@lkennagiost01 ~]# service postgresql restart
Stopping postgresql service:                               [FAILED]
Starting postgresql service:                               [  OK  ]
You have new mail in /var/spool/mail/root
[root@lkennagiost01 ~]# echo "\d"|psql nagiosxi nagiosxi
psql: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/tmp/.s.PGSQL.5432"?
You have new mail in /var/spool/mail/root
[root@lkennagiost01 ~]#

See that second "FAILED"? that happened ablot 3 mins after the service was successfully started.
and it gets worse...

Code: Select all

Stopping postgresql service:                               [FAILED]
Starting postgresql service:                               [  OK  ]
You have new mail in /var/spool/mail/root
[root@lkennagiost01 ~]# echo "\d"|psql nagiosxi nagiosxi
psql: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/tmp/.s.PGSQL.5432"?
You have new mail in /var/spool/mail/root
[root@lkennagiost01 ~]# service postgresql restart
Stopping postgresql service:                               [FAILED]
Starting postgresql service:                               [  OK  ]
You have new mail in /var/spool/mail/root
[root@lkennagiost01 ~]# echo "\d"|psql nagiosxi nagiosxi
psql: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/tmp/.s.PGSQL.5432"?
[root@lkennagiost01 ~]# service postgresql restart
Stopping postgresql service:                               [  OK  ]
Starting postgresql service:                               [  OK  ]
[root@lkennagiost01 ~]# echo "\d"|psql nagiosxi nagiosxi
psql: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/tmp/.s.PGSQL.5432"?
[root@lkennagiost01 ~]#

This next line is not for children, the faint of heart, or those who suffer from panic attacks. you have been warned....:



I even went windows on it and rebooted the server.

[sreinhardt]

Since we are at an interesting point in time... 2012 or 2014? Has this happened post *gasp* reboot? Any messages in /var/log/messages?

Code: Select all

grep -Ri 'pgsql' /var/log/messages
grep -Ri 'postgres' /var/log/messages

[benhank]

yeah the reboot did nuffin
no no messages. gotta bouch, can we continue tommorow?

[scottwilkerson]

Is the disk full?

Code: Select all

df -h
df -i

[benhank]

wait a minute. wait aminte wait a dang minUTE.
Is the the genuine originall 100% all natual no addatives or preservatives, not available in any stores, deadeye,bullseye, "I forgot more than you will ever know", Iam the man rappers bragabout, when God made me,there was no mold to break, I am runnin things up in here, Yeah I stole you girl, you can have her back, Red hot, you cant handle what I got:

y scottwilkerson

SAAAYYYY whaaat!
where you been bro? we can discuss that in pm's.


ok on to bidness:

Code: Select all

[root@lkennagiost01 ~]# df -h
Filesystem                            Size  Used Avail Use% Mounted on
/dev/mapper/vg_lkennagiost01-lv_root   50G  7.5G   40G  16% /
tmpfs                                 7.8G   12K  7.8G   1% /dev/shm
/dev/sda1                             485M  149M  311M  33% /boot
/dev/mapper/vg_lkennagiost01-lv_home  1.6T   76G  1.4T   6% /home
[root@lkennagiost01 ~]# df -i
Filesystem                              Inodes  IUsed     IFree IUse% Mounted on
/dev/mapper/vg_lkennagiost01-lv_root   3276800 146492   3130308    5% /
tmpfs                                  2041124      4   2041120    1% /dev/shm
/dev/sda1                               128016     67    127949    1% /boot
/dev/mapper/vg_lkennagiost01-lv_home 105013248  69312 104943936    1% /home
[root@lkennagiost01 ~]#

[abrist]

Hey Ben, any interesting info in the postgres startup log?

Code: Select all

tail -55 /var/lib/pgsql/pgstartup.log

[benhank]

sorry about the slipup fellas. =(

[abrist]

No problem. How about that log?

[benhank]

Code: Select all

[root@lkennagiost01 ~]# tail -55 /var/lib/pgsql/pgstartup.log
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale en_US.UTF-8.
The default database encoding has accordingly been set to UTF8.
The default text search configuration will be set to "english".

fixing permissions on existing directory /var/lib/pgsql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 32MB
creating configuration files ... ok
creating template1 database in /var/lib/pgsql/data/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok

Success. You can now start the database server using:

    /usr/bin/postgres -D /var/lib/pgsql/data
or
    /usr/bin/pg_ctl -D /var/lib/pgsql/data -l logfile start

FATAL:  too many private dirs demanded
FATAL:  too many private dirs demanded
FATAL:  too many private dirs demanded
FATAL:  too many private dirs demanded
FATAL:  too many private dirs demanded
FATAL:  too many private dirs demanded
FATAL:  too many private dirs demanded
FATAL:  too many private dirs demanded
FATAL:  too many private dirs demanded
FATAL:  too many private dirs demanded
FATAL:  too many private dirs demanded
You have new mail in /var/spool/mail/root
[root@lkennagiost01 ~]#

so I ran those two cmd lines and got:

You have new mail in /var/spool/mail/root

Code: Select all

[root@lkennagiost01 ~]# /usr/bin/postgres -D /var/lib/pgsql/data
"root" execution of the PostgreSQL server is not permitted.
The server must be started under an unprivileged user ID to prevent
possible system security compromise.  See the documentation for
more information on how to properly start the server.
[root@lkennagiost01 ~]# /usr/bin/pg_ctl -D /var/lib/pgsql/data -l logfile start
pg_ctl: cannot be run as root
Please log in (using, e.g., "su") as the (unprivileged) user that will
own the server process.

and THEN

Code: Select all

[root@lkennagiost01 ~]# /var/spool/mail/root
-bash: /var/spool/mail/root: Permission denied

I dont get it I made no changes to my system. I cant even get to var log messages.

Is there a way to see who made changes to my machine and what those changes were?

[abrist]

Well, this is an esoteric error. Rare indeed.
There was a postgres bug floating around for a while, does the following command fail due to a detected loop?

Code: Select all

find -L /usr/share/zoneinfo/ 

Lets also get an strace of this:

Code: Select all

yum install -y strace
su postgres
strace -o gregtest -ff -t pg_ctl start -D /var/lib/pgsql/data -l /tmp/postgres.startup.log

Once it stops, ctrl-c it and PM me a copy of /tmp/postgres.startup.log.