Postgresql and msqld fail to start

[matt.lilek]

Hello Team,

It has been quite some time and hope all is well on your end. It seems as though i was having a DB error which i tried to run the repair script on and it was unable to start mysqld. I tried restarting all services in order and noticed postresql failing to start as well. I tried to run the same type of repair that would occur after the disks get full (was not the case just thought a clean start would help) just to clear everything out and didnt get anywhere there either. Reboot didnt help either, think i tried that somewhere inbetween the other stuff. Let me know what i should try next.

Thank you,

Matt

[mbellerue]

Hi Matt,

First, do you know if your Nagios databases are hosted on MySQL or PostgreSQL?

Next, go ahead and try to start the database. Then when it fails run,

Code: Select all

journalctl -xe

and send us the output.

[matt.lilek]

[root@ATT3-NAG1 ~]# journalctl -xe
bash: journalctl: command not found

[mbellerue]

Yikes. What OS is this?

We can do it the hard way and do,

Code: Select all

tail -20 /var/log/syslog

We'll see if we can find anything related to the failed startup in there.

There may also be something in,

Code: Select all

tail -20 /var/log/mysql/error.log

Assuming the Nagios databases actually live in MySQL rather than Postgres.

[matt.lilek]

Linux ATT3-NAG1.global.amec.com 2.6.32-754.10.1.el6.x86_64 #1 SMP Tue Jan 15 17:07:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Am i doing something horribly wrong here, lol
[root@ATT3-NAG1 ~]# tail -20 /var/log/syslog
tail: cannot open `/var/log/syslog' for reading: No such file or directory
[root@ATT3-NAG1 ~]# tail -20 /var/log/mysql/error.log
tail: cannot open `/var/log/mysql/error.log' for reading: No such file or directory

[mbellerue]

Nope, you're doing everything right. I'm giving you file locations and commands for newer versions of Linux. That's my bad. Let's try this again.

Code: Select all

tail -20 /var/log/messages
tail -20 /var/log/mysqld.log

[matt.lilek]

Sep 9 22:53:14 ATT3-NAG1 ntpd[1800]: Listen normally on 5 eth0 fe80::250:56ff:fea2:79e1 UDP 123
Sep 9 22:53:14 ATT3-NAG1 ntpd[1800]: Listening on routing socket on fd #22 for interface updates
Sep 9 22:53:19 ATT3-NAG1 ntpd[1800]: 0.0.0.0 c016 06 restart
Sep 9 22:53:19 ATT3-NAG1 ntpd[1800]: 0.0.0.0 c012 02 freq_set kernel 0.000 PPM
Sep 9 22:53:19 ATT3-NAG1 ntpd[1800]: 0.0.0.0 c011 01 freq_not_set
Sep 9 22:53:25 ATT3-NAG1 ntpd[1800]: 0.0.0.0 c61c 0c clock_step -0.383738 s
Sep 9 22:53:25 ATT3-NAG1 ntpd[1800]: 0.0.0.0 c614 04 freq_mode
Sep 9 22:53:26 ATT3-NAG1 ntpd[1800]: 0.0.0.0 c618 08 no_sys_peer
Sep 9 22:55:18 ATT3-NAG1 nrpe[26281]: Starting up daemon
Sep 9 22:55:18 ATT3-NAG1 nrpe[26281]: Server listening on 0.0.0.0 port 5666.
Sep 9 22:55:18 ATT3-NAG1 nrpe[26281]: Server listening on :: port 5666.
Sep 9 22:55:18 ATT3-NAG1 nrpe[26281]: Listening for connections on port 5666
Sep 9 22:55:18 ATT3-NAG1 nrpe[26281]: Allowing connections from: 127.0.0.1,::1
Sep 9 22:55:20 ATT3-NAG1 ndo2db: Warning: Unable to write pid to lockfile (errno 28): No space left on device
Sep 9 23:00:27 ATT3-NAG1 init: tty (/dev/tty1) main process ended, respawning
Sep 9 23:07:47 ATT3-NAG1 ndo2db: Warning: Unable to write pid to lockfile (errno 28): No space left on device
Sep 9 23:08:32 ATT3-NAG1 ntpd[1800]: 0.0.0.0 c612 02 freq_set kernel 16.025 PPM
Sep 9 23:08:32 ATT3-NAG1 ntpd[1800]: 0.0.0.0 c615 05 clock_sync
Sep 9 23:14:07 ATT3-NAG1 ndo2db: Warning: Unable to write pid to lockfile (errno 28): No space left on device
Sep 9 23:34:07 ATT3-NAG1 ndo2db: Warning: Unable to write pid to lockfile (errno 28): No space left on device
[root@ATT3-NAG1 ~]# tail -20 /var/log/mysqld.log
InnoDB: buffer...
190910 5:04:40 InnoDB: Started; log sequence number 0 48336
190910 5:04:40 [ERROR] /usr/libexec/mysqld: Error writing file '/var/run/mysqlmysqld.pid' (Errcode: 28)
190910 5:04:40 [ERROR] Can't start server: can't create PID file: No space lefon device
190910 05:04:40 mysqld_safe Number of processes running now: 0
190910 05:04:40 mysqld_safe mysqld restarted
190910 5:04:41 InnoDB: Initializing buffer pool, size = 8.0M
190910 5:04:41 InnoDB: Completed initialization of buffer pool
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
190910 5:04:41 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
190910 5:04:41 InnoDB: Started; log sequence number 0 48336
190910 5:04:41 [ERROR] /usr/libexec/mysqld: Error writing file '/var/run/mysqlmysqld.pid' (Errcode: 28)
190910 5:04:41 [ERROR] Can't start server: can't create PID file: No space lefon device
190910 05:04:41 mysqld_safe Number of processes running now: 0
190910 05:04:41 mysqld_safe mysqld restarted

[mbellerue]

Looks like you're out of space. What does df say?

Code: Select all

df -h