Nagios XI Database Connect Problem

[EnvBroker1]

I'm getting the same problem as "by bsivavani » Tue Nov 17, 2015 12:56 pm" was getting.
So I tried to follow along and see if his fix would work for me.

I had previously ran /usr/local/nagiosxi/scripts/repair_databases.sh and it never worked, but I ran it again.
Came up with the same errors as usual...

MySQL Daemon failed to start.
Starting mysqld: [FAILED]
/usr/local/nagiosxi/scripts

===============
REPAIR COMPLETE
===============
Stopping ndo2db: head: cannot open `/usr/local/nagios/var/ndo2db.lock' for reading: No such file or directory
done.
Starting ndo2db: done.
Running configuration check...done.
Stopping nagios: .done.
Starting nagios: done.

Checked my filesystem sizes, but that didn't seem to be a problem.

# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_root-lv_root 31G 11G 19G 37% /
tmpfs 935M 0 935M 0% /dev/shm
/dev/sda1 477M 75M 377M 17% /boot
/dev/mapper/vg_root-lv_var 23G 3.1G 19G 15% /var
/dev/mapper/vg_root-lv_tmp 976M 1.3M 924M 1% /tmp
/dev/mapper/vg_root-lv_home 1.5G 5.4M 1.4G 1% /home
/dev/mapper/vg_root-lv_apps 5.8G 26M 5.5G 1% /apps

# df -ih
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/vg_root-lv_root 2.0M 317K 1.7M 16% /
tmpfs 234K 1 234K 1% /dev/shm
/dev/sda1 126K 52 125K 1% /boot
/dev/mapper/vg_root-lv_var 1.5M 413K 1.1M 28% /var
/dev/mapper/vg_root-lv_tmp 64K 25 64K 1% /tmp
/dev/mapper/vg_root-lv_home 96K 500 96K 1% /home
/dev/mapper/vg_root-lv_apps 384K 970 384K 1% /apps

ran the postgresql restart

# service postgresql restart
Stopping postgresql service: [ OK ]
Starting postgresql service: [ OK ]

greped the processes for post, but if you needed to see everything, I'll paste the whole thing here later.

# ps -ef | grep post
postgres 19719 1 0 12:10 ? 00:00:00 /usr/bin/postmaster -p 5432 -D /var/lib/pgsql/data
postgres 19763 19719 0 12:10 ? 00:00:00 postgres: logger process
postgres 19765 19719 0 12:10 ? 00:00:00 postgres: writer process
postgres 19766 19719 0 12:10 ? 00:00:00 postgres: wal writer process
postgres 19767 19719 0 12:10 ? 00:00:00 postgres: autovacuum launcher process
postgres 19768 19719 0 12:10 ? 00:00:00 postgres: stats collector process
postgres 19820 19719 0 12:11 ? 00:00:00 postgres: nagiosxi nagiosxi [local] idle
postgres 19918 19719 0 12:11 ? 00:00:00 postgres: nagiosxi nagiosxi [local] idle
postgres 20002 19719 0 12:11 ? 00:00:00 postgres: nagiosxi nagiosxi [local] idle
postgres 20113 19719 0 12:12 ? 00:00:00 postgres: nagiosxi nagiosxi [local] idle
postgres 20188 19719 0 12:12 ? 00:00:00 postgres: nagiosxi nagiosxi [local] idle
postgres 20263 19719 0 12:12 ? 00:00:00 postgres: nagiosxi nagiosxi [local] idle
postgres 20271 19719 0 12:12 ? 00:00:00 postgres: nagiosxi nagiosxi [local] idle
postgres 20389 19719 0 12:13 ? 00:00:00 postgres: nagiosxi nagiosxi [local] idle
postgres 23609 19719 0 12:25 ? 00:00:00 postgres: nagiosxi nagiosxi [local] idle
postgres 23611 19719 0 12:25 ? 00:00:00 postgres: nagiosxi nagiosxi [local] idle
postgres 23615 19719 0 12:25 ? 00:00:00 postgres: nagiosxi nagiosxi [local] idle
postgres 23616 19719 0 12:25 ? 00:00:00 postgres: nagiosxi nagiosxi [local] idle
postgres 23621 19719 0 12:25 ? 00:00:00 postgres: nagiosxi nagiosxi [local] idle
root 23836 9372 0 12:25 pts/0 00:00:00 grep post

Thanks
Eric

[lmiltchev]

Can you run the following commands and show the output?

Code: Select all

uname -a
cat /etc/*release
service mysqld start
service mysqld status
cat /etc/my.cnf
tail -20 /var/log/mysqld.log

[EnvBroker1]

We're still on an older version of XI. I'm trying to get the upgrade moved to a higher priority.

This server is my test server, so I can run the upgrade on it. Only other thing I could think of to mention is that our UNIX guys, cloned my main Nagios XI server and this is it. So some things may not have been properly changed. Also you'll notice near the end of the log, it does ask if mysqld server already running.

So here is what I got back when I put in the commands you requested.

# uname -a
Linux nagiostest 2.6.32-504.1.3.el6.x86_64 #1 SMP Tue Nov 11 17:57:25 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/*release
CentOS release 6.6 (Final)
LSB_VERSION=base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
CentOS release 6.6 (Final)
CentOS release 6.6 (Final)
# service mysqld start
MySQL Daemon failed to start.
Starting mysqld: [FAILED]
# service mysqld status
mysqld is stopped
# cat /etc/my.cnf
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
bind-address=10.131.86.58
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
# tail -20 /var/log/mysqld.log
151119 11:39:49 [ERROR] Aborting

151119 11:39:49 InnoDB: Starting shutdown...
151119 11:39:54 InnoDB: Shutdown completed; log sequence number 0 44293
151119 11:39:54 [Note] /usr/libexec/mysqld: Shutdown complete

151119 11:39:54 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
151119 13:54:48 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
151119 13:54:49 InnoDB: Initializing buffer pool, size = 8.0M
151119 13:54:49 InnoDB: Completed initialization of buffer pool
151119 13:54:49 InnoDB: Started; log sequence number 0 44293
151119 13:54:49 [ERROR] Can't start server: Bind on TCP/IP port: Cannot assign requested address
151119 13:54:49 [ERROR] Do you already have another mysqld server running on port: 3306 ?
151119 13:54:49 [ERROR] Aborting

151119 13:54:49 InnoDB: Starting shutdown...
151119 13:54:54 InnoDB: Shutdown completed; log sequence number 0 44293
151119 13:54:54 [Note] /usr/libexec/mysqld: Shutdown complete

151119 13:54:54 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

[rkennedy]

Please run the following and post the output -

Code: Select all

netstat -tulpn|grep 3306

[EnvBroker1]

No output at all. No port 3306

# netstat -tulpn | grep 3306
#

[lmiltchev]

This server is my test server, so I can run the upgrade on it. Only other thing I could think of to mention is that our UNIX guys, cloned my main Nagios XI server and this is it. So some things may not have been properly changed.

It is possible. We don't have all of the info to be sure. Talking about your "main", production XI server - was the mysql database offloaded to a remote server? Is the "test" sever actually your "offloaded" mysql server? I can see that in my.cnf you have:

Code: Select all

bind-address=10.131.86.58

Is this the IP address of your "main" XI or the clone?

Can you run the following commands on both servers and show the output:

Code: Select all

ip addr
grep "db_host=" /usr/local/nagios/etc/ndo2db.cfg

[EnvBroker1]

.58 is my main Nagios server. They cloned that to make my test server which is .60. So how do I get that .58 changed to a .60, seems like that's the problem.

I'll run the commands and post those in a minute.

Thanks

[EnvBroker1]

From my main Nagios XI server - .58
# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:50:56:a0:6f:4b brd ff:ff:ff:ff:ff:ff
inet 10.131.86.58/24 brd 10.131.86.255 scope global eth0
inet6 fe80::250:56ff:fea0:6f4b/64 scope link
valid_lft forever preferred_lft forever
# grep "db_host=" /usr/local/nagios/etc/ndo2db.cfg
db_host=localhost
#

From my test server - .60
# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:50:56:a0:79:38 brd ff:ff:ff:ff:ff:ff
inet 10.131.86.60/24 brd 10.131.86.255 scope global eth0
inet6 fe80::250:56ff:fea0:7938/64 scope link
valid_lft forever preferred_lft forever
# grep "db_host=" /usr/local/nagios/etc/ndo2db.cfg
db_host=localhost
#

[EnvBroker1]

I went into the my.cnf file and changed the bind-address from .58 to .60 and it's working now.

Thanks for all the help,
I appreciate it.

[lmiltchev]

OK, so the mysql db is not offloaded to a remote server - on both boxes you have:

Code: Select all

db_host=localhost

Comment out the line in the my.cnf file, so that it would look like this:

Code: Select all

# bind-address=10.131.86.58

then try to start mysql:

Code: Select all

service mysqld start

Let me know if this helped.