Starting httpd ... could not bind to address [::]:80

[awilson]

Hi. I attempted to stop and restart services on a test system that hasn't seen activity in a while.
When attempting to restart httpd, we get the following:
Note: When submitting and looking at the listing, I noticed that httpd does have port 80.
We still get this:

Code: Select all

[root@testserver~]# service httpd restart
Stopping httpd:                                            [  OK  ]
Starting httpd: (98)Address already in use: make_sock: could not bind to address [::]:80
(98)Address already in use: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
Unable to open logs
                                                           [FAILED]
[root@testserver~]#

~~~~~~~~~~

Code: Select all

[root@testserver~]# service httpd start
Starting httpd: (98)Address already in use: make_sock: could not bind to address [::]:80
(98)Address already in use: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
Unable to open logs          [FAILED]

Port 80 doesn't seem to be used by another process:

Code: Select all

[root@testserver~]# netstat -tunlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name
tcp        0      0 0.0.0.0:7002                0.0.0.0:*                   LISTEN      1438/python
tcp        0      0 127.0.0.1:42300             0.0.0.0:*                   LISTEN      1560/tcl
tcp        0      0 127.0.0.1:32000             0.0.0.0:*                   LISTEN      2153/java
tcp        0      0 0.0.0.0:514                 0.0.0.0:*                   LISTEN      1326/rsyslogd
tcp        0      0 0.0.0.0:3306                0.0.0.0:*                   LISTEN      9841/mysqld
tcp        0      0 0.0.0.0:111                 0.0.0.0:*                   LISTEN      1380/rpcbind
tcp        0      0 0.0.0.0:8080                0.0.0.0:*                   LISTEN      1531/lighttpd
tcp        0      0 0.0.0.0:2003                0.0.0.0:*                   LISTEN      1438/python
tcp        0      0 0.0.0.0:2004                0.0.0.0:*                   LISTEN      1438/python
tcp        0      0 127.0.0.1:8022              0.0.0.0:*                   LISTEN      2260/python
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      1767/sshd
tcp        0      0 0.0.0.0:5432                0.0.0.0:*                   LISTEN      9936/postmaster
tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      2023/master
tcp        0      0 :::443                      :::*                        LISTEN      9527/httpd
tcp        0      0 :::5666                     :::*                        LISTEN      1778/xinetd
tcp        0      0 :::514                      :::*                        LISTEN      1326/rsyslogd
tcp        0      0 :::5667                     :::*                        LISTEN      1778/xinetd
tcp        0      0 :::111                      :::*                        LISTEN      1380/rpcbind
tcp        0      0 :::80                       :::*                        LISTEN      9527/httpd
tcp        0      0 :::22                       :::*                        LISTEN      1767/sshd
tcp        0      0 :::23                       :::*                        LISTEN      1778/xinetd
tcp        0      0 ::1:25                      :::*                        LISTEN      2023/master
udp        0      0 0.0.0.0:707                 0.0.0.0:*                               1380/rpcbind
udp        0      0 0.0.0.0:111                 0.0.0.0:*                               1380/rpcbind
udp     4320      0 0.0.0.0:631                 0.0.0.0:*                               1316/portreserve
udp        0      0 10.10.83.94:123             0.0.0.0:*                               13626/ntpd
udp        0      0 127.0.0.1:123               0.0.0.0:*                               13626/ntpd
udp        0      0 0.0.0.0:123                 0.0.0.0:*                               13626/ntpd
udp        0      0 0.0.0.0:514                 0.0.0.0:*                               1326/rsyslogd
udp        0      0 0.0.0.0:514                 0.0.0.0:*                               1326/rsyslogd
udp        0      0 :::707                      :::*                                    1380/rpcbind
udp        0      0 :::111                      :::*                                    1380/rpcbind
udp        0      0 fe80::250:56ff:feb8:788f:123 :::*                                    13626/ntpd
udp        0      0 ::1:123                     :::*                                    13626/ntpd
udp        0      0 :::123                      :::*                                    13626/ntpd
udp        0      0 :::514                      :::*                                    1326/rsyslogd
udp        0      0 :::514                      :::*                                    1326/rsyslogd

I see that 8080 is taken. Could that be the cause with an incorrect error message?

Thanks!

[mbellerue]

It looks like port 80 on IPv6 is being used by httpd. Is this a RHEL/CentOS 6 or 7 machine? You might try to stop httpd, and then see if :::80 is still in use.

netstat -tunlp | grep ":::80"

If it is, kill off the PID. Verify that it isn't bound to port 80, and then try to start httpd again.

[awilson]

The server is running RHEL 6.10. It seems that something is rapidly respawning an httpd process.

Code: Select all

[root@testserver ~]# netstat -tunlp | grep ":::80"
tcp        0      0 :::80                       :::*                        LISTEN      9527/httpd
[root@testserver ~]# kill -9 9527
[root@testserver  ~]# kill -9 9527
-bash: kill: (9527) - No such process
[root@testserver  ~]# netstat -tunlp | grep ":::80"
tcp        0      0 :::80                       :::*                        LISTEN      9528/httpd
[root@testserver ~]# uname -a
Linux testserver 2.6.32-754.2.1.el6.x86_64 #1 SMP Tue Jul 3 16:37:52 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@testserver ~]#
[root@testserver  ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.10 (Santiago)

[mbellerue]

That's strange. Let's do a service nagios stop, followed by a service httpd stop, and see if that shuts down :::80.

Let's also take a look at /var/log/messages, and /var/log/httpd/error_log to see if there's something about what application is trying to start httpd.

[awilson]

Out of curiosity, I checked and Nagios XI is running. I can use the web GUI and it appears to be running service checks.

[mbellerue]

Is Nagios configured to run on 443? Or maybe another custom port?

[awilson]

I added the last update before seeing your reply. This is the result of the last suggestion.

Code: Select all

[root@testserver ~]# service nagios stop
Stopping nagios:..........................................................................................
Warning - nagios did not exit in a timely manner
[root@testserver  ~]# service nagios stop
Stopping nagios:No lock file found in /usr/local/nagios/var/nagios.lock
[root@testserver ~]# service httpd stop
Stopping httpd:                                            [  OK  ]
[root@testserver ~]#
[root@testserver ~]# grep httpd /var/log/messages
[root@testserver ~]# ls -l /var/log/httpd/error_log
-rw-r--r-- 1 root root 0 Nov 10 03:21 /var/log/httpd/error_log
[root@testserver  ~]# cat /var/log/httpd/error_log
[root@testserver ~]#

.

[awilson]

I forgot that we started testing an https configuration. We didn't completely roll it back. There was still an ../httpd/conf.d/ssl.conf file. I renamed it.

Code: Select all

[root@lusrcdnagmn01 conf.d]# service nagios stop
Stopping nagios:No lock file found in /usr/local/nagios/var/nagios.lock
[root@lusrcdnagmn01 conf.d]# service httpd stop
Stopping httpd:                                            [  OK  ]
[root@lusrcdnagmn01 conf.d]# netstat -tunlp | grep httpd
tcp        0      0 0.0.0.0:8080                0.0.0.0:*                   LISTEN      1531/lighttpd
tcp        0      0 :::443                      :::*                        LISTEN      3631/httpd
tcp        0      0 :::80                       :::*                        LISTEN      3631/httpd
[root@lusrcdnagmn01 conf.d]# pkill -9 httpd
[root@lusrcdnagmn01 conf.d]# pkill -9 httpd
[root@lusrcdnagmn01 conf.d]# netstat -tunlp | grep httpd
[root@lusrcdnagmn01 conf.d]#

Looks like it's finally dead.

[awilson]

It looks clean now.

Code: Select all

[root@testserver ~]# service nagios stop
Stopping nagios:No lock file found in /usr/local/nagios/var/nagios.lock
[root@testserver ~]# service httpd stop
Stopping httpd:                                            [  OK  ]
[root@testserver ~]#
[root@testserver ~]# service httpd stop
Stopping httpd:                                            [  OK  ]
[root@testserver ~]# service crond stop
Stopping crond:                                            [  OK  ]
[root@testserver ~]# service npcd stop
NPCD Stopped.
[root@testserver ~]# service nagios stop
Stopping nagios:No lock file found in /usr/local/nagios/var/nagios.lock
[root@testserver ~]# service ndo2db stop
Stopping ndo2db: done.
[root@testserver ~]# service postgresql stop
Stopping postgresql service:                               [  OK  ]
[root@testserver ~]# pkill -9 -u apache
[root@testserver ~]# pkill -9 -u postgres
[root@testserver ~]# pkill -9 -u postgres
[root@testserver ~]# pkill -9 -u postgres
[root@testserver ~]# pkill -9 -u apache
[root@testserver ~]# ps -ef | grep nagios
root      1433  1767  0 20:01 ?        00:00:00 sshd: nagios [priv]
nagios    1714  1433  0 20:02 ?        00:00:00 sshd: nagios@pts/0
nagios    1715  1714  0 20:02 pts/0    00:00:00 -bash
nagios   10017     1  0 22:00 ?        00:00:00 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios   10538 10017  0 22:01 ?        00:00:00 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   10539 10017  0 22:01 ?        00:00:00 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   10540 10017  0 22:01 ?        00:00:00 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   10541 10017  0 22:01 ?        00:00:00 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   16256 10017  0 22:17 ?        00:00:00 [nagios] <defunct>
root     16403 31831  0 23:42 pts/2    00:00:00 grep nagios
root     22329  1767  0 20:57 ?        00:00:00 sshd: nagios [priv]
nagios   22726 22329  0 20:58 ?        00:00:00 sshd: nagios@pts/1
nagios   22727 22726  0 20:58 pts/1    00:00:00 -bash
root     30657  1767  0 22:55 ?        00:00:00 sshd: nagios [priv]
nagios   31254 30657  0 22:57 ?        00:00:00 sshd: nagios@pts/2
nagios   31255 31254  0 22:57 pts/2    00:00:00 -bash
[root@testserver ~]# pkill -9 -u nagios
login as: nagios
This server is for Fossil internal use ONLY.
If you are not an authorized employee, please disconnect NOW!
[email protected]'s password:
//snip
[nagios@testserver ~]$ sudo su -
[root@testserver ~]# for i in $(ipcs -q | grep nagios |awk '{print $2}'); do ipcrm -q $i; done
rm -f /usr/local/nagiosxi/var/event_handler.lock
rm -f /usr/local/nagiosxi/scripts/reconfigure_nagios.lock
rm -f /usr/local/nagios/var/ndo2db.lock
rm -f /usr/local/nagios/var/ndo2db.pid
rm -f /usr/local/nagios/var/ndo2db.sock
rm -f /usr/local/nagios/var/ndo.sock
rm -f /us/local/nagiosxi/var/subsys/ndo2db
rm -f /var/run/nagios/nagios.lock
[root@testserver ~]# rm -f /usr/local/nagiosxi/var/dbmaint.lock
[root@testserver ~]# rm -f /usr/local/nagiosxi/var/event_handler.lock
[root@testserver ~]# rm -f /usr/local/nagiosxi/scripts/reconfigure_nagios.lock
[root@testserver ~]# rm -f /usr/local/nagios/var/ndo2db.lock
[root@testserver ~]# rm -f /usr/local/nagios/var/ndo2db.pid
[root@testserver ~]# rm -f /usr/local/nagios/var/ndo2db.sock
[root@testserver ~]# rm -f /usr/local/nagios/var/ndo.sock
[root@testserver ~]# rm -f /us/local/nagiosxi/var/subsys/ndo2db
[root@testserver ~]# rm -f /var/run/nagios/nagios.lock
[root@testserver ~]# rm -f /var/run/nagios.lock
[root@testserver ~]# rm -f /usr/local/nagios/var/nagios.lock
[root@testserver ~]# rm -f /var/run/httpd/httpd.pid
[root@testserver ~]# service mysqld restart
Stopping mysqld:                                           [  OK  ]
Starting mysqld:                                           [  OK  ]
[root@testserver ~]# service postgresql start
Starting postgresql service:                               [  OK  ]
[root@testserver ~]# service ndo2db start
Starting ndo2db: done.
[root@testserver ~]# service nagios start
Starting nagios: done.
[root@testserver ~]# service npcd start
NPCD started.
[root@testserver ~]# service crond start
Starting crond:                                            [  OK  ]
[root@testserver ~]# service httpd start
Starting httpd:                                            [  OK  ]
[root@testserver ~]# service snmptt restart
snmptt: unrecognized service
[root@testserver ~]#

[mbellerue]

Excellent! Glad you were able to locate that. Is everything working as expected now?