Cannot access web UI

[rkymtnhigh]

When I got to work this morning, our remote Nagios instance seemed to be having problems.

The web UI gives me "This site can't be reached - took too long to respond" from both the public internet and from another server on it's LAN.

I SSH'd into the CentOS 6 server and made sure iptables and ip6tables were disabled.
I was not able to telnet to the nagios server from another internal address or over the public internet on 443.
I've restarted the nagios service as well as the httpd service with no change in behavior. cd /var

In the httpd error_log file I have this entry over and over. Doesn't seem to be related or triggered when attempting to access the webpage.

[Tue Nov 15 11:30:23.500932 2016] [:error] [pid 1899] [client ::1:34387] PHP Notice: Undefined variable: f2 in /usr/local/nagiosxi/html/includes/components/snmptrapsender/snmptrapsender.inc.php on line 485
[Tue Nov 15 11:30:43.858855 2016] [:error] [pid 1598] [client ::1:34392] PHP Notice: Undefined variable: f2 in /usr/local/nagiosxi/html/includes/components/snmptrapsender/snmptrapsender.inc.php on line 485
[Tue Nov 15 11:31:03.136705 2016] [:error] [pid 1594] [client ::1:34396] PHP Notice: Undefined variable: f2 in /usr/local/nagiosxi/html/includes/components/snmptrapsender/snmptrapsender.inc.php on line 485
[Tue Nov 15 11:31:23.496846 2016] [:error] [pid 1595] [client ::1:34402] PHP Notice: Undefined variable: f2 in /usr/local/nagiosxi/html/includes/components/snmptrapsender/snmptrapsender.inc.php on line 485

At this point I'm starting to suspect iptables is not actually disabled, as I can't see any other reason I wouldn't be able to connect.

Any input or advice is greatly appreciated.

Thank you,

RMH

[avandemore]

What is the output of iptables -L and netstat -plnt?

[rkymtnhigh]

Sure, here is iptables -L

Code: Select all

Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere
INPUT_direct  all  --  anywhere             anywhere
INPUT_ZONES_SOURCE  all  --  anywhere             anywhere
INPUT_ZONES  all  --  anywhere             anywhere
ACCEPT     icmp --  anywhere             anywhere
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere
FORWARD_direct  all  --  anywhere             anywhere
FORWARD_IN_ZONES_SOURCE  all  --  anywhere             anywhere
FORWARD_IN_ZONES  all  --  anywhere             anywhere
FORWARD_OUT_ZONES_SOURCE  all  --  anywhere             anywhere
FORWARD_OUT_ZONES  all  --  anywhere             anywhere
ACCEPT     icmp --  anywhere             anywhere
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
OUTPUT_direct  all  --  anywhere             anywhere

Chain FORWARD_IN_ZONES (1 references)
target     prot opt source               destination
FWDI_public  all  --  anywhere             anywhere            [goto]
FWDI_public  all  --  anywhere             anywhere            [goto]

Chain FORWARD_IN_ZONES_SOURCE (1 references)
target     prot opt source               destination

Chain FORWARD_OUT_ZONES (1 references)
target     prot opt source               destination
FWDO_public  all  --  anywhere             anywhere            [goto]
FWDO_public  all  --  anywhere             anywhere            [goto]

Chain FORWARD_OUT_ZONES_SOURCE (1 references)
target     prot opt source               destination

Chain FORWARD_direct (1 references)
target     prot opt source               destination

Chain FWDI_public (2 references)
target     prot opt source               destination
FWDI_public_log  all  --  anywhere             anywhere
FWDI_public_deny  all  --  anywhere             anywhere
FWDI_public_allow  all  --  anywhere             anywhere

Chain FWDI_public_allow (1 references)
target     prot opt source               destination

Chain FWDI_public_deny (1 references)
target     prot opt source               destination

Chain FWDI_public_log (1 references)
target     prot opt source               destination

Chain FWDO_public (2 references)
target     prot opt source               destination
FWDO_public_log  all  --  anywhere             anywhere
FWDO_public_deny  all  --  anywhere             anywhere
FWDO_public_allow  all  --  anywhere             anywhere

Chain FWDO_public_allow (1 references)
target     prot opt source               destination

Chain FWDO_public_deny (1 references)
target     prot opt source               destination

Chain FWDO_public_log (1 references)
target     prot opt source               destination

Chain INPUT_ZONES (1 references)
target     prot opt source               destination
IN_public  all  --  anywhere             anywhere            [goto]
IN_public  all  --  anywhere             anywhere            [goto]

Chain INPUT_ZONES_SOURCE (1 references)
target     prot opt source               destination

Chain INPUT_direct (1 references)
target     prot opt source               destination

Chain IN_public (2 references)
target     prot opt source               destination
IN_public_log  all  --  anywhere             anywhere
IN_public_deny  all  --  anywhere             anywhere
IN_public_allow  all  --  anywhere             anywhere

Chain IN_public_allow (1 references)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:nrpe ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:nsca ctstate NEW

Chain IN_public_deny (1 references)
target     prot opt source               destination

Chain IN_public_log (1 references)
target     prot opt source               destination

Chain OUTPUT_direct (1 references)
target     prot opt source               destination

And netstat -plnt

Code: Select all

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:3306            0.0.0.0:*               LISTEN      1543/mysqld
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      504/rpcbind
tcp        0      0 127.0.0.1:8022          0.0.0.0:*               LISTEN      1296/python
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1148/sshd
tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN      1221/postgres
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1510/master
tcp6       0      0 :::111                  :::*                    LISTEN      504/rpcbind
tcp6       0      0 :::80                   :::*                    LISTEN      1177/httpd
tcp6       0      0 :::22                   :::*                    LISTEN      1148/sshd
tcp6       0      0 ::1:5432                :::*                    LISTEN      1221/postgres
tcp6       0      0 ::1:25                  :::*                    LISTEN      1510/master
tcp6       0      0 :::443                  :::*                    LISTEN      1177/httpd
tcp6       0      0 :::5666                 :::*                    LISTEN      1153/xinetd
tcp6       0      0 :::5667                 :::*                    LISTEN      1153/xinetd

Thank you very much.

[avandemore]

iptables is active.

To stop it, /etc/init.d/iptables stop

To disable it from starting at boot, chkconfig iptables off

Please retry the test with iptables stopped.

[rkymtnhigh]

When I run /etc/init.d/iptables stop, it gives me

Code: Select all

-bash: /etc/init.d/iptables: No such file or directory

When I run service iptables stop, it gives me

Code: Select all

Redirecting to /bin/systemctl stop  iptables.service

When I run the chkconfig iptables off command it gives me

Code: Select all

Note: Forwarding request to 'systemctl disable iptables.service'.

There is no change in behavior after a reboot.

Thank you!

[avandemore]

What is the output of cat /etc/redhat-release, ls /etc/init.d/ and systemctl?

[rkymtnhigh]

Code: Select all

CentOS Linux release 7.1.1503 (Core)

ls /etc/init.d/

Code: Select all

ajaxterm  functions  nagios  nagiosxi  ndo2db  netconsole  network  npcd  pdagent  README

systemctl

Code: Select all

UNIT                                             LOAD   ACTIVE SUB       DESCRIPTION
proc-sys-fs-binfmt_misc.automount                loaded active waiting   Arbitrary Executable File Formats File System Automount Point
sys-devices-platform-serial8250-tty-ttyS1.device loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS1
sys-devices-platform-serial8250-tty-ttyS2.device loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS2
sys-devices-platform-serial8250-tty-ttyS3.device loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS3
sys-devices-pnp0-00:09-tty-ttyS0.device          loaded active plugged   /sys/devices/pnp0/00:09/tty/ttyS0
sys-devices-vbd\x2d768-block-xvda-xvda1.device   loaded active plugged   /sys/devices/vbd-768/block/xvda/xvda1
sys-devices-vbd\x2d768-block-xvda.device         loaded active plugged   /sys/devices/vbd-768/block/xvda
sys-devices-vif\x2d0-net-eth0.device             loaded active plugged   /sys/devices/vif-0/net/eth0
sys-module-configfs.device                       loaded active plugged   /sys/module/configfs
sys-subsystem-net-devices-eth0.device            loaded active plugged   /sys/subsystem/net/devices/eth0
-.mount                                          loaded active mounted   /
dev-hugepages.mount                              loaded active mounted   Huge Pages File System
dev-mqueue.mount                                 loaded active mounted   POSIX Message Queue File System
proc-fs-nfsd.mount                               loaded active mounted   NFSD configuration filesystem
sys-kernel-config.mount                          loaded active mounted   Configuration File System
sys-kernel-debug.mount                           loaded active mounted   Debug File System
var-lib-nfs-rpc_pipefs.mount                     loaded active mounted   RPC Pipe File System
brandbot.path                                    loaded active waiting   Flexible branding
systemd-ask-password-console.path                loaded active waiting   Dispatch Password Requests to Console Directory Watch
systemd-ask-password-wall.path                   loaded active waiting   Forward Password Requests to Wall Directory Watch
session-1.scope                                  loaded active running   Session 1 of user centos
session-621.scope                                loaded active running   Session 621 of user nagios
session-622.scope                                loaded active running   Session 622 of user nagios
session-625.scope                                loaded active running   Session 625 of user nagios
session-626.scope                                loaded active running   Session 626 of user nagios
session-627.scope                                loaded active running   Session 627 of user nagios
session-c1.scope                                 loaded active abandoned Session c1 of user pdagent
ajaxterm.service                                 loaded active running   SYSV: Ajaxterm is a web-based terminal
auditd.service                                   loaded active running   Security Auditing Service
chronyd.service                                  loaded active running   NTP client/server
cloud-config.service                             loaded active exited    Apply the settings specified in cloud-config
cloud-final.service                              loaded active exited    Execute cloud user/final scripts
cloud-init-local.service                         loaded active exited    Initial cloud-init job (pre-networking)
cloud-init.service                               loaded active exited    Initial cloud-init job (metadata service crawler)
crond.service                                    loaded active running   Command Scheduler
dbus.service                                     loaded active running   D-Bus System Message Bus
firewalld.service                                loaded active running   firewalld - dynamic firewall daemon
[email protected]                               loaded active running   Getty on tty1
gssproxy.service                                 loaded active running   GSSAPI Proxy Daemon
httpd.service                                    loaded active running   The Apache HTTP Server
kmod-static-nodes.service                        loaded active exited    Create list of required static device nodes for the current kernel
mariadb.service                                  loaded active running   MariaDB database server
nagios.service                                   loaded active running   LSB: Starts and stops the Nagios monitoring server
nagiosxi.service                                 loaded active exited    SYSV: NagiosXI subtasks
ndo2db.service                                   loaded active running   LSB: Nagios NDO2DB Initscript
network.service                                  loaded active running   LSB: Bring up/down networking
nfs-config.service                               loaded active exited    Preprocess NFS configuration
npcd.service                                     loaded active running   SYSV: Visit the Website at http://sourceforge.net/projects/pnp4nagios/
pdagent.service                                  loaded active exited    LSB: PagerDuty Agent
polkit.service                                   loaded active running   Authorization Manager
postfix.service                                  loaded active running   Postfix Mail Transport Agent
postgresql.service                               loaded active running   PostgreSQL database server
rhel-dmesg.service                               loaded active exited    Dump dmesg to /var/log/dmesg
rhel-import-state.service                        loaded active exited    Import network configuration from initramfs
rhel-readonly.service                            loaded active exited    Configure read-only root support
rpcbind.service                                  loaded active running   RPC bind service
rsyslog.service                                  loaded active running   System Logging Service
[email protected]                       loaded active running   Serial Getty on ttyS0
sshd.service                                     loaded active running   OpenSSH server daemon
sysstat.service                                  loaded active exited    Resets System Activity Logs
systemd-fsck-root.service                        loaded active exited    File System Check on Root Device
systemd-journald.service                         loaded active running   Journal Service
systemd-logind.service                           loaded active running   Login Service
systemd-random-seed.service                      loaded active exited    Load/Save Random Seed
systemd-remount-fs.service                       loaded active exited    Remount Root and Kernel File Systems
systemd-sysctl.service                           loaded active exited    Apply Kernel Variables
systemd-tmpfiles-setup-dev.service               loaded active exited    Create static device nodes in /dev
systemd-tmpfiles-setup.service                   loaded active exited    Create Volatile Files and Directories
systemd-udev-trigger.service                     loaded active exited    udev Coldplug all Devices
systemd-udevd.service                            loaded active running   udev Kernel Device Manager
systemd-update-utmp.service                      loaded active exited    Update UTMP about System Reboot/Shutdown
systemd-user-sessions.service                    loaded active exited    Permit User Sessions
systemd-vconsole-setup.service                   loaded active exited    Setup Virtual Console
tuned.service                                    loaded active running   Dynamic System Tuning Daemon
xinetd.service                                   loaded active running   Xinetd A Powerful Replacement For Inetd
-.slice                                          loaded active active    Root Slice
system-getty.slice                               loaded active active    system-getty.slice
system-serial\x2dgetty.slice                     loaded active active    system-serial\x2dgetty.slice
system.slice                                     loaded active active    System Slice
user-0.slice                                     loaded active active    user-0.slice
user-1000.slice                                  loaded active active    user-1000.slice
user-1001.slice                                  loaded active active    user-1001.slice
user-995.slice                                   loaded active active    user-995.slice
user.slice                                       loaded active active    User and Session Slice
dbus.socket                                      loaded active running   D-Bus System Message Bus Socket
rpcbind.socket                                   loaded active running   RPCbind Server Activation Socket
systemd-initctl.socket                           loaded active listening /dev/initctl Compatibility Named Pipe
systemd-journald.socket                          loaded active running   Journal Socket
systemd-shutdownd.socket                         loaded active listening Delayed Shutdown Socket
systemd-udevd-control.socket                     loaded active running   udev Control Socket
systemd-udevd-kernel.socket                      loaded active running   udev Kernel Socket
basic.target                                     loaded active active    Basic System
cloud-config.target                              loaded active active    Cloud-config availability
cryptsetup.target                                loaded active active    Encrypted Volumes
getty.target                                     loaded active active    Login Prompts
local-fs-pre.target                              loaded active active    Local File Systems (Pre)
local-fs.target                                  loaded active active    Local File Systems
multi-user.target                                loaded active active    Multi-User System
network-online.target                            loaded active active    Network is Online
network.target                                   loaded active active    Network
nfs-client.target                                loaded active active    NFS client services
paths.target                                     loaded active active    Paths
remote-fs-pre.target                             loaded active active    Remote File Systems (Pre)
remote-fs.target                                 loaded active active    Remote File Systems
slices.target                                    loaded active active    Slices
sockets.target                                   loaded active active    Sockets
swap.target                                      loaded active active    Swap
sysinit.target                                   loaded active active    System Initialization
timers.target                                    loaded active active    Timers
systemd-tmpfiles-clean.timer                     loaded active waiting   Daily Cleanup of Temporary Directories

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

110 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.

Thank you!

[avandemore]

So you are running CentOS 7, not 6.

To disable the service do this:

systemctl disable firewalld.service

and reboot, then test again.

[rkymtnhigh]

Oh my apologies! That command did the trick, thank you for all the help!

[avandemore]

No worries, are we ok to lock this thread?