Gateway timeout

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
User avatar
BanditBBS
Posts: 2454
Joined: Tue May 31, 2011 12:57 pm
Location: Scio, OH
Contact:

Gateway timeout

Post by BanditBBS »

Both my NLS hosts are coming up with Gateway Timeout errors. I restarted httpd and it has made no difference.
2 of XI5.6.14 Prod/DR/DEV - Nagios LogServer 2 Nodes
See my projects on the Exchange at BanditBBS - Also check out my Nagios stuff on my personal page at Bandit's Home and at github
User avatar
BanditBBS
Posts: 2454
Joined: Tue May 31, 2011 12:57 pm
Location: Scio, OH
Contact:

Re: Gateway timeout

Post by BanditBBS »

This is critical on both hosts:
Screenshot 2021-07-26 101000.jpg
You do not have the required permissions to view the files attached to this post.
2 of XI5.6.14 Prod/DR/DEV - Nagios LogServer 2 Nodes
See my projects on the Exchange at BanditBBS - Also check out my Nagios stuff on my personal page at Bandit's Home and at github
ssax
Dreams In Code
Posts: 7682
Joined: Wed Feb 11, 2015 12:54 pm

Re: Gateway timeout

Post by ssax »

Please PM me a copy of your profile.zip from each node, you can download it from Admin > System Status by clicking the Download System Profile button.

What OS/version are they running?

Code: Select all

uname -a
cat /etc/*release
User avatar
BanditBBS
Posts: 2454
Joined: Tue May 31, 2011 12:57 pm
Location: Scio, OH
Contact:

Re: Gateway timeout

Post by BanditBBS »

Wish I could. Since my post one started working periodically but not enough for me to get to the admin page. The other started loading the "waiting on elasticsearch" page but is stuck on that.

Both systems are identical:

Code: Select all

[root@cms-chi-nls02 ~]# uname -a
Linux cms-chi-nls02.itciss.com 4.18.0-240.el8.x86_64 #1 SMP Wed Sep 23 05:13:10 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux
[root@cms-chi-nls02 ~]# cat /etc/*release
NAME="Red Hat Enterprise Linux"
VERSION="8.3 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.3"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.3
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.3"
Red Hat Enterprise Linux release 8.3 (Ootpa)
Red Hat Enterprise Linux release 8.3 (Ootpa)
2 of XI5.6.14 Prod/DR/DEV - Nagios LogServer 2 Nodes
See my projects on the Exchange at BanditBBS - Also check out my Nagios stuff on my personal page at Bandit's Home and at github
User avatar
BanditBBS
Posts: 2454
Joined: Tue May 31, 2011 12:57 pm
Location: Scio, OH
Contact:

Re: Gateway timeout

Post by BanditBBS »

I've rebooted both nodes and now on the Cluster Status screen I get http 500 error.

EDIT: I updated the memory limit in php.ini and that fixed the 500 error.

EDIT2: The 500 error is back, the php.ini change only "fixed" it for a minute. I see no errors in the httpd error_log.
2 of XI5.6.14 Prod/DR/DEV - Nagios LogServer 2 Nodes
See my projects on the Exchange at BanditBBS - Also check out my Nagios stuff on my personal page at Bandit's Home and at github
ssax
Dreams In Code
Posts: 7682
Joined: Wed Feb 11, 2015 12:54 pm

Re: Gateway timeout

Post by ssax »

You should be able to get the profile through SSH/WinSCP like so:

Code: Select all

rm -rf /tmp/system-profile*
/usr/local/nagioslogserver/scripts/profile.sh
Then send me the resulting

Code: Select all

/tmp/system-profile.tar.gz
If that doesn't work, run this tail command as root and leave it running:

Code: Select all

tail -Fn0 /var/log/httpd/*
Then replicate the 500 error and send the output.

Are you seeing any errors in /var/log/messages or anything?
User avatar
BanditBBS
Posts: 2454
Joined: Tue May 31, 2011 12:57 pm
Location: Scio, OH
Contact:

Re: Gateway timeout

Post by BanditBBS »

I sent you the profile. I rebooted the boxes one more time and the 500 error is no longer there. I restarted httpd but the php memory limit didnt seem to fix it until a reboot.

That said, I am continuing to get the gateway timeout on the nodes and I PM'd the profile from one of them. I am not seeing any errors in any log files (messages, http error log, etc).
2 of XI5.6.14 Prod/DR/DEV - Nagios LogServer 2 Nodes
See my projects on the Exchange at BanditBBS - Also check out my Nagios stuff on my personal page at Bandit's Home and at github
ssax
Dreams In Code
Posts: 7682
Joined: Wed Feb 11, 2015 12:54 pm

Re: Gateway timeout

Post by ssax »

Edit this file:

Code: Select all

/etc/httpd/conf.d/php.conf
Change this specific part (around line 46):

Code: Select all

<IfModule !mod_php5.c>
  <IfModule !mod_php7.c>
    # Enable http authorization headers
    SetEnvIfNoCase ^Authorization$ "(.+)" HTTP_AUTHORIZATION=$1

    <FilesMatch \.(php|phar)$>
        SetHandler "proxy:unix:/run/php-fpm/www.sock|fcgi://localhost"
    </FilesMatch>
  </IfModule>
</IfModule>
Add this:

Code: Select all

    Timeout 600
    RequestReadTimeout header=600,minrate=50
    RequestReadTimeout body=600,minrate=50
    <Proxy "unix:/run/php-fpm/www.sock|fcgi://localhost">
        ProxySet timeout=600
    </Proxy>
So it looks like this:

Code: Select all

<IfModule !mod_php5.c>
  <IfModule !mod_php7.c>
    # Enable http authorization headers
    SetEnvIfNoCase ^Authorization$ "(.+)" HTTP_AUTHORIZATION=$1

    Timeout 600
    RequestReadTimeout header=600,minrate=50
    RequestReadTimeout body=600,minrate=50
    <Proxy "unix:/run/php-fpm/www.sock|fcgi://localhost">
        ProxySet timeout=600
    </Proxy>

    <FilesMatch \.(php|phar)$>
        SetHandler "proxy:unix:/run/php-fpm/www.sock|fcgi://localhost"
    </FilesMatch>
  </IfModule>
</IfModule>
Then edit this file:

Code: Select all

/etc/php-fpm.d/www.conf
Set:

Code: Select all

pm.max_requests = 50
Then restart apache/php-fpm:

Code: Select all

systemctl restart httpd php-fpm
Let me know if that resolves it.
User avatar
BanditBBS
Posts: 2454
Joined: Tue May 31, 2011 12:57 pm
Location: Scio, OH
Contact:

Re: Gateway timeout

Post by BanditBBS »

I don't get the gateway timeout issue anymore but any clicks take forever to load the webpage. The hosts are not under any sort of load.
2 of XI5.6.14 Prod/DR/DEV - Nagios LogServer 2 Nodes
See my projects on the Exchange at BanditBBS - Also check out my Nagios stuff on my personal page at Bandit's Home and at github
ssax
Dreams In Code
Posts: 7682
Joined: Wed Feb 11, 2015 12:54 pm

Re: Gateway timeout

Post by ssax »

Can you PM a fresh copy of your profile so I can look at the latest logs?

What is top showing?

Code: Select all

top -c -n3
Locked