Gateway timeout

[BanditBBS]

I sent the updated profile.

Code: Select all

top - 13:19:41 up 3 days,  1:56,  1 user,  load average: 1.22, 1.41, 1.49
Tasks: 276 total,   1 running, 275 sleeping,   0 stopped,   0 zombie
%Cpu(s): 12.4 us,  0.1 sy,  0.0 ni, 87.1 id,  0.0 wa,  0.3 hi,  0.1 si,  0.1 st
MiB Mem :   7768.0 total,    156.0 free,   5468.9 used,   2143.1 buff/cache
MiB Swap:   8076.0 total,   7527.7 free,    548.2 used.   1741.7 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   1448 nagios    20   0   39.0g   4.4g  18780 S  99.3  57.4   5986:43 /bin/java -Xms3884m -Xmx3884m -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupanc+
 834915 root      20   0   65692   5228   4156 R   1.0   0.1   0:00.09 top -c -n3
   1415 root      39  19 6404628 394364  17676 S   0.7   5.0 145:47.92 /bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+Us+
     10 root      20   0       0      0      0 I   0.3   0.0   2:46.05 [rcu_sched]
   1050 rngd      20   0  381340   6160   5476 S   0.3   0.1   4:33.83 /sbin/rngd -f --fill-watermark=0
 833728 root      20   0       0      0      0 I   0.3   0.0   0:00.58 [kworker/5:1-events]
      1 root      20   0  252248  12040   8236 S   0.0   0.2  14:38.45 /usr/lib/systemd/systemd --switched-root --system --deserialize 18
      2 root      20   0       0      0      0 S   0.0   0.0   0:00.52 [kthreadd]
      3 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 [rcu_gp]
      4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 [rcu_par_gp]
      6 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 [kworker/0:0H-events_highpri]
      8 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 [mm_percpu_wq]
      9 root      20   0       0      0      0 S   0.0   0.0   0:04.81 [ksoftirqd/0]
     11 root      rt   0       0      0      0 S   0.0   0.0   0:00.36 [migration/0]
     12 root      rt   0       0      0      0 S   0.0   0.0   0:00.28 [watchdog/0]
     13 root      20   0       0      0      0 S   0.0   0.0   0:00.00 [cpuhp/0]
     14 root      20   0       0      0      0 S   0.0   0.0   0:00.00 [cpuhp/1]
     15 root      rt   0       0      0      0 S   0.0   0.0   0:00.56 [watchdog/1]
     16 root      rt   0       0      0      0 S   0.0   0.0   0:00.38 [migration/1]
     17 root      20   0       0      0      0 S   0.0   0.0   0:03.52 [ksoftirqd/1]
     19 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 [kworker/1:0H-events_highpri]
     20 root      20   0       0      0      0 S   0.0   0.0   0:00.00 [cpuhp/2]
     21 root      rt   0       0      0      0 S   0.0   0.0   0:00.58 [watchdog/2]
     22 root      rt   0       0      0      0 S   0.0   0.0   0:00.38 [migration/2]
     23 root      20   0       0      0      0 S   0.0   0.0   0:03.84 [ksoftirqd/2]
     25 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 [kworker/2:0H-events_highpri]
     26 root      20   0       0      0      0 S   0.0   0.0   0:00.00 [cpuhp/3]
     27 root      rt   0       0      0      0 S   0.0   0.0   0:00.59 [watchdog/3]
     28 root      rt   0       0      0      0 S   0.0   0.0   0:00.36 [migration/3]
     29 root      20   0       0      0      0 S   0.0   0.0   0:03.64 [ksoftirqd/3]
     31 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 [kworker/3:0H-events_highpri]
     32 root      20   0       0      0      0 S   0.0   0.0   0:00.00 [cpuhp/4]
     33 root      rt   0       0      0      0 S   0.0   0.0   0:00.64 [watchdog/4]
     34 root      rt   0       0      0      0 S   0.0   0.0   0:00.35 [migration/4]
     35 root      20   0       0      0      0 S   0.0   0.0   0:04.00 [ksoftirqd/4]
     37 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 [kworker/4:0H-events_highpri]
     38 root      20   0       0      0      0 S   0.0   0.0   0:00.00 [cpuhp/5]
     39 root      rt   0       0      0      0 S   0.0   0.0   0:00.82 [watchdog/5]
     40 root      rt   0       0      0      0 S   0.0   0.0   0:00.36 [migration/5]
     41 root      20   0       0      0      0 S   0.0   0.0   0:09.16 [ksoftirqd/5]
     43 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 [kworker/5:0H-events_highpri]
     44 root      20   0       0      0      0 S   0.0   0.0   0:00.00 [cpuhp/6]
     45 root      rt   0       0      0      0 S   0.0   0.0   0:00.67 [watchdog/6]
     46 root      rt   0       0      0      0 S   0.0   0.0   0:00.37 [migration/6]
     47 root      20   0       0      0      0 S   0.0   0.0   0:04.47 [ksoftirqd/6]
     49 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 [kworker/6:0H-events_highpri]
     50 root      20   0       0      0      0 S   0.0   0.0   0:00.00 [cpuhp/7]
     51 root      rt   0       0      0      0 S   0.0   0.0   0:00.82 [watchdog/7]
     52 root      rt   0       0      0      0 S   0.0   0.0   0:00.42 [migration/7]
     53 root      20   0       0      0      0 S   0.0   0.0   0:05.93 [ksoftirqd/7]
     55 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 [kworker/7:0H-events_highpri]
     64 root      20   0       0      0      0 S   0.0   0.0   0:00.00 [kdevtmpfs]
     65 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 [netns]

[ssax]

You have a ton of indices open, I would close the old indices in Admin > Index Status (say anything over 30 days to start with, that should give you better performance).

Do you have Admin > Snapshots & Maintenance setup to auto-close?

Do you have these or higher set in your /etc/php.ini?

Code: Select all

max_execution_time = 300
memory_limit = 1024M

Just remember that because you're running php-fpm now, you need to restart that to pick up any php changes, I usually just use these on the systems that run php-fpm (EL8+/Ubuntu/etc):

Code: Select all

systemctl restart httpd php-fpm

EDIT: Yep, either add memory or close some indices, that's why elastic is flooding the /var/log/elasticsearch log with these warnings:

Code: Select all

[2021-07-30 13:28:29,352][WARN ][monitor.jvm              ] [e68eac14-2e1d-42cd-ccc-xxxxxxxx] [gc][old][95420][15349] duration [14.3s], collections [1]/[15.1s], total [14.3s]/[2d], memory [3.4gb]->[3.5gb]/[3.7gb], all_pools {[young] [361.6mb]->[409.7mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old] [3.1gb]->[3.1gb]/[3.1gb]}

[BanditBBS]

Thanks, that was the issue. Set them to auto close after 90 days and seems much better now!

[ssax]

Glad that fixed, it let us know when we're okay to lock this up and mark it as resolved.