Server stops collecting logs after increasing PHP memory siz

[bpizzutiWHI]

Ok, so I was having an issue where the admin page was not accessible. As recommended, I increased the PHP memory size from 128 to 256. Now the server collects logs for a while after starting and then just stops. I should add that this server's got 64 MB of RAM. I grabbed some pertinent info from the Instance status page:


1m, 5m, 15m Load: 0.34, 0.43, 0.22
Memory (Used/Free): 40.73 GB / 21.92 GB
Swap (Used/Free): 0 bytes / 2.00 GB
Total Memory: 62.65 GB
Total Swap: 2.00 GB
CPU User/Sys: 0% / 0%
CPU Idle: 98%
CPU Vendor: Intel
CPU Model: Xeon
Total Cores: 28


Open File Descriptors: 2435
CPU Usage: 14% of 2800%
CPU System: 2.3m
CPU User: 32.8m
CPU Total: 35.2m
Resident Memory: 32.67 GB
Shared Memory: 144.80 MB
Total Virtual Memory: 62.38 GB

[scottwilkerson]

This doesn't seem right at all.

What OS version are you running?

Can you post the output of the following and maybe we will be able to see the error generated

Code: Select all

tail -100 /var/log/httpd/error_log

[bpizzutiWHI]

It's running Cent7, fully updated as of a month ago.

Code: Select all

[Sun Sep 17 03:39:01.435424 2017] [auth_digest:notice] [pid 2213] AH01757: generating secret for digest authentication ...
[Sun Sep 17 03:39:01.443382 2017] [lbmethod_heartbeat:notice] [pid 2213] AH02282: No slotmem from mod_heartmonitor
[Sun Sep 17 03:39:01.457891 2017] [mpm_prefork:notice] [pid 2213] AH00163: Apache/2.4.6 (CentOS) PHP/5.4.16 configured -- resuming normal operations
[Sun Sep 17 03:39:01.457906 2017] [core:notice] [pid 2213] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Tue Sep 19 15:11:13.318662 2017] [mpm_prefork:notice] [pid 2213] AH00170: caught SIGWINCH, shutting down gracefully
[Tue Sep 19 15:13:34.556996 2017] [suexec:notice] [pid 2244] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Tue Sep 19 15:13:34.596354 2017] [auth_digest:notice] [pid 2244] AH01757: generating secret for digest authentication ...
[Tue Sep 19 15:13:34.597459 2017] [lbmethod_heartbeat:notice] [pid 2244] AH02282: No slotmem from mod_heartmonitor
[Tue Sep 19 15:13:34.620579 2017] [mpm_prefork:notice] [pid 2244] AH00163: Apache/2.4.6 (CentOS) PHP/5.4.16 configured -- resuming normal operations
[Tue Sep 19 15:13:34.620611 2017] [core:notice] [pid 2244] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Wed Sep 20 09:51:52.444767 2017] [mpm_prefork:notice] [pid 2244] AH00170: caught SIGWINCH, shutting down gracefully
[Wed Sep 20 09:51:53.531184 2017] [suexec:notice] [pid 39889] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Wed Sep 20 09:51:53.565594 2017] [auth_digest:notice] [pid 39889] AH01757: generating secret for digest authentication ...
[Wed Sep 20 09:51:53.566513 2017] [lbmethod_heartbeat:notice] [pid 39889] AH02282: No slotmem from mod_heartmonitor
[Wed Sep 20 09:51:53.583357 2017] [mpm_prefork:notice] [pid 39889] AH00163: Apache/2.4.6 (CentOS) PHP/5.4.16 configured -- resuming normal operations
[Wed Sep 20 09:51:53.583379 2017] [core:notice] [pid 39889] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   893  100   893    0     0   3470      0 --:--:-- --:--:-- --:--:--  3474
[Thu Sep 21 10:54:58.116823 2017] [mpm_prefork:notice] [pid 39889] AH00170: caught SIGWINCH, shutting down gracefully
[Thu Sep 21 10:57:14.707680 2017] [suexec:notice] [pid 2215] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Sep 21 10:57:14.746011 2017] [auth_digest:notice] [pid 2215] AH01757: generating secret for digest authentication ...
[Thu Sep 21 10:57:14.747480 2017] [lbmethod_heartbeat:notice] [pid 2215] AH02282: No slotmem from mod_heartmonitor
[Thu Sep 21 10:57:14.762529 2017] [mpm_prefork:notice] [pid 2215] AH00163: Apache/2.4.6 (CentOS) PHP/5.4.16 configured -- resuming normal operations
[Thu Sep 21 10:57:14.762555 2017] [core:notice] [pid 2215] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'

Not sure what might be going on. I have a decent number of listeners going (9) and 90 devices sending logs, but this does seem kinda wierd.

[scottwilkerson]

I'm sorry, I misread the issue, I thought the admin page wasn't loading anymore, but I see that it is more about the logs stopping being collected.

Can you pull the following 2 files after this happens and PM them to me
/var/log/logstash/logstash.log
/var/log/elasticsearch/<CLUSTER_ID>.log

[bpizzutiWHI]

Yep, as soon as I can make it happen again. I've been playing around with the PHP memory size to see how that affects it. Still happened at 192, though it took longer. 160 lasted about the same amont of time.

[scottwilkerson]

Yep, as soon as I can make it happen again. I've been playing around with the PHP memory size to see how that affects it. Still happened at 192, though it took longer. 160 lasted about the same amont of time.

I do not think the memory is related. Just a coincidence.

[scottwilkerson]

We would likely need to also see your logstash configurations, but one thing that stands out from the logs you PM'd me was a tom of these errors

Code: Select all

IP Field contained invalid IP address or hostname

where the following was a hostname and not an IP address

Code: Select all

"clientip"=>"your.hostname.com"

So you may need to make some modification if you are going to use the ip filter with geolocation.

I do want to point out, what I saw in the logs didn't specifically point to why it would stop listening, but with errors, it could be eating up your maximum threads.

[bpizzutiWHI]

Ok, they're kinda messing around with the Apache logs right now, I bet that broke something.

[scottwilkerson]

That would be my guess, you may have to fiddle with your config once they change them to what they are going to stay as.

[bpizzutiWHI]

Ok looks like that's not it. I turned off all of the filters using GeoIP and it's still happening.