Cap the amount of memory logstash and elasticsearch can use?

[rferebee]

Hello,

Is it possible to place a hard cap on the amount of memory that logstash and elasticsearch are allowed to utilize? Perhaps by percentage rather than a specific GB amount?

Each node in my Log Server instances has 64GBs of memory, but whenever the logstash or elasticsearch services need to restart there isn't enough available memory for them to do it and they end up failing to stop properly. For example, I updated my nodes to version 2.1.1 this morning and when one of them tried to stop elasticsearch it threw the following error:

Code: Select all

root@nagioslscc2:/root> systemctl status elasticsearch
● elasticsearch.service - LSB: This service manages the elasticsearch daemon
   Loaded: loaded (/etc/rc.d/init.d/elasticsearch; bad; vendor preset: disabled)
   Active: active (running) since Thu 2019-11-07 11:10:08 PST; 11min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 15631 ExecStop=/etc/rc.d/init.d/elasticsearch stop (code=exited, status=1/FAILURE)
  Process: 15857 ExecStart=/etc/rc.d/init.d/elasticsearch start (code=exited, status=0/SUCCESS)
    Tasks: 100
   CGroup: /system.slice/elasticsearch.service
           └─15883 /bin/java -Xms32133m -Xmx32133m -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDump...

Nov 07 11:10:08 nagioslscc2 systemd[1]: Starting LSB: This service manages the elasticsearch daemon...
Nov 07 11:10:08 nagioslscc2 runuser[15874]: pam_unix(runuser:session): session opened for user nagios by (uid=0)
Nov 07 11:10:08 nagioslscc2 runuser[15874]: pam_unix(runuser:session): session closed for user nagios
Nov 07 11:10:08 nagioslscc2 elasticsearch[15857]: Starting elasticsearch: [  OK  ]
Nov 07 11:10:08 nagioslscc2 systemd[1]: Started LSB: This service manages the elasticsearch daemon.

It happens every time. I think it's because elasticsearch is using more than 50% of the total memory.

Adding more memory to the system wouldn't help would it? I was thinking we could cap the memory usage for elasticsearch to like 49% or something like that? Not sure if that's even possible.

Thank you.

[cdienger]

Elasticsearch will attempt to use half of the system's total memory with this line in /etc/sysconfig/elasticsearch:

Code: Select all

ES_HEAP_SIZE=$(expr $(free -m|awk '/^Mem:/{print $2}') / 2 )m

If you wanted to lower the value you could modify the expression, or just comment out the line and set it with something like:

Code: Select all

#ES_HEAP_SIZE=$(expr $(free -m|awk '/^Mem:/{print $2}') / 2 )m
ES_HEAP_SIZE=3100m

Make sure to restart elasticsearch afterwards:

Code: Select all

service elasticsearch restart

Logstash memory is controlled by the LS_HEAP_SIZE line in /etc/init.d/logstash, but this is usually much lower usage.

[rferebee]

Thank you, this can be locked.

[cdienger]

Locking.