Max Source Limit

[goengineer]

How can the max number of sources be increased, or how is it currently limited? For instance, we have been adding sources but have reached a limit with the number of sources that can run at once. This maxes out at 124 concurrent sources running. When we start sources beyond that number, the respective service starts, but then stops. If we stop a source that is already running, we can then start a source that was not able to run. The server is not being taxed on resources (CPU, RAM, etc.). Rather this must have something to do with maximum number of processes that can be run by the 'nna' user (we bumped the nproc limit for that user from 1024 to 16384, to no avail) or the maximum number of open files for the 'nna' user (also bumped from 1024 to 16834, to no avail). Or... perhaps this was a hard-set limit somewhere in the code?

Anyway, this is a critical problem that we need resolved immediately. Any suggestions?

[scottwilkerson]

I think you are going down the right path, but not sure which limit you are hitting. Are you getting any clues in the syslog on the server?

Code: Select all

tail -f /var/log/messages

[goengineer]

Thank you for the response. We actually solved this a few hours after posting, but had to wait for our post to be approved to even show up.

Agreed. That was not the correct path initially. The limit we were running into was the semaphore kernel parameters, specifically semms1. This is set by default to be 250, which matches the observed limit in the number of processes run by the 'nna' user. The nna user launches the nna process itself, plus two processes for each connection. When we had 124 sources running, the number of processes running as the nna user were 249 (as observed by 'ps -u nna | wc -l'). When we started additional sources, that number would jump up 2 per additional source, but then run into kernel limitations and each new source would shut down until the number dropped down below 250.

We had observed this corresponding error in the syslog: 'semget() error in bookkeeper.c line 258: No space left on device'

We solved this by bumping the parameters of kernel.sem (semms1, semmns, semopm, semmni) from the defaults of 250, 32000, 100, 128 respectively to be 4096, 1048576, 4096, 256. This was done by placing the following line in /etc/sysctl.conf, then running 'sysctl -p':

kernel.sem = 4096 1048576 4096 256

This will now allow us to have up 2047 sources running (unless we hit another limit somewhere else first).

[scottwilkerson]

Nice!

Thanks for posting back!