Long reboots on Ubuntu after installing NLS

[mlabbepg]

Thanks for your reply.

First reboot after increasing ram (and restarting services) went smooth (around 20 secs), then all subsequent reboots take 5 mins again:

Code: Select all

Oct 13 11:45:49 syslog-mgmt systemd[1]: Stopping LSB: Logstash...
Oct 13 11:45:49 syslog-mgmt logstash[244448]:  * Stopping Logstash Daemon
Oct 13 11:46:08 syslog-mgmt logstash[244448]:    ...done.
Oct 13 11:46:08 syslog-mgmt systemd[1]: logstash.service: Succeeded.
Oct 13 11:46:08 syslog-mgmt systemd[1]: Stopped LSB: Logstash.
Oct 13 11:46:08 syslog-mgmt systemd[1]: Starting LSB: Logstash...
Oct 13 11:46:08 syslog-mgmt logstash[244575]:  * Starting Logstash Daemon
Oct 13 11:46:08 syslog-mgmt logstash[244575]: /etc/init.d/logstash: invalid arguments
Oct 13 11:46:08 syslog-mgmt logstash[244575]:    ...done.
Oct 13 11:46:08 syslog-mgmt systemd[1]: Started LSB: Logstash.
Oct 13 11:46:30 syslog-mgmt systemd[1]: Stopping LSB: Logstash...
Oct 13 11:46:30 syslog-mgmt logstash[244732]:  * Stopping Logstash Daemon
Oct 13 11:46:31 syslog-mgmt logstash[244732]:    ...done.
Oct 13 11:46:31 syslog-mgmt systemd[1]: logstash.service: Succeeded.
Oct 13 11:46:31 syslog-mgmt systemd[1]: Stopped LSB: Logstash.
-- Reboot --
Oct 13 11:46:49 syslog-mgmt systemd[1]: Starting LSB: Logstash...
Oct 13 11:46:50 syslog-mgmt logstash[862]:  * Starting Logstash Daemon
Oct 13 11:46:50 syslog-mgmt logstash[862]: /etc/init.d/logstash: invalid arguments
Oct 13 11:46:50 syslog-mgmt logstash[862]:    ...done.
Oct 13 11:46:50 syslog-mgmt systemd[1]: Started LSB: Logstash.
Oct 13 11:47:45 syslog-mgmt systemd[1]: Stopping LSB: Logstash...
Oct 13 11:47:45 syslog-mgmt logstash[1691]:  * Stopping Logstash Daemon
Oct 13 11:52:45 syslog-mgmt systemd[1]: logstash.service: Stopping timed out. Terminating.
Oct 13 11:52:45 syslog-mgmt systemd[1]: logstash.service: Control process exited, code=killed, status=15/TERM
Oct 13 11:52:45 syslog-mgmt systemd[1]: logstash.service: Failed with result 'timeout'.
Oct 13 11:52:45 syslog-mgmt systemd[1]: Stopped LSB: Logstash.
-- Reboot --
Oct 13 11:54:32 syslog-mgmt systemd[1]: Starting LSB: Logstash...
Oct 13 11:54:33 syslog-mgmt logstash[861]:  * Starting Logstash Daemon
Oct 13 11:54:33 syslog-mgmt logstash[861]: /etc/init.d/logstash: invalid arguments
Oct 13 11:54:33 syslog-mgmt logstash[861]:    ...done.
Oct 13 11:54:33 syslog-mgmt systemd[1]: Started LSB: Logstash.
Oct 13 12:37:38 syslog-mgmt systemd[1]: Stopping LSB: Logstash...
Oct 13 12:37:38 syslog-mgmt logstash[8515]:  * Stopping Logstash Daemon
Oct 13 12:42:38 syslog-mgmt systemd[1]: logstash.service: Stopping timed out. Terminating.
Oct 13 12:42:38 syslog-mgmt systemd[1]: logstash.service: Control process exited, code=killed, status=15/TERM
Oct 13 12:42:38 syslog-mgmt systemd[1]: logstash.service: Failed with result 'timeout'.
Oct 13 12:42:38 syslog-mgmt systemd[1]: Stopped LSB: Logstash.
-- Reboot --
Oct 13 12:44:26 syslog-mgmt systemd[1]: Starting LSB: Logstash...
Oct 13 12:44:26 syslog-mgmt logstash[860]:  * Starting Logstash Daemon
Oct 13 12:44:26 syslog-mgmt logstash[860]: /etc/init.d/logstash: invalid arguments
Oct 13 12:44:26 syslog-mgmt logstash[860]:    ...done.
Oct 13 12:44:26 syslog-mgmt systemd[1]: Started LSB: Logstash.

I'm not sure what to look for to check if it's still running out of resources or another error.

Should I increase http.max_content_length value even more?

[pbroste]

Hello @mlabbepg

Went back through to review what we have touched on to see if previous info would shed some light on what is going on.

As we recall, previously we saw that 'Elasticsearch is unreachable';

message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://localhost:9200\"]', but Elasticsearch appears to be unreachable or down!"

Appears that those messages and others are suggesting that 'elasticsearch' is shutdown while 'logstash' is trying to tie up loose ends. Going forward we want elasticsearch to keep on chugging along waiting for logstash service to stop first. Then we want to see thing elasticsearch service stop only after logstash. To do that we see can edit the logstash.service systemd, and found the following:

[Unit]
After=elasticsearch.service

Which again in theory should force ElasticSearch to await Logstash to shuts down.

And then also adding this too:

[Service]
TimeoutStopSec=400

Please let us know how things bounce,
Perry

[mlabbepg]

It took me some time to figure out how to change the service config (for the records: sudo systemctl edit logstash.service), but adding that part alone seems to work great so far:

Code: Select all

[Unit]
After=elasticsearch.service

Reboot times are much better :

Code: Select all

Oct 14 16:16:44 syslog-mgmt logstash[271945]:  * Stopping Logstash Daemon
Oct 14 16:16:50 syslog-mgmt logstash[271945]:    ...done.
Oct 14 16:16:50 syslog-mgmt systemd[1]: logstash.service: Succeeded.
Oct 14 16:16:50 syslog-mgmt systemd[1]: Stopped LSB: Logstash.
-- Reboot --
Oct 14 16:17:03 syslog-mgmt systemd[1]: Starting LSB: Logstash...

Oct 14 16:18:06 syslog-mgmt systemd[1]: Stopping LSB: Logstash...
Oct 14 16:18:06 syslog-mgmt logstash[1693]:  * Stopping Logstash Daemon
Oct 14 16:18:12 syslog-mgmt logstash[1693]:    ...done.
Oct 14 16:18:12 syslog-mgmt systemd[1]: logstash.service: Succeeded.
Oct 14 16:18:12 syslog-mgmt systemd[1]: Stopped LSB: Logstash.
-- Reboot --
Oct 14 16:18:24 syslog-mgmt systemd[1]: Starting LSB: Logstash...

Oct 14 16:19:17 syslog-mgmt systemd[1]: Stopping LSB: Logstash...
Oct 14 16:19:18 syslog-mgmt logstash[1812]:  * Stopping Logstash Daemon
Oct 14 16:19:24 syslog-mgmt logstash[1812]:    ...done.
Oct 14 16:19:24 syslog-mgmt systemd[1]: logstash.service: Succeeded.
Oct 14 16:19:24 syslog-mgmt systemd[1]: Stopped LSB: Logstash.
-- Reboot --
Oct 14 16:19:36 syslog-mgmt systemd[1]: Starting LSB: Logstash...

Oct 14 16:20:52 syslog-mgmt systemd[1]: Stopping LSB: Logstash...
Oct 14 16:20:52 syslog-mgmt logstash[1841]:  * Stopping Logstash Daemon
Oct 14 16:20:58 syslog-mgmt logstash[1841]:    ...done.
Oct 14 16:20:58 syslog-mgmt systemd[1]: logstash.service: Succeeded.
Oct 14 16:20:58 syslog-mgmt systemd[1]: Stopped LSB: Logstash.
-- Reboot --
Oct 14 16:21:11 syslog-mgmt systemd[1]: Starting LSB: Logstash...

I strongly suggest you add that setting to your install.sh script.

Thanks.

[pbroste]

Hello @mlabbepg

Excellent, glad to hear that it is working much better now. I will make the suggestion to add that feature request to the install script.

Please let me know if it is okay to close this thread,
Perry