Nagios Support Forum

Support for Nagios products and services
https://support.nagios.com/forum/

Long reboots on Ubuntu after installing NLS

https://support.nagios.com/forum/viewtopic.php?t=63648

Page 2 of 2

Re: Long reboots on Ubuntu after installing NLS

Posted: Wed Oct 13, 2021 1:33 pm
by mlabbepg
Thanks for your reply.

First reboot after increasing ram (and restarting services) went smooth (around 20 secs), then all subsequent reboots take 5 mins again:

Code: Select all

Oct 13 11:45:49 syslog-mgmt systemd[1]: Stopping LSB: Logstash...
Oct 13 11:45:49 syslog-mgmt logstash[244448]:  * Stopping Logstash Daemon
Oct 13 11:46:08 syslog-mgmt logstash[244448]:    ...done.
Oct 13 11:46:08 syslog-mgmt systemd[1]: logstash.service: Succeeded.
Oct 13 11:46:08 syslog-mgmt systemd[1]: Stopped LSB: Logstash.
Oct 13 11:46:08 syslog-mgmt systemd[1]: Starting LSB: Logstash...
Oct 13 11:46:08 syslog-mgmt logstash[244575]:  * Starting Logstash Daemon
Oct 13 11:46:08 syslog-mgmt logstash[244575]: /etc/init.d/logstash: invalid arguments
Oct 13 11:46:08 syslog-mgmt logstash[244575]:    ...done.
Oct 13 11:46:08 syslog-mgmt systemd[1]: Started LSB: Logstash.
Oct 13 11:46:30 syslog-mgmt systemd[1]: Stopping LSB: Logstash...
Oct 13 11:46:30 syslog-mgmt logstash[244732]:  * Stopping Logstash Daemon
Oct 13 11:46:31 syslog-mgmt logstash[244732]:    ...done.
Oct 13 11:46:31 syslog-mgmt systemd[1]: logstash.service: Succeeded.
Oct 13 11:46:31 syslog-mgmt systemd[1]: Stopped LSB: Logstash.
-- Reboot --
Oct 13 11:46:49 syslog-mgmt systemd[1]: Starting LSB: Logstash...
Oct 13 11:46:50 syslog-mgmt logstash[862]:  * Starting Logstash Daemon
Oct 13 11:46:50 syslog-mgmt logstash[862]: /etc/init.d/logstash: invalid arguments
Oct 13 11:46:50 syslog-mgmt logstash[862]:    ...done.
Oct 13 11:46:50 syslog-mgmt systemd[1]: Started LSB: Logstash.
Oct 13 11:47:45 syslog-mgmt systemd[1]: Stopping LSB: Logstash...
Oct 13 11:47:45 syslog-mgmt logstash[1691]:  * Stopping Logstash Daemon
Oct 13 11:52:45 syslog-mgmt systemd[1]: logstash.service: Stopping timed out. Terminating.
Oct 13 11:52:45 syslog-mgmt systemd[1]: logstash.service: Control process exited, code=killed, status=15/TERM
Oct 13 11:52:45 syslog-mgmt systemd[1]: logstash.service: Failed with result 'timeout'.
Oct 13 11:52:45 syslog-mgmt systemd[1]: Stopped LSB: Logstash.
-- Reboot --
Oct 13 11:54:32 syslog-mgmt systemd[1]: Starting LSB: Logstash...
Oct 13 11:54:33 syslog-mgmt logstash[861]:  * Starting Logstash Daemon
Oct 13 11:54:33 syslog-mgmt logstash[861]: /etc/init.d/logstash: invalid arguments
Oct 13 11:54:33 syslog-mgmt logstash[861]:    ...done.
Oct 13 11:54:33 syslog-mgmt systemd[1]: Started LSB: Logstash.
Oct 13 12:37:38 syslog-mgmt systemd[1]: Stopping LSB: Logstash...
Oct 13 12:37:38 syslog-mgmt logstash[8515]:  * Stopping Logstash Daemon
Oct 13 12:42:38 syslog-mgmt systemd[1]: logstash.service: Stopping timed out. Terminating.
Oct 13 12:42:38 syslog-mgmt systemd[1]: logstash.service: Control process exited, code=killed, status=15/TERM
Oct 13 12:42:38 syslog-mgmt systemd[1]: logstash.service: Failed with result 'timeout'.
Oct 13 12:42:38 syslog-mgmt systemd[1]: Stopped LSB: Logstash.
-- Reboot --
Oct 13 12:44:26 syslog-mgmt systemd[1]: Starting LSB: Logstash...
Oct 13 12:44:26 syslog-mgmt logstash[860]:  * Starting Logstash Daemon
Oct 13 12:44:26 syslog-mgmt logstash[860]: /etc/init.d/logstash: invalid arguments
Oct 13 12:44:26 syslog-mgmt logstash[860]:    ...done.
Oct 13 12:44:26 syslog-mgmt systemd[1]: Started LSB: Logstash.
I'm not sure what to look for to check if it's still running out of resources or another error.

Should I increase http.max_content_length value even more?

Re: Long reboots on Ubuntu after installing NLS

Posted: Thu Oct 14, 2021 1:49 pm
by pbroste
Hello @mlabbepg

Went back through to review what we have touched on to see if previous info would shed some light on what is going on.

As we recall, previously we saw that 'Elasticsearch is unreachable';
message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://localhost:9200\"]', but Elasticsearch appears to be unreachable or down!"
Appears that those messages and others are suggesting that 'elasticsearch' is shutdown while 'logstash' is trying to tie up loose ends. Going forward we want elasticsearch to keep on chugging along waiting for logstash service to stop first. Then we want to see thing elasticsearch service stop only after logstash. To do that we see can edit the logstash.service systemd, and found the following:
[Unit]
After=elasticsearch.service
Which again in theory should force ElasticSearch to await Logstash to shuts down.

And then also adding this too:
[Service]
TimeoutStopSec=400
Please let us know how things bounce,
Perry

Re: Long reboots on Ubuntu after installing NLS

Posted: Thu Oct 14, 2021 3:58 pm
by mlabbepg
It took me some time to figure out how to change the service config (for the records: sudo systemctl edit logstash.service), but adding that part alone seems to work great so far:

Code: Select all

[Unit]
After=elasticsearch.service
Reboot times are much better :D :

Code: Select all

Oct 14 16:16:44 syslog-mgmt logstash[271945]:  * Stopping Logstash Daemon
Oct 14 16:16:50 syslog-mgmt logstash[271945]:    ...done.
Oct 14 16:16:50 syslog-mgmt systemd[1]: logstash.service: Succeeded.
Oct 14 16:16:50 syslog-mgmt systemd[1]: Stopped LSB: Logstash.
-- Reboot --
Oct 14 16:17:03 syslog-mgmt systemd[1]: Starting LSB: Logstash...

Oct 14 16:18:06 syslog-mgmt systemd[1]: Stopping LSB: Logstash...
Oct 14 16:18:06 syslog-mgmt logstash[1693]:  * Stopping Logstash Daemon
Oct 14 16:18:12 syslog-mgmt logstash[1693]:    ...done.
Oct 14 16:18:12 syslog-mgmt systemd[1]: logstash.service: Succeeded.
Oct 14 16:18:12 syslog-mgmt systemd[1]: Stopped LSB: Logstash.
-- Reboot --
Oct 14 16:18:24 syslog-mgmt systemd[1]: Starting LSB: Logstash...

Oct 14 16:19:17 syslog-mgmt systemd[1]: Stopping LSB: Logstash...
Oct 14 16:19:18 syslog-mgmt logstash[1812]:  * Stopping Logstash Daemon
Oct 14 16:19:24 syslog-mgmt logstash[1812]:    ...done.
Oct 14 16:19:24 syslog-mgmt systemd[1]: logstash.service: Succeeded.
Oct 14 16:19:24 syslog-mgmt systemd[1]: Stopped LSB: Logstash.
-- Reboot --
Oct 14 16:19:36 syslog-mgmt systemd[1]: Starting LSB: Logstash...

Oct 14 16:20:52 syslog-mgmt systemd[1]: Stopping LSB: Logstash...
Oct 14 16:20:52 syslog-mgmt logstash[1841]:  * Stopping Logstash Daemon
Oct 14 16:20:58 syslog-mgmt logstash[1841]:    ...done.
Oct 14 16:20:58 syslog-mgmt systemd[1]: logstash.service: Succeeded.
Oct 14 16:20:58 syslog-mgmt systemd[1]: Stopped LSB: Logstash.
-- Reboot --
Oct 14 16:21:11 syslog-mgmt systemd[1]: Starting LSB: Logstash...
I strongly suggest you add that setting to your install.sh script.

Thanks.

Re: Long reboots on Ubuntu after installing NLS

Posted: Fri Oct 15, 2021 11:17 am
by pbroste
Hello @mlabbepg

Excellent, glad to hear that it is working much better now. I will make the suggestion to add that feature request to the install script.

Please let me know if it is okay to close this thread,
Perry
All times are UTC-05:00
Page 2 of 2

Powered by phpBB® Forum Software © phpBB Limited