Logstash.log is not created when logstash is started or restarted

[kboppana1]

Hello,

I'm using NLS 2024R1.3.1. I'm not seeing logstash.log created at all in my system as part of process startup. I tried restarting using command line systmctl restart/start/stop logstash and also from GUI. System status restarting the logstash collector.


1. logstash.log is not created.
2. Logstash is consuming higher CPU process and eventually crashing

ps -ef | grep -i logstash
nagios 750936 1 99 06:40 ? 00:17:32 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=$APP_DIR/tmp -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintClassHistogram -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:./logstash-gc.log -Xmx1g -Xss2048k -Djffi.boot.library.path=/usr/local/nagioslogserver/logstash/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=$APP_DIR/tmp -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintClassHistogram -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:./logstash-gc.log -XX:HeapDumpPath=/usr/local/nagioslogserver/logstash/heapdump.hprof -Xbootclasspath/a:/usr/local/nagioslogserver/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/local/nagioslogserver/logstash/vendor/jruby -Djruby.lib=/usr/local/nagioslogserver/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /usr/local/nagioslogserver/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f /usr/local/nagioslogserver/logstash/etc/conf.d

I believe the
Logstash is started by Nagios Log Server via /etc/init.d/logstash (wrapped in runuser).

It should write logs to /var/log/logstash/logstash.log, but not happening.

The process is showing high CPU (Java at ~99%) but Debugging is blocked because we don’t see the logs.

Please let me know if I need to open a case if this route takes longer to get a response since this is happening in production, we are expecting some response as soon as possible.

[kg2857]

Do the permissions of the logstash dir allow the user running logstash to write in it?
Read the Community Support Forum announcement for info on this forum.

[rpitts]

some logstash log output is redirected to a separate location, and can be found in /usr/local/nagioslogserver/logstash/logs

[kboppana1]

yes.

drwxr-xr-x. 2 nagios nagios 4.0K Aug 18 13:07 logstash

The directory is owned by Logstash and other servers in the cluster are able to write the logs.

[kg2857]

Restart it from the shell and look at the output.