Logstash crashes

[lleung]

[root@shnagiossrv02 ~]# service logstash status
Logstash Daemon● logstash.service - LSB: Logstash
Loaded: loaded (/etc/rc.d/init.d/logstash; bad; vendor preset: disabled)
Active: active (exited) since Sat 2019-03-30 15:04:16 HKT; 4 days ago
Docs: man:systemd-sysv-generator(8)
Process: 28999 ExecStop=/etc/rc.d/init.d/logstash stop (code=exited, status=0/SUCCESS)
Process: 29030 ExecStart=/etc/rc.d/init.d/logstash start (code=exited, status=0/SUCCESS)
Tasks: 0

Mar 30 15:04:15 shnagiossrv02 systemd[1]: Starting LSB: Logstash...
Mar 30 15:04:16 shnagiossrv02 runuser[29041]: pam_unix(runuser:session): session opened for user root by (uid=0)
Mar 30 15:04:16 shnagiossrv02 logstash[29030]: Starting Logstash Daemon: [ OK ]
Mar 30 15:04:16 shnagiossrv02 systemd[1]: Started LSB: Logstash.
Apr 02 21:56:14 shnagiossrv02 logstash[29030]: Error: Your application used more memory than the safety cap of 4G.
Apr 02 21:56:14 shnagiossrv02 logstash[29030]: Specify -J-Xmx####m to increase it (#### = cap size in MB).
Apr 02 21:56:14 shnagiossrv02 logstash[29030]: Specify -w for full OutOfMemoryError stack trace
Apr 02 21:56:15 shnagiossrv02 runuser[29041]: pam_unix(runuser:session): session closed for user root
[root@shnagiossrv02 ~]# service logstash restart
Restarting logstash (via systemctl): [ OK ]
[root@shnagiossrv02 ~]# service logstash status
Logstash Daemon● logstash.service - LSB: Logstash
Loaded: loaded (/etc/rc.d/init.d/logstash; bad; vendor preset: disabled)
Active: active (running) since Wed 2019-04-03 17:36:00 HKT; 57s ago
Docs: man:systemd-sysv-generator(8)
Process: 15023 ExecStop=/etc/rc.d/init.d/logstash stop (code=exited, status=0/SUCCESS)
Process: 15038 ExecStart=/etc/rc.d/init.d/logstash start (code=exited, status=0/SUCCESS)
Tasks: 60
CGroup: /system.slice/logstash.service
├─15049 runuser -s /bin/sh -c exec /usr/local/nagioslogserver/logstash/bin/logstash agent -f /usr/local/nagioslogser...
└─15051 /bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFracti...

Apr 03 17:36:00 shnagiossrv02 systemd[1]: Starting LSB: Logstash...
Apr 03 17:36:00 shnagiossrv02 runuser[15049]: pam_unix(runuser:session): session opened for user root by (uid=0)
Apr 03 17:36:00 shnagiossrv02 logstash[15038]: Starting Logstash Daemon: [ OK ]
Apr 03 17:36:00 shnagiossrv02 systemd[1]: Started LSB: Logstash.
[root@shnagiossrv02 ~]# ps -ef | grep logstash
root 15049 1 0 17:35 ? 00:00:00 runuser -s /bin/sh -c exec /usr/local/nagioslogserver/logstash/bin/logstash agent -f /usr/local/nagioslogserver/logstash/etc/conf.d -l /var/log/logstash/logstash.log -w 4 root
root 15051 15049 99 17:35 ? 00:03:11 /bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -Xmx4g -Xss2048k -Djffi.boot.library.path=/usr/local/nagioslogserver/logstash/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -XX:HeapDumpPath=/usr/local/nagioslogserver/logstash/heapdump.hprof -Xbootclasspath/a:/usr/local/nagioslogserver/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/local/nagioslogserver/logstash/vendor/jruby -Djruby.lib=/usr/local/nagioslogserver/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /usr/local/nagioslogserver/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f /usr/local/nagioslogserver/logstash/etc/conf.d -l /var/log/logstash/logstash.log -w 4
root 16198 10971 0 17:38 pts/0 00:00:00 grep --color=auto logstash


[root@shnagiossrv02 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rhel-root 1.0T 7.4G 1017G 1% /
devtmpfs 32G 0 32G 0% /dev
tmpfs 32G 0 32G 0% /dev/shm
tmpfs 32G 298M 32G 1% /run
tmpfs 32G 0 32G 0% /sys/fs/cgroup
/dev/sda2 1016M 259M 757M 26% /boot
/dev/sda1 200M 9.8M 190M 5% /boot/efi
/dev/mapper/rhel-home 1.0T 40M 1.0T 1% /home
/dev/mapper/rhel-nagios 14T 6.8T 6.5T 52% /nagios
tmpfs 6.3G 12K 6.3G 1% /run/user/42
tmpfs 6.3G 0 6.3G 0% /run/user/1001
tmpfs 6.3G 0 6.3G 0% /run/user/0
[root@shnagiossrv02 ~]# meminfo
bash: meminfo: command not found...
[root@shnagiossrv02 ~]# free
total used free shared buff/cache available
Mem: 65674664 35909600 4626008 301712 25139056 28721900
Swap: 67100668 99072 67001596
[root@shnagiossrv02 ~]#

[cdienger]

Please PM me a profile from the system. It can be gathered under Admin > System > System Status > Download System Profile or from the command line with:

Code: Select all

/usr/local/nagioslogserver/scripts/profile.sh

This will create /tmp/system-profile.tar.gz.

Note that this file can be very large and may not be able to be uploaded through the messaging system. This is usually due to the logs in the logstash and/or elasticseach directories found in it. If it is too large, please open the file, extract these directories/files and send them separately.

[cdienger]

There are a lot of logstash errors regarding the date format of an Apache log source:

{:timestamp=>"2019-04-02T21:27:32.411000+0800", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"Apr 2 21:10:45", :exception=>"Invalid format: \"Apr 2 21:10:45\"", :config_parsers=>"dd/MMM/yyyy:HH:mm:ss Z,MMM dd HH:mm:ss,ISO8601", :config_locale=>"default=en_US", :level=>:warn}

Check the Apache servers that are sending log to make sure they're logging a date format that would match 'dd/MMM/yyyy:HH:mm:ss Z', 'MMM dd HH:mm:ss', or 'ISO8601'.

[lleung]

how do I know which source IP's apache log submitted the error?

[cdienger]

Unfortunately the logs don't tell us that at the moment. But we can try enabling debug logging to get more information. Do this by editing /etc/init.d/logstash and changing line 64:

DAEMON_OPTS="agent -f ${LS_CONF_DIR} -l ${LS_LOG_FILE} ${LS_OPTS}"

to:

DAEMON_OPTS="agent -f ${LS_CONF_DIR} -l ${LS_LOG_FILE} ${LS_OPTS} --debug"

and restarting the service:

systemctl daemon-reload
service logstash restart

the debug information will be logged to /var/log/logstash/logstash.log. Let it run this for 2 or 3 minutes and then revert the changes and restart the service to disable debug logging. Please PM me a copy of logstash.log.