The logstash service has been crashing off and on for the past week or so. I am not collecting the same amount of logs that I have in the past, used to collect more, so not sure what is happening.
We did start getting all auditd failure events which has increased the number events per minute quite a bit.
Help??
NLS Out of Memory Error
Re: NLS Out of Memory Error
Can you share the contents of your logstash and elasticsearch logs? They should be located at:
Code: Select all
/var/log/elasticsearch/*.log
/var/log/logtsash/logstash.logFormer Nagios employee
https://www.mcapra.com/
https://www.mcapra.com/
-
krobertson71
- Posts: 444
- Joined: Tue Feb 11, 2014 10:16 pm
Re: NLS Out of Memory Error
Here you go.
You do not have the required permissions to view the files attached to this post.
Re: NLS Out of Memory Error
Lots of java.util.concurrent.RejectedExecutionException in your elasticsearch log. That's probably the root of the issues.
Please share the output of the following commands executed from the CLI of one of your NLS nodes:
Please share the output of the following commands executed from the CLI of one of your NLS nodes:
Code: Select all
curl -XGET 'http://localhost:9200/_cluster/health/*?level=shards'
curl -XGET 'http://localhost:9200/_cat/shards'
curl -XGET 'http://localhost:9200/_cat/indices'
curl -XGET 'http://localhost:9200/_nodes/jvm'
curl -XGET 'http://localhost:9200/_nodes'
curl -XGET 'http://localhost:9200/_cluster/state'
free -m
ps -aef
df -h && df -ihFormer Nagios employee
https://www.mcapra.com/
https://www.mcapra.com/
-
krobertson71
- Posts: 444
- Joined: Tue Feb 11, 2014 10:16 pm
Re: NLS Out of Memory Error
Here you go .. In order of request . Files attached.
Remaining attachments in next post.
Remaining attachments in next post.
Code: Select all
free -m && ps -aef && df -h && df -ih
total used free shared buffers cached
Mem: 36149 35707 442 0 471 14996
-/+ buffers/cache: 20239 15910
Swap: 2047 35 2012
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 Nov19 ? 00:00:01 /sbin/init
root 2 0 0 Nov19 ? 00:00:00 [kthreadd]
root 3 2 0 Nov19 ? 00:00:03 [migration/0]
root 4 2 0 Nov19 ? 00:00:04 [ksoftirqd/0]
root 5 2 0 Nov19 ? 00:00:00 [stopper/0]
root 6 2 0 Nov19 ? 00:00:01 [watchdog/0]
root 7 2 0 Nov19 ? 00:00:03 [migration/1]
root 8 2 0 Nov19 ? 00:00:00 [stopper/1]
root 9 2 0 Nov19 ? 00:00:05 [ksoftirqd/1]
root 10 2 0 Nov19 ? 00:00:01 [watchdog/1]
root 11 2 0 Nov19 ? 00:00:03 [migration/2]
root 12 2 0 Nov19 ? 00:00:00 [stopper/2]
root 13 2 0 Nov19 ? 00:00:04 [ksoftirqd/2]
root 14 2 0 Nov19 ? 00:00:01 [watchdog/2]
root 15 2 0 Nov19 ? 00:00:03 [migration/3]
root 16 2 0 Nov19 ? 00:00:00 [stopper/3]
root 17 2 0 Nov19 ? 00:00:04 [ksoftirqd/3]
root 18 2 0 Nov19 ? 00:00:01 [watchdog/3]
root 19 2 0 Nov19 ? 00:00:03 [migration/4]
root 20 2 0 Nov19 ? 00:00:00 [stopper/4]
root 21 2 0 Nov19 ? 00:00:03 [ksoftirqd/4]
root 22 2 0 Nov19 ? 00:00:01 [watchdog/4]
root 23 2 0 Nov19 ? 00:00:03 [migration/5]
root 24 2 0 Nov19 ? 00:00:00 [stopper/5]
root 25 2 0 Nov19 ? 00:00:03 [ksoftirqd/5]
root 26 2 0 Nov19 ? 00:00:01 [watchdog/5]
root 27 2 0 Nov19 ? 00:00:43 [events/0]
root 28 2 0 Nov19 ? 00:00:40 [events/1]
root 29 2 0 Nov19 ? 00:00:42 [events/2]
root 30 2 0 Nov19 ? 00:00:41 [events/3]
root 31 2 0 Nov19 ? 00:00:41 [events/4]
root 32 2 0 Nov19 ? 00:00:52 [events/5]
root 33 2 0 Nov19 ? 00:00:00 [events/0]
root 34 2 0 Nov19 ? 00:00:00 [events/1]
root 35 2 0 Nov19 ? 00:00:00 [events/2]
root 36 2 0 Nov19 ? 00:00:00 [events/3]
root 37 2 0 Nov19 ? 00:00:00 [events/4]
root 38 2 0 Nov19 ? 00:00:00 [events/5]
root 39 2 0 Nov19 ? 00:00:00 [events_long/0]
root 40 2 0 Nov19 ? 00:00:00 [events_long/1]
root 41 2 0 Nov19 ? 00:00:00 [events_long/2]
root 42 2 0 Nov19 ? 00:00:00 [events_long/3]
root 43 2 0 Nov19 ? 00:00:00 [events_long/4]
root 44 2 0 Nov19 ? 00:00:00 [events_long/5]
root 45 2 0 Nov19 ? 00:00:00 [events_power_ef]
root 46 2 0 Nov19 ? 00:00:00 [events_power_ef]
root 47 2 0 Nov19 ? 00:00:00 [events_power_ef]
root 48 2 0 Nov19 ? 00:00:00 [events_power_ef]
root 49 2 0 Nov19 ? 00:00:00 [events_power_ef]
root 50 2 0 Nov19 ? 00:00:00 [events_power_ef]
root 51 2 0 Nov19 ? 00:00:00 [cgroup]
root 52 2 0 Nov19 ? 00:00:00 [khelper]
root 53 2 0 Nov19 ? 00:00:00 [netns]
root 54 2 0 Nov19 ? 00:00:00 [async/mgr]
root 55 2 0 Nov19 ? 00:00:00 [pm]
root 56 2 0 Nov19 ? 00:00:03 [sync_supers]
root 57 2 0 Nov19 ? 00:00:03 [bdi-default]
root 58 2 0 Nov19 ? 00:00:00 [kintegrityd/0]
root 59 2 0 Nov19 ? 00:00:00 [kintegrityd/1]
root 60 2 0 Nov19 ? 00:00:00 [kintegrityd/2]
root 61 2 0 Nov19 ? 00:00:00 [kintegrityd/3]
root 62 2 0 Nov19 ? 00:00:00 [kintegrityd/4]
root 63 2 0 Nov19 ? 00:00:00 [kintegrityd/5]
root 64 2 0 Nov19 ? 00:00:12 [kblockd/0]
root 65 2 0 Nov19 ? 00:00:13 [kblockd/1]
root 66 2 0 Nov19 ? 00:00:17 [kblockd/2]
root 67 2 0 Nov19 ? 00:00:13 [kblockd/3]
root 68 2 0 Nov19 ? 00:00:12 [kblockd/4]
root 69 2 0 Nov19 ? 00:00:13 [kblockd/5]
root 70 2 0 Nov19 ? 00:00:00 [kacpid]
root 71 2 0 Nov19 ? 00:00:00 [kacpi_notify]
root 72 2 0 Nov19 ? 00:00:00 [kacpi_hotplug]
root 73 2 0 Nov19 ? 00:00:00 [ata_aux]
root 74 2 0 Nov19 ? 00:00:00 [ata_sff/0]
root 75 2 0 Nov19 ? 00:00:00 [ata_sff/1]
root 76 2 0 Nov19 ? 00:00:00 [ata_sff/2]
root 77 2 0 Nov19 ? 00:00:00 [ata_sff/3]
root 78 2 0 Nov19 ? 00:00:00 [ata_sff/4]
root 79 2 0 Nov19 ? 00:00:00 [ata_sff/5]
root 80 2 0 Nov19 ? 00:00:00 [ksuspend_usbd]
root 81 2 0 Nov19 ? 00:00:00 [khubd]
root 82 2 0 Nov19 ? 00:00:00 [kseriod]
root 83 2 0 Nov19 ? 00:00:00 [md/0]
root 84 2 0 Nov19 ? 00:00:00 [md/1]
root 85 2 0 Nov19 ? 00:00:00 [md/2]
root 86 2 0 Nov19 ? 00:00:00 [md/3]
root 87 2 0 Nov19 ? 00:00:00 [md/4]
root 88 2 0 Nov19 ? 00:00:00 [md/5]
root 89 2 0 Nov19 ? 00:00:00 [md_misc/0]
root 90 2 0 Nov19 ? 00:00:00 [md_misc/1]
root 91 2 0 Nov19 ? 00:00:00 [md_misc/2]
root 92 2 0 Nov19 ? 00:00:00 [md_misc/3]
root 93 2 0 Nov19 ? 00:00:00 [md_misc/4]
root 94 2 0 Nov19 ? 00:00:00 [md_misc/5]
root 95 2 0 Nov19 ? 00:00:00 [linkwatch]
root 98 2 0 Nov19 ? 00:00:00 [khungtaskd]
root 99 2 0 Nov19 ? 00:00:27 [kswapd0]
root 100 2 0 Nov19 ? 00:00:00 [ksmd]
root 101 2 0 Nov19 ? 00:00:08 [khugepaged]
root 102 2 0 Nov19 ? 00:00:00 [aio/0]
root 103 2 0 Nov19 ? 00:00:00 [aio/1]
root 104 2 0 Nov19 ? 00:00:00 [aio/2]
root 105 2 0 Nov19 ? 00:00:00 [aio/3]
root 106 2 0 Nov19 ? 00:00:00 [aio/4]
root 107 2 0 Nov19 ? 00:00:00 [aio/5]
root 108 2 0 Nov19 ? 00:00:00 [crypto/0]
root 109 2 0 Nov19 ? 00:00:00 [crypto/1]
root 110 2 0 Nov19 ? 00:00:00 [crypto/2]
root 111 2 0 Nov19 ? 00:00:00 [crypto/3]
root 112 2 0 Nov19 ? 00:00:00 [crypto/4]
root 113 2 0 Nov19 ? 00:00:00 [crypto/5]
root 120 2 0 Nov19 ? 00:00:00 [kthrotld/0]
root 121 2 0 Nov19 ? 00:00:00 [kthrotld/1]
root 122 2 0 Nov19 ? 00:00:00 [kthrotld/2]
root 123 2 0 Nov19 ? 00:00:00 [kthrotld/3]
root 124 2 0 Nov19 ? 00:00:00 [kthrotld/4]
root 125 2 0 Nov19 ? 00:00:00 [kthrotld/5]
root 126 2 0 Nov19 ? 00:00:00 [pciehpd]
root 128 2 0 Nov19 ? 00:00:00 [kpsmoused]
root 129 2 0 Nov19 ? 00:00:00 [usbhid_resumer]
root 130 2 0 Nov19 ? 00:00:00 [deferwq]
root 162 2 0 Nov19 ? 00:00:00 [kdmremove]
root 163 2 0 Nov19 ? 00:00:00 [kstriped]
root 195 2 0 Nov19 ? 00:00:00 [ttm_swap]
root 375 2 0 Nov19 ? 00:00:00 [scsi_eh_0]
root 376 2 0 Nov19 ? 00:00:00 [scsi_eh_1]
root 386 2 0 Nov19 ? 00:00:00 [scsi_eh_2]
root 387 2 0 Nov19 ? 00:00:00 [vmw_pvscsi_wq_2]
root 416 2 0 Nov19 ? 00:00:00 [scsi_eh_3]
root 417 2 0 Nov19 ? 00:00:00 [scsi_eh_4]
root 418 2 0 Nov19 ? 00:00:00 [scsi_eh_5]
root 419 2 0 Nov19 ? 00:00:00 [scsi_eh_6]
root 420 2 0 Nov19 ? 00:00:00 [scsi_eh_7]
root 421 2 0 Nov19 ? 00:00:00 [scsi_eh_8]
root 422 2 0 Nov19 ? 00:00:00 [scsi_eh_9]
root 423 2 0 Nov19 ? 00:00:00 [scsi_eh_10]
root 424 2 0 Nov19 ? 00:00:00 [scsi_eh_11]
root 425 2 0 Nov19 ? 00:00:00 [scsi_eh_12]
root 426 2 0 Nov19 ? 00:00:00 [scsi_eh_13]
root 427 2 0 Nov19 ? 00:00:00 [scsi_eh_14]
root 428 2 0 Nov19 ? 00:00:00 [scsi_eh_15]
root 429 2 0 Nov19 ? 00:00:00 [scsi_eh_16]
root 430 2 0 Nov19 ? 00:00:00 [scsi_eh_17]
root 431 2 0 Nov19 ? 00:00:00 [scsi_eh_18]
root 432 2 0 Nov19 ? 00:00:00 [scsi_eh_19]
root 433 2 0 Nov19 ? 00:00:00 [scsi_eh_20]
root 434 2 0 Nov19 ? 00:00:00 [scsi_eh_21]
root 435 2 0 Nov19 ? 00:00:00 [scsi_eh_22]
root 436 2 0 Nov19 ? 00:00:00 [scsi_eh_23]
root 437 2 0 Nov19 ? 00:00:00 [scsi_eh_24]
root 438 2 0 Nov19 ? 00:00:00 [scsi_eh_25]
root 439 2 0 Nov19 ? 00:00:00 [scsi_eh_26]
root 440 2 0 Nov19 ? 00:00:00 [scsi_eh_27]
root 441 2 0 Nov19 ? 00:00:00 [scsi_eh_28]
root 442 2 0 Nov19 ? 00:00:00 [scsi_eh_29]
root 443 2 0 Nov19 ? 00:00:00 [scsi_eh_30]
root 444 2 0 Nov19 ? 00:00:00 [scsi_eh_31]
root 445 2 0 Nov19 ? 00:00:00 [scsi_eh_32]
root 527 2 0 Nov19 ? 00:00:00 [kdmflush]
root 529 2 0 Nov19 ? 00:00:00 [kdmflush]
root 547 2 0 Nov19 ? 00:00:00 [jbd2/dm-0-8]
root 548 2 0 Nov19 ? 00:00:00 [ext4-dio-unwrit]
root 625 1 0 Nov19 ? 00:00:00 /sbin/udevd -d
root 816 2 0 Nov19 ? 00:00:14 [vmmemctl]
root 962 2 0 Nov19 ? 00:00:00 [kdmflush]
root 968 2 0 Nov19 ? 00:00:00 [kdmflush]
root 972 2 0 Nov19 ? 00:00:00 [kdmflush]
root 974 2 0 Nov19 ? 00:00:00 [kdmflush]
root 975 2 0 Nov19 ? 00:00:00 [kdmflush]
root 976 625 0 Nov19 ? 00:00:00 /sbin/udevd -d
root 978 2 0 Nov19 ? 00:00:00 [kdmflush]
root 982 625 0 Nov19 ? 00:00:00 /sbin/udevd -d
root 1057 2 0 Nov19 ? 00:00:00 [jbd2/sda1-8]
root 1058 2 0 Nov19 ? 00:00:00 [ext4-dio-unwrit]
root 1059 2 0 Nov19 ? 00:00:00 [jbd2/dm-2-8]
root 1060 2 0 Nov19 ? 00:00:00 [ext4-dio-unwrit]
root 1061 2 0 Nov19 ? 00:00:00 [jbd2/dm-6-8]
root 1062 2 0 Nov19 ? 00:00:00 [ext4-dio-unwrit]
root 1063 2 0 Nov19 ? 00:00:15 [jbd2/dm-7-8]
root 1064 2 0 Nov19 ? 00:00:00 [ext4-dio-unwrit]
root 1065 2 0 Nov19 ? 00:00:08 [jbd2/dm-3-8]
root 1066 2 0 Nov19 ? 00:00:00 [ext4-dio-unwrit]
root 1067 2 0 Nov19 ? 00:01:54 [jbd2/dm-4-8]
root 1068 2 0 Nov19 ? 00:00:00 [ext4-dio-unwrit]
root 1069 2 0 Nov19 ? 00:02:21 [jbd2/dm-5-8]
root 1070 2 0 Nov19 ? 00:00:00 [ext4-dio-unwrit]
root 1158 2 0 Nov19 ? 00:00:36 [kauditd]
root 1403 1 0 Nov19 ? 00:12:11 /usr/sbin/vmtoolsd
root 1440 2 0 Nov19 ? 00:00:14 [flush-253:3]
root 1441 2 0 Nov19 ? 00:04:02 [flush-253:4]
root 1442 2 0 Nov19 ? 00:00:15 [flush-253:5]
root 1444 2 0 Nov19 ? 00:00:05 [flush-253:7]
root 1504 1 0 Nov19 ? 00:01:33 auditd
nslcd 1559 1 0 Nov19 ? 00:00:00 /usr/sbin/nslcd
root 1575 1 0 Nov19 ? 00:00:53 /sbin/rsyslogd -i /var/run/syslogd.pid -c 5
rpc 1606 1 0 Nov19 ? 00:00:01 rpcbind
root 1628 2 0 Nov19 ? 00:00:00 [rpciod/0]
root 1629 2 0 Nov19 ? 00:00:00 [rpciod/1]
root 1630 2 0 Nov19 ? 00:00:00 [rpciod/2]
root 1631 2 0 Nov19 ? 00:00:00 [rpciod/3]
root 1632 2 0 Nov19 ? 00:00:00 [rpciod/4]
root 1633 2 0 Nov19 ? 00:00:00 [rpciod/5]
rpcuser 1640 1 0 Nov19 ? 00:00:00 rpc.statd -p 50001
dbus 1667 1 0 Nov19 ? 00:00:00 dbus-daemon --system
root 1688 1 0 Nov19 ? 00:00:00 cupsd -C /etc/cups/cupsd.conf
root 1757 1 0 Nov19 ? 00:00:00 rpc.mountd -p 50000
root 1763 2 0 Nov19 ? 00:00:00 [lockd]
root 1764 2 0 Nov19 ? 00:00:00 [nfsd4]
root 1765 2 0 Nov19 ? 00:00:00 [nfsd4_callbacks]
root 1766 2 0 Nov19 ? 00:00:00 [nfsd]
root 1767 2 0 Nov19 ? 00:00:00 [nfsd]
root 1768 2 0 Nov19 ? 00:00:00 [nfsd]
root 1769 2 0 Nov19 ? 00:00:00 [nfsd]
root 1770 2 0 Nov19 ? 00:00:00 [nfsd]
root 1771 2 0 Nov19 ? 00:00:00 [nfsd]
root 1772 2 0 Nov19 ? 00:00:00 [nfsd]
root 1773 2 0 Nov19 ? 00:00:00 [nfsd]
root 1802 1 0 Nov19 ? 00:00:00 rpc.idmapd
root 1820 1 0 Nov19 ? 00:00:00 /usr/sbin/sshd
ntp 1861 1 0 Nov19 ? 00:00:01 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
root 1932 1 0 Nov19 ? 00:00:23 sendmail: accepting connections
smmsp 1941 1 0 Nov19 ? 00:00:00 sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue
root 1953 1 0 Nov19 ? 00:00:31 /usr/sbin/httpd
root 1970 1 0 Nov19 ? 00:05:43 /opt/numara-software/footprints-asset-core/client/bin/mtxagent
root 1988 1 0 Nov19 ? 00:00:15 crond
root 2079 1 0 Nov19 ? 00:00:00 /usr/sbin/atd
root 2090 1 0 Nov19 ? 00:00:30 /usr/sbin/nsrexecd
root 2144 1 0 Nov19 ? 00:23:09 python /usr/bin/goferd
root 2178 1 0 Nov19 ? 00:00:00 /usr/bin/rhsmcertd
root 2204 1 3 Nov19 ? 08:14:39 /opt/BESClient/bin/BESClient
root 2224 1 0 Nov19 tty1 00:00:00 /sbin/mingetty /dev/tty1
root 2226 1 0 Nov19 tty2 00:00:00 /sbin/mingetty /dev/tty2
root 2228 1 0 Nov19 tty3 00:00:00 /sbin/mingetty /dev/tty3
root 2230 1 0 Nov19 tty4 00:00:00 /sbin/mingetty /dev/tty4
root 2232 1 0 Nov19 tty5 00:00:00 /sbin/mingetty /dev/tty5
root 2234 1 0 Nov19 tty6 00:00:00 /sbin/mingetty /dev/tty6
apache 4098 1953 0 12:40 ? 00:00:07 /usr/sbin/httpd
root 4177 1820 0 12:40 ? 00:00:00 sshd: kar54 [priv]
kar54 4239 4177 0 12:41 ? 00:00:00 sshd: kar54@pts/0
kar54 4240 4239 0 12:41 pts/0 00:00:00 -bash
root 4262 4240 0 12:41 pts/0 00:00:00 sudo su - nagios
root 4264 4262 0 12:41 pts/0 00:00:00 su - nagios
nagios 4277 4264 0 12:41 pts/0 00:00:00 -bash
nagios 5233 1 11 12:46 ? 00:24:34 /usr/bin/java -Xms18074m -Xmx18074m -Djava.awt.headless=true -XX:+UseParN
root 9084 1820 0 13:18 ? 00:00:00 sshd: murug001 [priv]
murug001 9119 9084 0 13:18 ? 00:00:00 sshd: murug001@pts/1
murug001 9120 9119 0 13:18 pts/1 00:00:00 -ksh
root 9144 9120 0 13:18 pts/1 00:00:00 sudo su - nagios
root 9160 9144 0 13:19 pts/1 00:00:00 su - nagios
nagios 9172 9160 0 13:19 pts/1 00:00:00 -bash
nagios 12138 9172 0 13:42 pts/1 00:00:00 view logstash.log
root 17029 1932 0 14:22 ? 00:00:01 sendmail: ./uAO2MEP1005787 from queue
apache 20174 1953 0 Nov27 ? 00:00:09 /usr/sbin/httpd
apache 20175 1953 0 Nov27 ? 00:00:09 /usr/sbin/httpd
apache 20176 1953 0 Nov27 ? 00:00:07 /usr/sbin/httpd
apache 20177 1953 0 Nov27 ? 00:00:08 /usr/sbin/httpd
apache 20178 1953 0 Nov27 ? 00:00:09 /usr/sbin/httpd
apache 20179 1953 0 Nov27 ? 00:00:07 /usr/sbin/httpd
apache 20180 1953 0 Nov27 ? 00:00:08 /usr/sbin/httpd
apache 20181 1953 0 Nov27 ? 00:00:08 /usr/sbin/httpd
root 23812 1932 0 15:22 ? 00:00:00 sendmail: ./uAQCMEiB027280 from queue
root 30415 1932 0 16:22 ? 00:00:00 sendmail: ./uASCMEJT012835 from queue
root 31062 1988 0 16:28 ? 00:00:00 CROND
root 31063 1988 0 16:28 ? 00:00:00 CROND
nagios 31065 31062 0 16:28 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jo
nagios 31066 31063 0 16:28 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php po
nagios 31069 31066 0 16:28 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
nagios 31071 31065 0 16:28 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
root 31159 31069 0 16:28 ? 00:00:00 sudo /etc/init.d/logstash status
root 31160 2 0 16:28 ? 00:00:00 [flush-253:6]
nagios 31162 4277 0 16:28 pts/0 00:00:00 ps -aef
apache 32201 1953 0 Nov27 ? 00:00:08 /usr/sbin/httpd
apache 32262 1953 0 Nov27 ? 00:00:08 /usr/sbin/httpd
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg-root 976M 232M 693M 26% /
tmpfs 18G 0 18G 0% /dev/shm
/dev/sda1 93M 29M 60M 33% /boot
/dev/mapper/vgbu-bu 148G 7.5G 133G 6% /backups
/dev/mapper/vg-home 976M 116M 810M 13% /home
/dev/mapper/vg-opt 2.0G 868M 982M 47% /opt
/dev/mapper/vg-tmp 976M 9.1M 916M 1% /tmp
/dev/mapper/vg-usr 692G 343G 314G 53% /usr
/dev/mapper/vg-var 40G 986M 37G 3% /var
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/vg-root 64K 6.2K 58K 10% /
tmpfs 4.5M 1 4.5M 1% /dev/shm
/dev/sda1 26K 38 26K 1% /boot
/dev/mapper/vgbu-bu 9.4M 367 9.4M 1% /backups
/dev/mapper/vg-home 64K 126 64K 1% /home
/dev/mapper/vg-opt 128K 1.9K 127K 2% /opt
/dev/mapper/vg-tmp 64K 37 64K 1% /tmp
/dev/mapper/vg-usr 44M 87K 44M 1% /usr
/dev/mapper/vg-var 2.5M 9.9K 2.5M 1% /var
[nagios@nagilgp01 logstash]$
You do not have the required permissions to view the files attached to this post.
-
krobertson71
- Posts: 444
- Joined: Tue Feb 11, 2014 10:16 pm
Re: NLS Out of Memory Error
You do not have the required permissions to view the files attached to this post.
Re: NLS Out of Memory Error
What's your average search time period when running searches/queries? If you're looking back more than 14 or so days, I can see where elasticsearch might get exhausted in this environment.
This might also be relating to the number of writes happening to your elasticsearch database exhausting the available memory. Is increasing the amount of memory these machines have available an option? We don't usually recommend allocating more than 64GB per machine due to limitations within the Java virtual machine.
This might also be relating to the number of writes happening to your elasticsearch database exhausting the available memory. Is increasing the amount of memory these machines have available an option? We don't usually recommend allocating more than 64GB per machine due to limitations within the Java virtual machine.
Former Nagios employee
https://www.mcapra.com/
https://www.mcapra.com/
-
krobertson71
- Posts: 444
- Joined: Tue Feb 11, 2014 10:16 pm
Re: NLS Out of Memory Error
Sorry for the late reply. This can be closed. This turned out to be a RSYSLOG configuration issue that was pushed out to our Linux environment. We are good now.
Thanks..
You can close this.
Thanks..
You can close this.