NLS Stops Intermittently

34Bearman · Post by **34Bearman** » Mon Mar 02, 2015 4:08 pm

Once in a while NLS will stop responding and won't respond to logon requests either. Usually restarting the elasticsearch service will resolve the issue. However today when I tried restarting the service I got the following message:

[root@nagiosls ~]# service elasticsearch restart
Stopping elasticsearch: [ OK ]
Starting elasticsearch: [ OK ]
[root@nagiosls ~]# OpenJDK 64-Bit Server VM warning: Insufficient space for shared memory file:
11743
Try using the -Djava.io.tmpdir= option to select an alternate temp location.

{1.3.2}: pid Failed ...
- IOException[No space left on device]
java.io.IOException: No space left on device
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:325)
at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:157)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)

Intel Quad core 3.2Ghz
8GB RAM

Any suggestions?

34Bearman · Post by **34Bearman** » Mon Mar 02, 2015 4:16 pm

Ok, so I should have looked deeper prior to the post. The issue is I've ran out of disk space. I thought I had the Backup settings configured properly but I guess I don't.

What are the recommendations for freeing up disk space?

Thanks.

jolson · Post by **jolson** » Mon Mar 02, 2015 4:34 pm

Please access the page at "Administration > Backup & Maintenance" and review the options there. Ensure that you have an off-site backup repository so that your backups aren't using local disk space. The most relevant setting on this page for you will be 'Delete indexes older than *', which you can set to a value of your choice. After setting that value, click 'Command Subsystem' and select "Edit" next to "backup_maintenance" - make the next run time 'Now' and NLS will begin the process of removing your old indexes.

34Bearman · Post by **34Bearman** » Mon Mar 02, 2015 5:50 pm

Ok. Due to insufficient disk space I can't get Elasticsearch to stay running thus not allowing me to logon to the Web Interface. What I can delete or remove safely to get some disk space?

ssax · Post by **ssax** » Mon Mar 02, 2015 6:08 pm

Please post the output of the command below.

Code: Select all

ls -lhrt /store/backups/nagioslogserver

34Bearman · Post by **34Bearman** » Tue Mar 03, 2015 9:32 am

[root@nagiosls ~]# ls -lhrt /store/backups/nagioslogserver
total 0

cmerchant · Post by **cmerchant** » Tue Mar 03, 2015 10:03 am

OpenJDK 64-Bit Server VM warning: Insufficient space for shared memory file:

It appears that you have run out of space in the tmpfs share memory filesystem. Could you show us the output of the following commands:

Code: Select all

df -h
cat /etc/fstab

34Bearman · Post by **34Bearman** » Tue Mar 03, 2015 10:15 am

[root@nagiosls ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_nagiosls-lv_root
50G 47G 13M 100% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
/dev/sda1 477M 68M 384M 15% /boot
/dev/mapper/vg_nagiosls-lv_home
401G 71M 381G 1% /home
[root@nagiosls ~]# cat /etc/fstab

#
# /etc/fstab
# Created by anaconda on Mon Feb 2 14:34:57 2015
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/vg_nagiosls-lv_root / ext4 defaults 1 1
UUID=ed86a6aa-c3ef-4c1c-a541-4b6c3d46edb7 /boot ext4 defaults 1 2
/dev/mapper/vg_nagiosls-lv_home /home ext4 defaults 1 2
/dev/mapper/vg_nagiosls-lv_swap swap swap defaults 0 0
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0

jolson · Post by **jolson** » Tue Mar 03, 2015 10:36 am

34Bearman wrote: 401G 71M 381G 1% /home

It looks like there is a lot of free space in your home volume. A good solution would be to resize your volumes so that /dev/mapper/vg_nagiosls-lv_home is reduced by ~300GB, and /dev/mapper/vg_nagiosls-lv_root is increased by the same amount.

To see your logical volumes, you can run:

Code: Select all

lvscan

Please be aware that decreasing the size of a volume can cause data loss. Be sure to have proper backups of all files in the home directory before decreasing its size. The root directory should have no problem expanding when done properly.

Be aware that you should shrink the filesystem size of the home directory before you reduce the volume size.

To increase the size of a logical volume (your root directory), follow these instructions:
https://access.redhat.com/documentation ... xtend.html

To reduce the size of a logical volume (your home directory), follow these instructions:
https://access.redhat.com/documentation ... educe.html

Example commands-
Unmount the home directory:

Code: Select all