SWAP CRITICAL - 0% free (0 MB out of 255 MB)

[WillemDH]

Hello,

This weekend, it seems I ran out of free swap space on one of my NLS instances. So I was planning to exapnd the lv_swap volume, but when I execute a lvdisplay or pvdisplay, I get:

Code: Select all

 lvdisplay
-bash: lvdisplay: command not found

Is it normal I can't execute lv commands? I was planning to follow http://assets.nagios.com/downloads/nagi ... M_Disk.pdf to expand the swap volume. Can I safely follow this guide?

Grtz

Willem

[cmerchant]

Wondering if your NLS was setup with LVM, could you do the following:

Code: Select all

lsblk

and can you show us the output from this command:

Code: Select all

which lvm

[WillemDH]

Hey CMerchant,

I'm using the 'official' Nagios Log Server ovf templates.
This is the output:

Code: Select all

lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sr0     11:0    1 1024M  0 rom
sda      8:0    0  100G  0 disk
└─sda1   8:1    0  100G  0 part /

Code: Select all

 which lvm
/usr/bin/which: no lvm in (/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin)

Grtz

Willem

[cmerchant]

LVM was not used to create the NLS VM image. It looks like the swap is not allocated to a seperate partition, and is likely a special swap file.

Could you show us the output of the following:

Code: Select all

df -h
free
swapon -s

If the swap is allocated as a special file instead of a partition,
you can disable the swap file,

Code: Select all

swapoff /swapfile

delete the current swap file,

Code: Select all

rm -rf /swapfile

reallocate the swap file or create a separate swap partition - a 2GB swapfile

Code: Select all

dd if=/dev/zero bs=1M of=/swapfile count=2000

and then activate the swap file:

Code: Select all

swapon /swapfile
free

If you keep the same swap filename as before, you will not need to modify the swap mount in the /etc/fstab.

My personal preference would be to create separate swap partition, but it would be a little more involved depending what your VM options are for adding an additional disk, or expanding the existing VM image file (without LVM, and adding a partition via fdisk onto /dev/sda).

[WillemDH]

Output:

Code: Select all

df -h
Filesystem      Size  Used Avail Use% Mounted on
rootfs           99G   30G   68G  31% /
devtmpfs        995M  152K  995M   1% /dev
tmpfs          1004M     0 1004M   0% /dev/shm
/dev/sda1        99G   30G   68G  31% /

Code: Select all

free
             total       used       free     shared    buffers     cached
Mem:       2054404    1952168     102236          8      13776     301240
-/+ buffers/cache:    1637152     417252
Swap:       262136      78172     183964

Code: Select all

swapon -s
Filename                                Type            Size    Used    Priority
/swapfile1                              file            262136  78160   -1

As the name of the swapfile appears to be swapfile1 I did as you suggested:

Code: Select all

swapoff /swapfile
swapoff: /swapfile: swapoff failed: No such file or directory
[root@nls01 ~]# swapoff /swapfile1
[root@nls01 ~]# rm -rf /swapfile1
[root@nls01 ~]# dd if=/dev/zero bs=1M of=/swapfile1 count=2000
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB) copied, 36.0223 s, 58.2 MB/s

Result:

Code: Select all

[root@nls01 ~]# swapon -s
Filename                                Type            Size    Used    Priority
[root@nls01 ~]# free
             total       used       free     shared    buffers     cached
Mem:       2054404    1981868      72536        248       7824     274500
-/+ buffers/cache:    1699544     354860
Swap:            0          0          0
[root@nls01 ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
rootfs           99G   32G   67G  33% /
devtmpfs        995M  152K  995M   1% /dev
tmpfs          1004M     0 1004M   0% /dev/shm
/dev/sda1        99G   32G   67G  33% /

Code: Select all

swapon /swapfile1
swapon: /swapfile1: read swap header failed: Invalid argument

Now I get the following error in Nagios XI for the host nls01:

SWAP CRITICAL - 0% free (0 MB out of 0 MB) - Swap is either disabled, not present, or of zero size.

Grtz

[cmerchant]

I missed a step:

Code: Select all

mkswap /swapfile1
swapon /swapfile1

sorry.

[WillemDH]

No problem cmerchant.
It worked with your missing commands. I guess for now I won't make a new partition. I have no idea how to make one anyway without lvm. I'll monitor the swap usage the coming weeks.Please leave the thread open for a week or so, I'll let you know if everything works well next week.

Grtz and tx

WIllem

[cmerchant]

Ok, I'll leave this open. I suspect there are others who might have the same issue. The VM image for NLS that is available for download from our site has the same small swapfile setup, including having no LVM.

[WillemDH]

Well, I just read through this document: http://www.elasticsearch.org/guide/en/e ... izing.html

Especially the part:

swapping is the death of performanceedit

It should be obvious, but it bears spelling out clearly: swapping main memory to disk will crush server performance. Think about it: an in-memory operation is one that needs to execute quickly.

If memory swaps to disk, a 100-microsecond operation becomes one that take 10 milliseconds. Now repeat that increase in latency for all other 10us operations. It isn’t difficult to see why swapping is terrible for performance.

The best thing to do is disable swap completely on your system. This can be done temporarily:

sudo swapoff -a
To disable it permanently, you’ll likely need to edit your /etc/fstab. Consult the documentation for your OS.

If disabling swap completely is not an option, you can try to lower swappiness. This value controls how aggressively the OS tries to swap memory. This prevents swapping under normal circumstances, but still allows the OS to swap under emergency memory situations.

For most Linux systems, this is configured using the sysctl value:

vm.swappiness = 1


A swappiness of 1 is better than 0, since on some kernel versions a swappiness of 0 can invoke the OOM-killer.

Finally, if neither approach is possible, you should enable mlockall. file. This allows the JVM to lock its memory and prevent it from being swapped by the OS. In your elasticsearch.yml, set this:

bootstrap.mlockall: true

Would htis document apply to NLS? Should I follow the directions about swapping and try setting vm.swappiness to 1?

[cmerchant]

Agreed, swapping is a bad thing (tm), but running out of virtual memory (committed) is even worse.
That is why you had the out of space error in the first place. (too small a swap space).

From your output of the free cmd it appears that you have 2GB RAM configured.

Good recommendation for setting vm.swappiness =1

Here is a good link to change it:

http://en.wikipedia.org/wiki/Swappiness