Nagios Log Server on RHEL 6

[krobertson71]

Our Linux admins break out our RHEL6 installs into separate partitions like so:

/
/boot
/var
/opt
/usr
/tmp

etc

What I need to know is where Log Server keeps most of it's data so I can tell them which partitions need to be what size. I understand the size depends on what I am collecting, I am just asking which partition(s) should I be putting the "bulk" of my capacity in. Or, if you have a recommendation on each of these then that would be helpful as well.

[slansing]

If I am reading what you are asking correctly, it should be "/usr", more specifically, your indicies, and cluster data are stored in the elasticsearch directory tree at:

Code: Select all

/usr/local/nagioslogserver/elasticsearch/data/<CLUSTER_ID>/nodes/<NODE_#>/

In addition, /var will grow quite a bit with both the current logstash.log, and it's tarred archives in:

Code: Select all

/var/log/logstash

[krobertson71]

Yeah that is what I was looking for. So Logstash doesn't need a big store as well? And log files... are they going to var or within /usr as well?

[slansing]

That would be another good one to provide some additional space on, more specifically:

Code: Select all

/var/log/logstash

As you had mentioned, that contains your current logstash.log, as well as the tarred archives. Apologies for not mentioning that in my original post, I'm hunting around with:

Code: Select all

du -m --max-depth 1 | sort -rn | head -11

There is also an elasticsearch directory in /var which houses cluster snippets, I need to do more looking into the elasticsearch mechanics we run on logserver to say what is actually being moved or copied out, and what is not.

[krobertson71]

Cool keep my posted. I will do my research on this end.

[cmerchant]

Good info to know regarding file locations and space allocation. I suspect that this will be a good FAQ or doc to be added to the standard NLS documentation. Thanks.

[krobertson71]

Yeah I need more info I believe..

So lets say I will be collecting 100 gigs of log data...

How big should /var be considering logstash...

How big should /usr be considering elasticsearch and cluster/indexes etc are located..

[krobertson71]

actually I think most everything is in /usr

Selection_031.png

This is from my laptop running logserver in VirtualBox.

Here is /var compared to /usr. To me it looks like most of my space allocation needs to go to /usr. Tell me if I am wrong about this.

Selection_032.png

[sreinhardt]

I completely agree with krobertson. /var may raise somewhat in size, but the vast vast majority of files should be in /usr/local/nagioslogserver/elasticsearch/data as slansing correctly pointed out originally. Especially if logrotate is used with the logstash log, I would not expect it or it's rotations to grow too out of control.

[bdgoecke]

For point of reference, on my test log server I have 3 lightly used linux machines loging to the server for a month or so..

_Cluster_Statistics_

Code: Select all

1,357,738 Documents
287MB Primary Size
287MB Total Size
1 Data Instances
352 Total Shards
36 Indices

_Disk_Usage_
[@LogServer-centos65-64bit /]$ sudo du -sh *

Code: Select all

5.8M    bin
88M     boot
220K    dev
29M     etc
488M    home
448M    lib
22M     lib64
16K     lost+found
4.0K    media
4.0K    mnt
4.0K    opt
179M    root
14M     sbin
4.0K    selinux
4.0K    srv
12K     store
0       sys
420K    tmp
1.4G    usr
346M    var

[@LogServer-centos65-64bit /]$ cd /var
[@LogServer-centos65-64bit var]$ sudo du -sh *

Code: Select all

53M     cache
24K     db
8.0K    empty
4.0K    games
51M     lib
4.0K    local
16K     lock
226M    log
0       mail
4.0K    nis
4.0K    opt
4.0K    preserve
104K    run
112K    spool
4.0K    tmp
18M     www
4.0K    yp

[@LogServer-centos65-64bit var]$ cd /usr
[@LogServer-centos65-64bit usr]$ sudo du -sh *

Code: Select all

30M     bin
4.0K    etc
4.0K    games
17M     include
232M    lib
109M    lib64
17M     libexec
695M    local
29M     sbin
292M    share
12K     src
0       tmp