Isilon Storage for a 2 node cluster

[gregwhite]

We are building a new 2 node log server that each node will have 2TB of NFS mounted Isilon storage. The Storage administrator is asking if he can provide one NFS export of 4TB, with each server having access or does he need to provide an NFS export of 2TB for each server?
He goes on to ask if he can also have our 2 Nagios test servers use the same NFS Export and increase the size to 5TB. I'm not sure if test data should live on the same export as production but thought I would ask if it is at least possible.

Thanks,
Greg

[gsmith]

Hi Greg

You are going to need separate mounts for each server. So you would want something
like:
Prod1 - 2TB
Prod2 - 2 TB
Test1 - 1 TB
Test2 - 1 TB

That way if one server or one mount point goes down you are still up and running.

Thanks

[gregwhite]

I followed up with our Storage Administrator. In the case of the log server, the scenario he was suggesting was creating a 4tb nfs export. Both servers in the log server cluster can read/write to all 4TB. As I understand your original response we need to create 2 - 2tb exports. One for each node. Is this correct?

I'm not sure if having one export for the test servers would work or not. Should we have a separate export for each of those also? One test server is for NagiosXI and one is for the log server.

Thanks,
Greg

[gsmith]

Hi

Each Production log server should have it's own NFS mount. So, yes, create two 2TB exports, and have each node mount one of
the exports. This will give you more flexibility and reliability then sharing one common mount of 4GB. If in the future you
need to move to a new storage device you would be able to move one node at a time and keep Production running.

As far as the test servers are concerned you could go either way. The logserver is going to use the lion's share of the
storage space, while the XI server will use far less.

xi requirements: https://assets.nagios.com/downloads/nag ... ements.pdf

Again, having separate mounts for the test servers would give you the maximum flexibility. If you wanted to clone the
XI server you could take it's mount offline and copy it while leaving the test logserver alone. Does the storage admin
give any reason for wanting minimal mount points?

Thanks

[gregwhite]

Will the log server application work with each node accessing the same single export of 4tbs? The redundancy would be there if we lost a node. Yes the single point of failure would be the storage. However, if our organization can withstand an outage while we restore the data, the single export of 4tb would give us more online data. Do the two nodes act like a traditional cluster and can read/write to the same data store or are they a two node cluster that is aware of each other but needs to have it's own data store. I.E. the log server application cannot work with 2 nodes accessing a single export.

Thanks,
Greg

[gsmith]

Hi Greg

if that 4 tb nfs mount has 2 directories at the top level, say /server1 and /server2 , and machine1 mounts /server1 while machine2 mounts /server2 you would be OK.

However if that 4 tb nfs mount has 1 directory at the top level, say /nls_store , and machine1 and machine2 both mount
/nls_store that would be unsupported.

Thanks

[gregwhite]

Thanks, I think that clears up our confusion. Our manager was looking at this as a traditional cluster and was concerned that data usage would grow exponentially because of the configuration. If I am understanding it, the primary node is in charge and the secondary node provides added performance under the direction of the primary node. Each node has a copy of the data to do searches and it provides redundancy.
The size of the data is managed by how much online data we maintain before archiving it. If we needed to add a 3rd node in the future would it require another 2tb?

Thanks again,
Greg

[gsmith]

HI,

You're close, but not quite right. Everyday there are 5 shards (pieces) created to hold the data. In a 2 node system
there will be 5 primary shards and 5 secondary shards every day. So server1 might have:
P1
S2
P3
S4
P5

and server 2 would have
S1
P2
S3
P4
S5

There is also a "retention" setting, so you can set the cluster to keep only XX days worth of data. Once you start
collecting data you will get a feel for how much storage per day you need. Then you can do the cost benefit analysis of
how far back in time you want to go vs. how much diskspace you will need.

This probably explains it a lot better than I am doing :
https://www.youtube.com/watch?v=wORXARN0TA0

It's a little over half an hour, but well worth it.

Let me know what you think.

Thanks

[gregwhite]

Thanks!! Watched the video and found it very helpful, very informative. This does not take away from all your help.

Thanks for all your help and for finding this link!!
Greg

[gsmith]

Glad it made things clearer for you.

I am going to close out (lock) this thread. But feel free to reach out as more
questions arise.

Have a good weekend.