Basic Clustering & Repository Question

[neil_davidson]

I have a few very basic questions regarding Log Server clustering and repository configuration.

It seems that when a cluster is created, all members are treated equally and no single member is the 'master' unit. Assuming that is the case, is it best practice to send logs to the cluster member closest to the log source? If so, is there a convenient way to keep track of what log sources are sending logs to which members? Or should we send all logs to one particular member? In which case, what happens when that member goes down?

Next, if Log Servers are located remotely from each other, what firewall ports would need to be opened to allow them to communicate?

With regards to the repository, the documentation example shows '/tmp' as the location, which presumably is the '/tmp' directory on the particular member where the configuration is being done. And it is necessary for all cluster members to have access to that directory, so they are all sending information to that specific directory (i.e. not the '/tmp' directory on their own local hosts). But if I want to store the repository on a remote location, say a CIFS file share, what would the syntax be for entering that location and what would I need to know to allow the Log Servers to access that directory?

Any assistance is greatly appreciated!

[Box293]

It seems that when a cluster is created, all members are treated equally and no single member is the 'master' unit. Assuming that is the case, is it best practice to send logs to the cluster member closest to the log source? If so, is there a convenient way to keep track of what log sources are sending logs to which members? Or should we send all logs to one particular member? In which case, what happens when that member goes down?

The cluster determines who the master is, generally this doesn't change unless the master reboots and another member gets promoted to be the master.

You can create a dashboard with a query that showed what cluster members are receiving logs from what devices.

Next, if Log Servers are located remotely from each other, what firewall ports would need to be opened to allow them to communicate?

If they are remote then you need to really have a 1GB link between the remote locations, a lot of data is sent around to all the members. Port 9200 is the default port used:
https://www.elastic.co/guide/en/elastic ... earch.html

With regards to the repository, the documentation example shows '/tmp' as the location, which presumably is the '/tmp' directory on the particular member where the configuration is being done. And it is necessary for all cluster members to have access to that directory, so they are all sending information to that specific directory (i.e. not the '/tmp' directory on their own local hosts). But if I want to store the repository on a remote location, say a CIFS file share, what would the syntax be for entering that location and what would I need to know to allow the Log Servers to access that directory?

I assume you are talking about Backup & Maintenance.
When you go to create a repository there is a ? next to the "Repository Location" field which says:

This location MUST be a shared filesystem accessible to all data instances in the cluster or either backups or restoration can fail

These links may help:
https://support.nagios.com/kb/article.php?id=303
https://support.nagios.com/kb/article.php?id=494

[neil_davidson]

Thanks very much for the responses. I'm afraid I need my hand held a little:

The cluster determines who the master is, generally this doesn't change unless the master reboots and another member gets promoted to be the master.

Is that just with regards to the Elasticsearch, which I believe needs a master? Is there any other way that the master is special? Just curious.

You can create a dashboard with a query that showed what cluster members are receiving logs from what devices.

Hmm. I see how I can go to the 'host' field and create a table using the 'Terms' drop-down that shows the top 10 hosts along with the hit counts, but I don't see how to add the corresponding cluster member to the table. Or am I going about it the wrong way?

If they are remote then you need to really have a 1GB link between the remote locations, a lot of data is sent around to all the members. Port 9200 is the default port used:

Then I think we'll try not to do that. Thanks.

I assume you are talking about Backup & Maintenance.
When you go to create a repository there is a ? next to the "Repository Location" field which says:

This location MUST be a shared filesystem accessible to all data instances in the cluster or either backups or restoration can fail

These links may help:
https://support.nagios.com/kb/article.php?id=303
https://support.nagios.com/kb/article.php?id=494

Thanks yes, I'd seen all that. I'm looking for more specific configuration details. Say, if I want to store the repository in a CIFS (Windows) File Share at 10.10.10.10 in the "\Backup\Nagios" directory, would I enter "\\10.10.10.10\Backup\Nagios" in the location field or is there some other syntax? And assuming I need to set up a 'nagios' user on that system to allow it write access, is there a default password?

Again, your help is much appreciated.

Cheers.

[Box293]

Is that just with regards to the Elasticsearch, which I believe needs a master? Is there any other way that the master is special? Just curious.

Yes it's just Elasticsearch related. Here's some more detailed information about nodes and their roles:

https://www.elastic.co/guide/en/elastic ... -node.html

Hmm. I see how I can go to the 'host' field and create a table using the 'Terms' drop-down that shows the top 10 hosts along with the hit counts, but I don't see how to add the corresponding cluster member to the table. Or am I going about it the wrong way?

I'm going to get the USA techs to follow up on this.

Thanks yes, I'd seen all that. I'm looking for more specific configuration details. Say, if I want to store the repository in a CIFS (Windows) File Share at 10.10.10.10 in the "\Backup\Nagios" directory, would I enter "\\10.10.10.10\Backup\Nagios" in the location field or is there some other syntax? And assuming I need to set up a 'nagios' user on that system to allow it write access, is there a default password?

It is how it is mounted, the same way on all servers. For Example:
/mnt/nagios_log_server_common_backups

I have a central NFS server that both log server instances have mounted via an entry in their /etc/fstab file:

Code: Select all

10.25.11.11:/mnt/nfs_disk_01/nagios_log_server_common_backups /mnt/nagios_log_server_common_backups nfs defaults 0 0

This ensures it's mounted on boot up.
In your instance you would reference it as /Backup/Nagios once you configured your CFS client to mount it. The Nagios user will require write access.

[neil_davidson]

Ah, got it now. Thanks very much!

Will wait for the USA update on the cluster member thing.

Cheers.

[hsmith]

I'm not entirely sure that there's a field created indicating which cluster member received which log. I might be misunderstanding this, but there's not a great way I can think of to do this. Something you COULD do is make some new inputs, only send the logs from a certain device to the port of that input on a certain IP, and have it be tagged as a certain type. That is super roundabout, and probably not the best way to do it, but I cannot think of another.

[neil_davidson]

I'm not entirely sure that there's a field created indicating which cluster member received which log. I might be misunderstanding this, but there's not a great way I can think of to do this. Something you COULD do is make some new inputs, only send the logs from a certain device to the port of that input on a certain IP, and have it be tagged as a certain type. That is super roundabout, and probably not the best way to do it, but I cannot think of another.

Ok, thanks. It's probably not important, just a thought that occurred. Circling back to what caused that thought, any comments on whether it's best to configure all sources to send logs to one particular cluster member, or distribute it around. If it isn't easy to keep track of which is sending logs where, it might be best to send everything to one, but then all the eggs are in that one basket as it were.

[hsmith]

It depends. I see a lot of load balanced setups that send the logs to all of the servers based on load, but I don't think it's required. I've seen a single instance take many thousands of logs per second and handle it. It might be best to have them distributed between the servers to eliminate a single point of failure, but it depends on your resources. Most setups I see have all the longs coming to one server.

[neil_davidson]

OK, thanks for the feedback. I think all my questions have been answered. Cheers!

[mcapra]

Is it alright if we lock this thread and mark the issue as resolved?