Page 1 of 2

Initial Setup of Layered Network

Posted: Thu Oct 08, 2015 12:33 pm
by snidley
Consider the attached PDF of our environment.

The server, nagios-001, must be connected to Net1. It is the collection point for our Nagios web server. The rules of this environment do not permit direct routing to/from Net1 and Net2. In this environment, I am able to monitor login-001, management-001, ganglia-001 and nagios-001 and display status on the web interface without any trouble.

Ganglia-001 collects and displays data for our Ganglia implementation. When polled, the nodes, compute-001 through compute-005 send their data to management-001 for Ganglia. So does login-001. And management-001 forwards the data all on to ganglia-001 when polled. I want to do the same sort of thing with Nagios so that it can monitor compute-001 through compute-005 without any routing to/from Net1 and Net2.

How do I do it? It seems that any implementation instructions I've seen assume there is at least a route available between all nodes. How do I monitor those nodes essentially "hidden" from Net1? What packages and/or plugins are required? And which ones cost money for product or support?

Re: Initial Setup of Layered Network

Posted: Thu Oct 08, 2015 12:38 pm
by tmcdonald
I do not see an attached PDF. Make sure that when you choose the file, you click "Attach" next to it.

Re: Initial Setup of Layered Network

Posted: Thu Oct 08, 2015 2:41 pm
by snidley
I was sure I attached it in the original. I'll try to also include a GIF.

Re: Initial Setup of Layered Network

Posted: Thu Oct 08, 2015 4:13 pm
by jdalrymple
It's unclear to me how you plan to get traffic from one network to another. The restrictions on your network aren't well defined. You mentioned things like:
snidley wrote:It is the collection point for our Nagios web server.
and
snidley wrote:Ganglia-001 collects and displays data for our Ganglia implementation.
Those statements are difficult to interpret though. Here is some documentation that might help you understand options, but we can be more detailed if you can clarify for us the limitations of your network.

https://assets.nagios.com/downloads/gen ... utions.pdf

Re: Initial Setup of Layered Network

Posted: Thu Oct 08, 2015 9:04 pm
by snidley
Our architecture does not allow machines attached to net1 to be routed to from net2. There are two nodes that are connected to both. But they can not provide routing. So there is no direct addressing allowed from, for instance, ganglia-001 to/from compute-001. There is nothing vague about that. It isn't unclear. It isn't allowed - period.

However, either login-002 or management-001 can collect data from the compute nodes that they can then send on to nagios-001 if there is a plugin/package or configuration option that would allow this.

As far as ganglia-001 is concerned, ganglia is an open source monitoring tool that we use to show performance stats. I don't know how much you know about it, and don't want to go into any details that aren't needed. I just use it as an example because it does the sort of thing I'd like to see in Nagios. There's a monitoring daemon that runs on every host you want to monitor. It gets polled by the metadata daemon that "collects" the data from the monitor daemons. Finally, there is a server where the data get collected from all the metadata daemons you have. This way a node that is only on net2 doesn't have to talk to or be talked to by a node that only resides on net1.

If I understand the options you were suggesting, DNX looks like it might provide the sort of thing I need, but I need to look into it more. I will let you know if it does the job.

Re: Initial Setup of Layered Network

Posted: Thu Oct 08, 2015 9:28 pm
by snidley
From a little reading about DNX, if I put it on management-001 and/or login-001 I can have it be a data collection point for compute-001 through compute-005. Also, nagios-001 should be able to go to management-001 to retrieve data that came from compute-001 through compute-005.

In production, rather than in this theoretical design diagram, I'll have 40 nodes on one private network 120 nodes on another, and nagios-001 will not be able to directly see any of those nodes. nagios-001, however, will have 1000 or so virtual machines on "net1" that it will be polling, too. Those are no problem, though I may need to use DNX to create collectors that will distribute the work from nagios-001.

Finally, there will be a cloud connection, too. It will come up and down on demand, and may have a variable number of nodes. Is there a good option for monitoring cloud hosts? I think I ought to be able to use DNX there, too, but you may have a better suggestion.

I will post a drawing in the morning.

Re: Initial Setup of Layered Network

Posted: Fri Oct 09, 2015 9:51 am
by jdalrymple
Depending upon interpretation, I could see the picture being indicative of 1 network, 3 networks, or possibly even 9 networks. I have no clue where the broadcast domains (vlans) are, what in this picture is a router, etc. I'm sorry, the picture simply doesn't clarify anything, so I'm going off what I am reading.
snidley wrote:The server, nagios-001, must be connected to Net1. It is the collection point for our Nagios web server. The rules of this environment do not permit direct routing to/from Net1 and Net2. In this environment, I am able to monitor login-001, management-001, ganglia-001 and nagios-001 and display status on the web interface without any trouble.
So these machines are all on connected and routable networks. (nagios-001, login-001, management-001, ganglia-001)
snidley wrote:Ganglia-001 collects and displays data for our Ganglia implementation.
This means nothing to me because I don't know what a Ganglia implementation is. Sorry, it's just not vernacular I know.
snidley wrote:When polled, the nodes, compute-001 through compute-005 send their data to management-001 for Ganglia. So does login-001. And management-001 forwards the data all on to ganglia-001 when polled.
More that I don't understand, but from reading this it sounds like ganglia is some other sort of monitoring infrastructure. Unimportant to me.
snidley wrote: I want to do the same sort of thing with Nagios so that it can monitor compute-001 through compute-005 without any routing to/from Net1 and Net2.
And here all that I can do is infer that you're trying to tell me there is valid route from nagios-001 to compute-001 through compute-005. Realistically that's all we need to know. The question is are there any intermediary hosts with valid routes? I think you're trying to tell me that management-001 does have communication between the compute nodes and ganglia-001 but that doesn't help me understand if there is similar communication to nagios-001. Again, the picture doesn't clarify Layer 2 or Layer 3 barriers, nor does your post indicate anywhere that ganglia-001 and nagios-001 are either on the same network, nor do we know if they share the same routing information.

So the best I can do:

If you do not have this intermediary host that can route to both your compute nodes and nagios-001 monitoring cannot be done over IP obviously, which is kind of the bread and butter of how Nagios communicates. What sort of communication protocol do you propose between nagios-001 and the remaining hosts? If you cannot form some lines of communication with them I fail to understand how monitoring could be at all possible.

If however there is either today or someday the possibility of an intermediary host that can relay checks, the documentation I sent applies. You can use mod_gearman on the Nagios server and a worker in your adjoining network segment to do all of the work. A similar but more difficult to maintain scenario is simply to have a man in the middle nrpe agent performing the checks for you. At a scale of only 5 hosts, either solution would be fine.

Re: Initial Setup of Layered Network

Posted: Fri Oct 09, 2015 11:54 am
by snidley
I am re-posting the pictures. It doesn't seem that you are getting it yet, and I just noticed that they don't seem to show the "net1" and "net2" network objects labeled.

Re: Initial Setup of Layered Network

Posted: Fri Oct 09, 2015 12:10 pm
by snidley
Simple network diagram.

Re: Initial Setup of Layered Network

Posted: Fri Oct 09, 2015 12:13 pm
by snidley
This is where we are going (logically) with cloud added. It would be very beneficial to be able to similarly monitor cloud-based services. I assume DNX will help, assuming the ports are open through the firewall, and routing is in place?