End to end monitoring

[philipjohnwilliams]

Hi,

Apologies if this is a very basic question but I'm a newbie to Nagios...

We currently have Nagios XI vers 5.2.3 and use it to monitor a 3 tier web based solution (web tier, app tier, database tier)

Whilst we have Nagios configured to monitor each node individually, we would like to be able to setup monitors to ensure the web tier nodes can see the app tier nodes and the app tier nodes can see the database tier nodes.

We had a situation recently where all nodes were reporting as being online but the app tier could not contact the database tier.

How can this setup be achieved?

[dwasswa]

Hi @philipjohnwilliams,

We need more details in order to help you resolve your issue....

How are the web,app,database tier setup to communicate with each other? We need more information on that.

You setup Nagios XI to monitor each tier individually..So,yes, the host (app tier) current state is OK(online) but a certain service
on the host that communicates with the database is not running correctly. This means that XI needs to be monitoring that service
in order notify you that its not running..

Therefore, we need more details on how you set this up....
web,app,database tier setup to communicate with each other
-wizards?
-plugin?
-command?

How are you doing your end to end monitoring?

[bolson]

One approach would be to perform a URL check for specific content and create a dynamic (php, java, etc.) web page which accesses both of the backend tiers.

For example an html page on your web server that redirects to a page on a tomcat server with a static string which is replaced by "select some_string from some_table" on your database server.

The redirect would fail if the app server is down. And the static string on the app server would NOT be replaced if the database server is down. So your content would either be:

404 if the redirect failed or static string if the redirect is ok but the DB server is down or single string result set if everything is working.

[bolson]

Are the web server, application server, and database server all on your private network? Are they multi-homed and in different subnets? The reason I ask is that if all three tiers are in the same subnet it's hard to conceive of how your Nagios server would be able to communicate with all three but the three would not be able to communicate with each other. Please send us detailed information regarding your n-tier topology when you can.

Thank you!

[philipjohnwilliams]

Thanks for all your posts

As requested a bit more information...

The solution resides in AWS and is protected by security groups at each tier level. The web servers will be on public IP ranges and the app and database tier on different private IP ranges, with the relevant security groups configured to allow access at each level (web > app, app > DB)

The Nagios server is allowed to connect to all tiers (via a VPC pier, network routing & security groups)

Essentially what we want to be able to achieve is an end to end monitor for web, app & database to ensure that if one tier can't see the other at a particular time (for whatever reason, misconfiguration, etc) then we are alerted. We can monitor each node in each tier in isolation from the Nagios server which is ok, but we would ideally like to know if one of the tiers can't see the other tier

Hope that makes sense, let me know if not

[bolson]

What is your database server (MSSql, mysql, postgress, etc?)
What is your application server (java php? etc?)

And do you just need to check connectivity? Any other functionality you need to check.

The solution that comes to mind is to use check_http against the http tier and create a simple web app as I described in my first post:

For example an html page on your web server that redirects to a page on a tomcat server with a static string which is replaced by "select some_string from some_table" on your database server.

The redirect would fail if the app server is down. And the static string on the app server would NOT be replaced if the database server is down. So your content would either be:

404 if the redirect failed or static string if the redirect is ok but the DB server is down or single string result set if everything is working.

[philipjohnwilliams]

DB server is MS SQL

App server is a Windows VM running an in house written application

At this stage, just checking basic TCP/IP connectivity between the web server, app server and SQL database server would be a good start, and then we can look at introducing more complex checks in

Thanks again

[tacolover101]

i think this post is getting over thought. use an agent, and use check_tcp - simple.

from your app servers, run check_tcp against 1433 to verify your DB connections and against port 80/443 (or whatever ports your web tier is running on)

then from your web / db servers, apply the same logic for the ports which the grouped servers used.

[dwhitfield]

use an agent

I think this is the key bit that has otherwise been missing.

More specifically, set up NRPE on each server, if you haven't yet. Have check_tcp from the web server check the app server. Have check_tcp from the app server check the db server. If you have trouble setting up NRPE, that should probably go in a separate thread.

[philipjohnwilliams]

Thanks all, I'll look into installing the agent if it hasn't already been done