Scaling Nagios Log Server

[SRC_Rob]

Hello,

I'm trying to put together a proposal for a new virtual data center and I could use some more info. Is it possible to build a heirarchy of Nagios Log Servers? We want each of our customers to have a separate instance of Log Server which they can query but we also want to have a super-user area that perform searches across all of the customers' logs which we can use for monitoring the data center as a whole. Obviously, we don't want the customers to be able to access log data that is not part of their project. Is this design feasible with your product? Would logs have to be duplicated at the customer domain and master domain levels or could the master instance of Nagios Log Server query the individual customer's log servers and generate aggregate search results without having to store duplicate data?

I've been told that unlike elasticsearch, Nagios Log Server provides some form of security controls. I have used elasticsearch before and am well aware that a single curl command can delete every bit of data from the elasticsearch database. What kind of access controls are available? Is it possible to continue allowing curl "get" queries directly into elasticsearch while also using Nagios security controls to prevent accidental deletes by unauthorized users?

Also, bear in mind that some of our customers log tens of thousands of entries per minute, so we need a solution that is highly scalable both for easily adding new customers as well as expanding to process more logs as customer needs grow. Is Nagios Log Server the tool I'm looking for?

Thanks!

Rob

[jolson]

Is it possible to build a heirarchy of Nagios Log Servers? We want each of our customers to have a separate instance of Log Server which they can query but we also want to have a super-user area that perform searches across all of the customers' logs which we can use for monitoring the data center as a whole.

In a sense, this is possible. A heirarchy can be achieved in the sense that you can output your customer data to a 'master cluster', but there is no official method for managing this heirarchy (no GUI components, for instance).

Obviously, we don't want the customers to be able to access log data that is not part of their project. Is this design feasible with your product?

This is no problem - you can simply change the 'cluster.name' variable in /usr/local/nagioslogserver/elasticsearch/config/elasticsearch.yml:

Code: Select all

################################### Cluster ###################################

# Cluster name identifies your cluster for auto-discovery. If you're running
# multiple clusters on the same network, make sure you're using unique names.
#
cluster.name: nagios_elasticsearch

Would logs have to be duplicated at the customer domain and master domain levels or could the master instance of Nagios Log Server query the individual customer's log servers and generate aggregate search results without having to store duplicate data?

Duplicate data will need to be stored - every customer will need to have a custom logstash output that outputs information to your master cluster, in addition to their default output that stores information in their own cluster.

One thing to note is that your master cluster will need to be very powerful to handle the data of all of your clients - a better alternative may be using the API to query customer information, or having a master account on every customer cluster that you could use.

I've been told that unlike elasticsearch, Nagios Log Server provides some form of security controls. I have used elasticsearch before and am well aware that a single curl command can delete every bit of data from the elasticsearch database. What kind of access controls are available?

Curl queries have to be run against localhost, and are not allowed from the outside world. Nagios Log Server has a built-in API which requires a users API token to take advantage of. You can turn API access off for users:

file.png

2015-05-14 09_20_01-Create User • Nagios Log Server.png

An example API call could look something like this:

Code: Select all

curl -XGET 'http://192.168.4.203/nagioslogserver/index.php/api/backend/logstash-2015.05.14,logstash-2015.05.13/_search?pretty&token=yourAPItokenhere'
    "took": 6,
    "timed_out": false,
    "_shards": {
        "total": 10,
        "successful": 10,
        "failed": 0
    },
    "hits": {
        "total": 79848,
        "max_score": 1,
        "hits": [
            {
                "_index": "logstash-2015.05.14",
                "_type": "eventlog",
                "_id": "xxxxxxx",
                "_score": 1,
                "_source": {
                    "EventTime": "2015-05-13 19: 08: 28",
                    "Hostname": "jolson.test.local",
                    "Keywords": -9214364837600034816,
                    "EventType": "AUDIT_SUCCESS",
                    "SeverityValue": 2,
                    "Severity": "INFO",
                    "EventID": 4624,
                    "SourceName": "Microsoft-Windows-Security-Auditing",
                    "ProviderGuid": "{
                        xxxx

The call would then go on to list a full display on the indices selected. This is a small piece of what you can do with the API.

If you want to open up curl commands beyond the localhost, that is do-able but not recommended. If you need to use curl commands, I recommend logging into the shell and running them as localhost.

Also, bear in mind that some of our customers log tens of thousands of entries per minute, so we need a solution that is highly scalable both for easily adding new customers as well as expanding to process more logs as customer needs grow. Is Nagios Log Server the tool I'm looking for?

Regarding expansion, it doesn't get much better than Nagios Log Server. I have witnessed an untuned Nagios Log Server instance with 4G RAM, 4 cores, and a SSD process 3 million logs in 20 minutes without a hiccup. This will depend on the hardware in question of course, but in general if you need more processing power you can always add a new node to the cluster.

Let me know if you have further questions. Thanks!

[SRC_Rob]

Sounds good so far. Another requirement is that the tools must follow the VA's 508 compliance rules. (Accessibility for users with disabilities) Assuming we purchased the entire Nagios suite with Log Server, XI, Network Analyzer, Incident Manager,etc, how close do these products come to meeting 508 compliance?

Here is a list of the specific 508 requirements we must follow:
http://www.section508.va.gov/docs/1194_ ... cklist.doc

If your products are not compliant out of the box, can I get an estimate of how much manpower would be required to make it compliant? Is it customizable enough that we could even do that?

[SRC_Rob]

I'd also like to repeat my question about hierarchy structure in reference to the other Nagios tools. For example, if we want to use network analyzer or XI at both the customer level as well as the master level, do we have to chose between duplicating the data at both levels or writing a custom aggregation website using API calls? Do all of the components support an API similar to Log Server?

[jolson]

Assuming we purchased the entire Nagios suite with Log Server, XI, Network Analyzer, Incident Manager,etc, how close do these products come to meeting 508 compliance?

Our products were not built with the purpose of meeting 508 compliance, and so I am under the impression that none of them are compliant.

If your products are not compliant out of the box, can I get an estimate of how much manpower would be required to make it compliant? Is it customizable enough that we could even do that?

Unfortunately our products are not customizable in this way, since much of the source code required to make these changes is not available.

If this is a problem, I highly recommend reaching out to [email protected] - it's possible that custom consulting could be worked out. Please shoot them an email and reference this thread.

if we want to use network analyzer or XI at both the customer level as well as the master level, do we have to chose between duplicating the data at both levels or writing a custom aggregation website using API calls? Do all of the components support an API similar to Log Server?

Regarding XI -
XI is certainly our most advanced product in this regard. We have a good piece of documentation built for this purpose: https://assets.nagios.com/downloads/gen ... r_MSPs.pdf

Regarding NNA -
Nagios Network Analyzer is likely going to be the most difficult product to deploy using a 'Managed Service' frame of mind. I would recommend setting up a seperate NNA instance for each client, and having a 'master' account on each NNA instance that you could log into to check on their data. NNA also has an API similar to NLS - I would have to dig more to understand the exact functionality of the NNA API.

Do you have any further questions that I could help you answer?