Global Configuration - Outputs

[ssoliveira]

Hello,

Good afternoon,

I'm trying to customize the "output" filter to break the logs by "type-date".

The configuration file warns you, it MUST not be modified manually;
For it will be rewritten (vim /usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf)

I'm trying to add a global configuration, through the interface "Global Configuration" >> "Outputs". So that the indexes creation pattern is:

output {
elasticsearch {
index => "%{type}-%{+YYYY.MM.dd}"
}
}

For the configuration does not pass the validation.

How can I configure the output for this?

Without having the risk of the settings being lost, if someone added new filters via the dashboard.


If I configure it manually, it works. Well, I'm going to have problems with the first filter change that is performed on the panel.

Code: Select all

vim /usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf

index => "%{type}-%{+YYYY.MM.dd}"
------------

#
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Fri, 07 Jul 2017 14:22:11 -0300
#

#
# Required output for Nagios Log Server
#

output {
    elasticsearch {
        cluster => '93f3e418-f86d-4d96-a812-1a8b50e3027d'
        host => 'localhost'
        document_type => '%{type}'
        node_name => '3a6aa95d-8e0e-4a97-9623-0a10ccfad7d6'
        protocol => 'transport'
        workers => 4
        index => "%{type}-%{+YYYY.MM.dd}"
    }
}

#
# Global outputs
#



#
# Local outputs
#


------------

[scottwilkerson]

This design would break much of how Nagios Log server works, so I'm not going to give the recommendation on how to do it.

And, it should be noted that the indexes already separate the logs by type.

Finally, it would cause a duplication of how the logs are created as the elasticsearch output that Nagios Log Server expects is hard coded to prevent it from being broken by misconfiguration.

[scottwilkerson]

With the above said, I will tell you how you could setup an ADDITIONAL output like the one mentioned through the UI.

In the configuration -> global config click "Show Outputs" upper left.

create new output and enter

Code: Select all

elasticsearch {
        cluster => '93f3e418-f86d-4d96-a812-1a8b50e3027d'
        host => 'localhost'
        document_type => '%{type}'
        node_name => '3a6aa95d-8e0e-4a97-9623-0a10ccfad7d6'
        protocol => 'transport'
        workers => 4
        index => "%{type}-%{+YYYY.MM.dd}"
    }

I'm gonna make another notation that when you upgrade, this is going to break because the format is different in 2.0.0+

[ssoliveira]

This did not work.

It generated two outputs, causing each log to be logged twice.

Once in [logstash-] YYYY.MM.DD
And again [syslog-] YYYY.MM.DD (which is the log type)

This problem is very critical here in the company.

The indexes are too large, and the inability to break indexes is causing problems.

I am researching which script is executed when a new configuration is applied. Thinking about changing it, so that the output is generated with the parameter that sets the name of the index.

Do you consider a valid alternative? Could you tell me what script is executed when a new configuration is applied?

The rotation of logs, I can do via crontab, with the curator (and disables the automatic rotation in the panel).

About version 2.0, I'm evaluating the update.

Code: Select all

#
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Mon, 26 Feb 2018 16:35:27 -0300
#

#
# Required output for Nagios Log Server
#

output {
    elasticsearch {
        cluster => '93f3e418-f86d-4d96-a812-1a8b50e3027d'
        host => 'localhost'
        document_type => '%{type}'
        node_name => '5675db95-6156-4cfc-b3bd-1acf77f22864'
        protocol => 'transport'
        workers => 4
    }
}

#
# Global outputs
#

output {
    elasticsearch {
            cluster => '93f3e418-f86d-4d96-a812-1a8b50e3027d'
            host => 'localhost'
            document_type => '%{type}'
            node_name => '3a6aa95d-8e0e-4a97-9623-0a10ccfad7d6'
            protocol => 'transport'
            workers => 4
            index => "%{type}-%{+YYYY.MM.dd}"
        }
}

#
# Local outputs
#

[scottwilkerson]

I said this very thing in the above post
https://support.nagios.com/forum/viewto ... 72#p245887

Finally, it would cause a duplication of how the logs are created as the elasticsearch output that Nagios Log Server expects is hard coded to prevent it from being broken by misconfiguration.

How large is each index getting? They are already split into 5 shards each. If you need to split them into smaller chunks for any reason, the correct thing to do would be to increase the number of shards. If there is good reason to do so, someone here can help with that.

What is the trouble you are having with the index size?

[ssoliveira]

Currently my logs have 100-200GB per day.

Being that large mass of data are logs I can get you a smaller spin, and delete.

With 30 days of active logs.

It is common for me to request that I make available entire months for logs. However this is impracticable, as heap memory ends.

I've talked about this in the past.
https://support.nagios.com/forum/viewto ... 38&t=45401

We are quoting the NLS cluster extension, in addition to purchasing Elastic.co's x-pack for replacing Nagios, but the root cause is that it is impossible to break the logs.

However, we will not make any decisions until we try to resolve this issue of breaking the logs. We have new laws in the country, which require logs for longer periods, as a consequence my log load will increase a lot.

If I can break the logs, I can have different log rotation, and can have logs on, different periods, by log type.

I have already upgraded my lab to version 2.x.

If it is possible to customize the script that generates the output content, I think we have found a solution. In future new updates, we can double-check this file, to mater the application running as we need.

[scottwilkerson]

Unfortunately this file is encrypted so I cann't just give code to fix.

Additionally, I need to point out that if you manually override the output, it will likely cause many features to fail including archiving, alerts, etc.

[ssoliveira]

Okay, thanks for the info.

I changed the file manually, and changed the permission to 555 so that the "nagios" users can not change the file. I ran some tests via UI, and checked that the filters are being "added" and "removed" correctly. Without changing the output.

I am evaluating this scenarios in the laboratory.

Code: Select all

[root@elklog01 nagioslogserver]# ls -la /usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf
-r-xr-xr-x 1 nagios nagios 432 Feb 26 19:09 /usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf

[root@elklog01 nagioslogserver]# cat /usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf
#
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Mon, 26 Feb 2018 18:55:35 -0300
#

#
# Required output for Nagios Log Server
#

output {
    elasticsearch {
        hosts => ['localhost']
        document_type => '%{type}'
        workers => 4
        index => "%{type}-%{+YYYY.MM.dd}"
    }
}

#
# Global outputs
#



#
# Local outputs
#

[scottwilkerson]

I wish you the best of luck