File Input on local log server

[CFT6Server]

we have some critical logging that is happening at the moment. Will this be a big upgrade? Will anything existing be impacted?

Also is this our only option to get file inputs to work? we have a time sensitive ask to get these file inputs into the log servers. Thanks!

[jolson]

Well, I would hate to tell you to upgrade outside of a maintenance window, and I'm afraid that we could cause more complications by doing that upgrade now, so I'd like to try and resolve this without taking down your cluster for upgrades/etc.

Do your global configurations push down appropriately *without* defining the instance-only configuration? I'd like you to delete the instance only configuration via the Web GUI, and add a fake input to your global configuration. When you apply config, does the config push down properly?

If the configuration does not push down properly, we need to reprimand that before we try to add an instance only configuration again. I tested this procedure on 2015R1.4 and experienced no problems. Thanks!

[CFT6Server]

The global configuration pushes fine when there is no global configuration and that is working. Actually been working on some global configurations last night and today.

[jolson]

On your node that is having problems inheriting an instance configuration, let's try the following.

First, configure everything properly in the Web GUI, and insert the following under your 'per instance' configuration as usual:

Code: Select all

file {
  path => "/nfs/shared/servers/logs/*"
  type => "apache"
}

After saving, log into the CLI and run the following:

Code: Select all

cd /usr/local/nagioslogserver/logstash/etc/conf.d/
mv 000_inputs.conf ~/000_inputs.conf
mv 500_filters.conf ~/500_filters.conf
mv 999_outputs.conf ~/999_outputs.conf

After your configuration files have been moved to your home directory, run an 'Apply Configuration' from the Web GUI. This will force NLS to generate brand new configurations, eliminating any possible weirdness on the config end of things. Do the configurations generate properly?

Code: Select all

ls -l /usr/local/nagioslogserver/logstash/etc/conf.d

Code: Select all

cat /usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf

[CFT6Server]

Thanks for the steps. Looks like the configuration was actually being pushed, but it doesn't show in the GUI.

So if I cat the configuration file, it has the local inputs. I've tested it with minor changes without removing the files and that seems to be working now. But I tried this for a second time and it is not longer making configuration changes to the local file again.

Looks this node is no longer taking updates. I have also made a fake global input and that is no longer being pushed again. Very odd.

[CFT6Server]

Update: I've removed the local input and applied global configuration and the global changes applied again.

[jolson]

Could you describe a method that you can use to reproduce this problem? As it stands, I can't get this to reproduce on my end. If this issue is reproducible we can likely figure out why it's happening and get a patch from the devs.

Jesse

[CFT6Server]

OK so current state:
no local input configuration, make changes and apply, this is fine on all nodes.

Testing:
go to per instance configuration, add local configuration via the web GUI, apply configuration, view inputs does not show local inputs and checking the file locally, it does not produce local inputs.

Testing 2:
Remove the local configuration files on node via SSH, apply configuration, view inputs does not show local inputs, no input files are created at all on this node. Directory is empty.

Testing 3:
Make fake global configuration, Apply configuration, the node is not making any configuration files. Directory is empty.

Now I go back to remove the local input from per instance, apply configuration, and all files are created. Remove Dummy global input, and applied configuration. everything is updating.

Update:
Hmm... So after this test I tried to run this again. This time, here is my full steps from a clean config.

Create local input, save but not apply. then remove local files on node, apply configuration in GUI. check files, they are there. and NOW make global or local changes, and it is working.... so not sure why this worked a second time around....

[jolson]

I have submitted a bug report describing the faulty 'Verify' and 'View' buttons on a per-instance basis. The task ID for this bug is 6116, feel free to reference this task ID to us and we'll check on the status of the bug for you.

Regarding the other issue where your global configurations stop functioning after a local config works, I will report my results to you now.

My test system:
Nagios Log Server 2015R1.4
Logstash configs:

Code: Select all

# 
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Fri, 24 Jul 2015 14:03:45 -0500
#

#
# Global Configuration
#

input {
    syslog {
        type => 'syslog'
        port => 5544
    }
    tcp {
        type => 'eventlog'
        port => 3515
        codec => json {
            charset => 'CP1252'
        }
    }
    tcp {
        type => 'import_raw'
        tags => 'import_raw'
        port => 2056
    }
    tcp {
        type => 'import_json'
        tags => 'import_json'
   0P;[
\9     port => 2057
        codec => json
    }
}

filter {
    if [program] == 'apache_access' {
        grok {
            match => [ 'message', '%{COMBINEDAPACHELOG}']
        }
        date {
            match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z' ]
        }
        mutate {
            replace => [ 'type', 'apache_access' ]
  -*           convert => [ 'bytes', 'integer' ]
             convert => [ 'response', 'integer' ]
        }
    }
     
    if [program] == 'apache_error' {
        grok {
            match => [ 'message', '\[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\] \[%{WORD:class}\] \[%{WORD:originator} %{IP:clientip}\] %{GREEDYDATA:errmsg}']
        }
        mutate {
            replace => [ 'type', 'apache_error' ]
        }
    }
}

#
# Local Configuration
#

go to per instance configuration, add local configuration via the web GUI, apply configuration, view inputs does not show local inputs and checking the file locally, it does not produce local inputs.

I went to 'per instance' configuration, and added the following config via the web GUI:

Code: Select all

    file {
      path => "/nfs/shared/servers/logs/*"
      type => "apache"
    }

2015-07-24 14_06_39-Instance Configuration • Nagios Log Server - Firefox Developer Edition.png

I performed an 'Apply Configuration' and the local configuration was pushed down to Nagios Log Server appropriately.

2015-07-24 14_07_49-root@ssc66ls1__usr_local_nagioslogserver_logstash_etc_conf.d.png

Testing 2: Remove the local configuration files on node via SSH, apply configuration, view inputs does not show local inputs, no input files are created at all on this node. Directory is empty.

I removed the local configuration files on the node via SSH:

Code: Select all

[root@ssc66ls1 conf.d]# pwd
/usr/local/nagioslogserver/logstash/etc/conf.d
[root@ssc66ls1 conf.d]# mv 000_inputs.conf ~/000_inputs.conf
[root@ssc66ls1 conf.d]# mv 500_filters.conf ~/500_filters.conf
[root@ssc66ls1 conf.d]# mv 999_outputs.conf ~/999_outputs.conf
[root@ssc66ls1 conf.d]# ls
[root@ssc66ls1 conf.d]#

Applied my configuration, and everything was re-populated correctly.

2015-07-24 14_20_58-root@ssc66ls1__usr_local_nagioslogserver_logstash_etc_conf.d.png

Testing 3: Make fake global configuration, Apply configuration, the node is not making any configuration files. Directory is empty.

I made the fake configuration:

2015-07-24 14_22_08-Instance Configuration • Nagios Log Server - Firefox Developer Edition.png

Applied config:

2015-07-24 14_23_35-root@ssc66ls1__usr_local_nagioslogserver_logstash_etc_conf.d.png

I'm not seeing the same problems that you are. Am I missing something? If you could, would you share your entire logstash configuration with me so that I could duplicate your setup exactly?

Jesse