Unacceptable: patterns directory deleted with no warning

[eloyd]

Finally got around to doing 1.4 -> 2.0.2 upgrade. Aside from the fact that elasticsearch crashed right as the upgrade started (which killed the upgrade), it went fine.

HOWEVER, if anyone has added any custom patterns files, they're gone. Not moved, not ignored, they're deleted. Previously, they were in /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-0.1.10/patterns/ Now they're in [same path, different version]. Except, the previous directory seems to be DELETED. With no warning, confirmation, or other information that says that this will be the case.

Luckily, like a good sysadmin, we have a backup. But this needs to be addressed for future upgrades. It's fairly easy to make new patterns, but if you don't know they're going to be deleted, you may not have a backup.

[jomann]

The upgrade to Logstash 2.4 with all plugins included is was what caused this, basically the vendors files are files that Logstash uses as libraries, that are just gems. We don't actually do anything in them and when you upgrade Logstash it overwrites them

Is there not a way to put the patterns into another place for it to load rather than in the library itself? I am pretty sure our documents don't say anything about it but if they do, we could update them.

[eloyd]

Grok filter normally allows a patterns_dir directive to tell it where to find patterns. By default, that's the bundle/.../patterns directory. I suppose we could specify a patterns_dir in every custom filter we have to specify /usr/local/nagioslogserver/patterns (which is what we were using as a symlink to the bundle directory) but I'd still like to see a warning that says that all patterns (including any custom ones you may have developed) that reside in bundle/.../patterns will be deleted with an upgrade. Sometimes, you just don't think of everything.

[mcapra]

The "actual" Logstash environment that is responsible for loading the various gems looks for the "patterns" path in the Logstash home:

Code: Select all

def pattern_path(path)
      return ::File.join(LOGSTASH_HOME, "patterns", path)
    end

Which for NLS's setup evaluates to /usr/local/nagioslogserver/logstash/patterns. There should probably be a KB article written on this topic.

[eloyd]

A previous upgrade negated that directory from actually working once 1.9 came into being. Which is why we symlinked it. We've updated our filters to include a specific patterns_dir outside the Nagios directory tree to stop this from happening again.

[mcapra]

Ah, +1 for persisting (or at least notifying of) changes to the Logstash environment as a feature

[eloyd]

I have a correction to make:

Yes, that's the correct location. I think the problem was that a 1.4->1.9 upgrade also deleted the directory, but I can't remember. All I know is that we've had bad luck with NLS upgrades and custom pattern files.

[dwhitfield]

There should probably be a KB article written on this topic.

I've made a note to our documentation team about the above.

Is there anything else on this specific topic that should be included in the KB?

[mcapra]

Well, if there was indeed a minor update to the Logstash shipped with Nagios Log Server, and that update wiped out the existing Logstash environment (not just the Ruby gems), I'd think that this should either be:

• Only done when absolutely necessary and with a "confirmation required" warning in the upgrade script
• Explicitly mentioned in the KB responsible for instruction regarding custom grok patterns
• Explicitly mentioned somewhere in the app itself (excessive work for an under-utilized feature IMO)

But I haven't taken the time to qualify that "bug", if it exists.

[eloyd]

Meh. I've moved on. We now ship our customers NLS intallation stuff with specific patterns_dir directives, using /usr/local/nls_patterns as the target, so no matter what happens in /usr/local/nagioslogserver, they won't be affected.