ELK now at version 5.6

[SteveBeauchemin]

So, if the ELK foundation of NLS is at 5.6, will the next version of NLS at least get to 5.something?

I looked at the https://www.nagios.com/roadmaps/ Nagios Roadmaps. The log server section worries me. Version wise.

Steve B

[dwhitfield]

I assume you are referring to the following:

Updated version of Elasticsearch (1.7.6 – requires rolling restarts)
Updated version of Logstash (2.4.x)

That's the plan. We'd need a NLS dev to weigh in as to why we are going with those versions.

[tacolover101]

Elastic moves so fast..

[scottwilkerson]

@SteveBeauchemin it has never been the intention to force Nagios Log Server users along a path that matches ELK development.

Every major version change often is void of any possibility of backward compatibility, and that just isn't in the best interest of commercial clients that require systems to be stable and retain the data they have for a long time to come.

Some upgrades we can do, and when it is appropriate we will do, but we aren't going to leave enterprise customers in the dust so we can incorporate features that are in the latest versions that we haven't heard requested from the mass of our Log Server user base.

For example, upgrading Elastic Search and Kibana required throwing away any custom dashboards as the new versions are not compatible with the old. We would make the change on this side if it was just a portion, but they fundamentally change the way faceting works.

[Envera IT]

@SteveBeauchemin it has never been the intention to force Nagios Log Server users along a path that matches ELK development.

Every major version change often is void of any possibility of backward compatibility, and that just isn't in the best interest of commercial clients that require systems to be stable and retain the data they have for a long time to come.

Some upgrades we can do, and when it is appropriate we will do, but we aren't going to leave enterprise customers in the dust so we can incorporate features that are in the latest versions that we haven't heard requested from the mass of our Log Server user base.

For example, upgrading Elastic Search and Kibana required throwing away any custom dashboards as the new versions are not compatible with the old. We would make the change on this side if it was just a portion, but they fundamentally change the way faceting works.

Then fork it and offer a "stable" branch and a non-stable branch. At a minimum you could be working on incremental updates; improving the integration between XI and NLS would be huge, something I've brought up in the past and put in feature requests for but have seen zero movement on in over a year. You're telling us there's nothing you could have fixed or improved in the last 11 months?

[bheden]

Then fork it and offer a "stable" branch and a non-stable branch.

At this point in time, we have no intention of forking ELK, or releasing non-stable branches of our software.

At a minimum you could be working on incremental updates;

We could be better at this, sure. We do what we can with what we have.

improving the integration between XI and NLS would be huge, something I've brought up in the past and put in feature requests for but have seen zero movement on in over a year. You're telling us there's nothing you could have fixed or improved in the last 11 months?

We're pretty excited for Nagios Log Server 2 which is coming out shortly, and includes the latest security fixes of the compatible version of Elasticsearch.
We've had a dedicated team on improving Log Server for a large portion of that time - focusing on maintaining our Customers' installations - without forcing them to do anything crazy in order to upgrade (like some companies do).

[mcapra]

In defense of ES 1.x, the fact that Nagios Log Server uses dynamic mapping for all records makes transitioning between ES 1.x to 2.x potentially devastating for large systems. You could make arguments for or against that particular design choice (frankly if I were an ops guy I wouldn't want to map every possible log type by hand), but I want to stress that keeping up between ES 1.x, 2.x, and 5.x is non-trivial even if strict mappings and formatting and rules were enforced (like they are doing at Target) to the effect of the Database having it's own version-controlled schema maintained by a handful of engineers.

Even for the carefully-planned cluster that I am partly responsible for, it will take at least one engineer working full-time for 1-3 months to have a proper plan in place to migrate all that stuff from ES 1.x to ES 2.x. This is with well structured data; There's lots of pitfalls we don't have to worry about because we're working with a very consistent set of parameters. NLS has more problems to consider due to being an agnostic log collector.

[dwhitfield]

While development is not done, here's what currently resides in the NLS 2.0 changelog:

- Added Czech as a selectable language (localized translation pending) -SW, JO
- Added alert history tracking and page -JO
- Added ability to give users specific permissions (such as viewing/editing alerts and configuration) -JO
- Added reset command subsystem commands in the upgrade script -JO
- Added ability to add q=<id> or a=<id> to the logserver.js dashboard for clicking through alerts and queries [TPS#10622] -JO
- Added more LDAP user account types for importing from different LDAP setups -JO
- Added activation for licenses -JO
- Added automatic activation for licenses with client ID (or token value) from the GUI -JO
- Added maintenance check and maintenance information including renewal link in "License Information" page -JO
- Added internal proxy settings for maintenance, activation, and upgrade checks [TPS#5095] -JO
- Added loading into the dashboards until one of the panels starts loading to indicate an action is happening -JO
- Added initial close time of 30 days to backup and maintenance section on clean installs [TPS#10748] -JO
- Added option for text only emails intead of HTML [TPS#12230] -JO
- Added functionality to the job subsystem that checks for stuck jobs and will reset them [TPS#7176] -JO
- Added last modified time to the snapshots and maintenance section of Admin area [TPS#10802] -JO
- Added a run button to the actions available for command subsystem jobs to manually run a command now easily -JO
- Added the ability to export the table data as a CSV with matching table headers to columns in CSV [TPS#4176] -JO
- Updated encrypted files to support PHP 7 and 7.1 -JO
- Updated style to the Nagios standard Modern theme -JO
- Updated Elasticsearch to version 1.7.6 -JO
- Updated Logstash to version 2.4.1 (with all plugins included) -JO
- Updated to elasticsearch-knapsack 1.7.3.0 -JO
- Updated CodeIgniter to 3.1.x -JO
- Updated wording for User permissions in the create user page [TPS#10187] -JO
- Updated license pages and trial expiration pages to a better format and to say the correct values based on the situation -JO
- Updated install to check for nagios users's home directory [TPS#10438] -JO
- Updated alerts page when using a custom query to not send to the wrong dashboard, instead it sends to a raw query result page [TPS#9256] -JO
- Updated Backup & Maintenance admin page to show amount of snapshots -JO
- Updated Backup & Maintenance admin page repository creation to use a modal and repository table to show repo size -JO
- Updated "Backup & Maintenance" page to be called "Snapshots & Maintanenace" to better distinguish the different types of backups -JO
- Updated alerts to no longer have Nagios Reactor outputs (people with Reactor outputs will still see their outputs) -JO
- Updated index and cluster index lists to not allow closing the current day's index, only deletion [TPS#9105] -JO
- Fixed issue when imporitng AD/LDAP users who were unchecked would still verify against username/email [TPS#10233] -JO
- Fixed issue with usernames with "-" character in them not working [TPS#10229] -JO
- Fixed issue where install would exit if ntpdate could not get time [TPS#10301] -JO
- Fixed rsyslog script to use Disk-Assisted Queues if connections cannot be made to Log Server -SW
- Fixed ip address resolution in install/upgrade scripts [TPS#10761] -JO
- Fixed dashboard style selection dropdown showing up even though there are not multiple supported themes yet -JO
- Fixed using GET requests against backend API to do searches with JSON body [TPS#10559] -JO
- Fixed ability to make redirect variable work with any url on login page -JO
- Fixed user permissions on alert notification management pages -JO
- Fixed various CSRF and XSS vulnerabilities -JO
- Fixed alert not sending formatted %time% output in alerting methods [TPS#11842] -JO
- Fixed issue with AD/LDAP importing folders with commas in the name [TPS#11393] -JO
- Fixed session user_id not being verified as existing in DB (deleted user stays logged in) [TPS#11342] -JO
- Fixed some of the high memory usage issues on the admin page (may still need to update PHP max memory on larger systems) -JO

[SteveBeauchemin]

Why are you making Jake work so hard?

[tmcdonald]

Why are you making Jake work so hard?

I think he might actually enjoy it.