_csvparsefailure

[firegoblin]

I'm testing porting of some of my ELK config into Nagios Log Server, to see if we should 'upgrade' and have found a difference in behaviour.

Currently we are processing some syslog streams that can have truncated elements that cause CSV parsing to fail (as they contain a nested URL that can be very long). We 'semi-gracefully' capture these with a snippet of code as below:-

if "_csvparsefailure" in [tags] {
mutate { add_field => { 'threat_type' => 'Warn - Long URL' } }
mutate { remove_tag => [ "_csvparsefailure"] }
mutate { add_tag => [ "pa_th_csv_error" ] }

My issue is that Nagios does not seem to capture these entries - they are discared automatically when the CSV parse fails (whereas at the moment we can at least dump the raw message into a bulk field for manual review). I assume there is an ELK setting being applied somewhere that changes this behaviour during startup, but not sure where it is?

Any pointers welcomed!

[abrist]

I assume there is an ELK setting being applied somewhere that changes this behaviour during startup, but not sure where it is?

Could you clarify the behavior of which you speak?
Could you show us an example of the failing query?

[firegoblin]

Apologies.

After a bit more testing we noted that we could get csvparse failures reported.

This issue was that - when transposing the config from native ELK into NLS we used the config 'block' function to make it easier to manage. When doing this we slightly changed the overall flow and this stopped the error occuring (not actually good for us, as the logic used it to trigger a different parse, but that's a different story!).

Thanks

[sreinhardt]

No problem, did you get it all figured out, it sounds like it but just want to be sure.