Logserver Alerts below threshold not working

[weveland]

Gentlemen,

Also as a troubleshooting measure I closed older indexes in case there was just too much to fit into memory. I did reopen them after there was no change.

[weveland]

Ok so I've just proven that my searching problem relies specifically on this field. In my filter configuration I used " rename => [ "response", "response_code" ] " and magically the response_code field is searchable.
I'm not sure what's going on here but this is getting really strange.

[jolson]

I have some theories about what might be happening here:

Your response field is actually a 'string' field - you can verify this by checking the following.

2015-09-21 10_05_51-Movies & TV.png

When you moved the 'response' field over to 'response_code', the 'string' assignment changed to 'long' or similar.

convert => [ 'response', 'integer' ]

If your field is in the 'string' format, and you're trying to mutate a value into an integer, it's possible that there will be problems. As a test, you could attempt to remove the above mutation from your filter.

Jesse

[weveland]

Jesse,

That works. But it also worked when I just renamed the response field. The rename happened after the field modification. Why would renaming the field matter the same as changing the string to integer?

Also the alert I have setup for this field uses a filter match of response:[500 TO 599] to catch a variety of critical errors. So I rely on this field being an integer.

mutate {
replace => [ 'type', 'apache_access' ]
convert => [ 'bytes', 'integer' ]
convert => [ 'response', 'integer' ]
rename => [ "response", "response_code" ]

--
Wayne

[jolson]

The rename happened after the field modification.

This is your answer - since you're converting the string to an integer first, then renaming the field. When you rename a field, a new field is generated (in your case response_code). At this point your field is generated with the proper type.

Why would renaming the field matter the same as changing the string to integer?

It is my understanding that when you are *just* changing the string to integer (using a field that already has the 'string' type), problems can occur. When a new field is generated after you convert the type to an integer, the new field is generated with the 'long' type instead of the 'string' type, which appears to work properly.

Does my answer line up with what you are seeing?

[weveland]

Yes. Kind of. I'm just not sure why it would just stop working all of the sudden when it had been working previously?

--
Wayne

[weveland]

So can I rename the field to the same name to get the proper effect? or should I just drop the %{APACHECOMMONLOG} and put in my own pattern that captures directly as a %{NUMBER}?

[jolson]

The problem we have is that when a field of any type is created, the type cannot be changed until a new index is generated. That being the case, I recommend using a new field name entirely e.g. response_code.

So can I rename the field to the same name to get the proper effect?

You cannot - only one field may exist with a specific name at a time. If you'd like to have the same name, rename the field as recommended above. After the rename, wait for the old field (responsecode) to drop out of existence. Once it drops, feel free to re-use that field name (ensuring that whatever enters that field *first* is certainly an integer).

[weveland]

So then following your logic.

This is probably what caused my rule to break. Even though it's in another type. The field name is reused as a catch_all field for the response which is both text and integer. So it's likely this rule was triggered first after the index changeover thereby causing the field to be varchar instead of int or float?

match => [ 'message', '<%{POSINT:syslog_priority}> %{DATA:barracuda_process}: %{IPORHOST:connected_host}(?:\[%{IP:host_ip}\])? %{DATA:message_id} %{INT:start_time} %{INT:end_time} SEND %{DATA:encrypted} %{POSINT:action} %{WORD:queue_id} %{GREEDYDATA:response}(?:\n)?' ]

--
Wayne

[jolson]

weveland,

You are exactly correct, and that's almost certainly what happened here. Good catch! Ensure that any data entering any field is of a consistent type - this is especially important with integers and dates, since strings are more flexible.