FortiOS 5.6 GrokParseFailure

[oliverwjones]

mcapra, scottwilkerson, and cdienger,

First, thank you very much for your assistance and patience. I apologize for the delay in posting this message. I have been out of town for some time.

Second, I discovered a couple of things while troubleshooting this issue and implementing the suggestions you provided me.

1. I missed two sets of \" \" in my GROK statement I posted last. These generated the GROK error.
2. After fixing these issues I had at least six variables with duplicate values in their tags. Some were parsed by GROK. Some were parsed by KV.
   1. ap AP221U,AP221U
   2. logtime 1502908275,1502908275
   3. signal -95,-95
   4. radioband 802.11n,802.11n
3. After implementing scottwilkerson's suggestion and reviewing more information and examples on the Internet, I discovered the reuse of the variable "message" between GROK and KV caused the duplication of values. One suggestion was to use a different variable name on each out subsequent function. Therefore, I used the new variable "message_remainder" in the GREEDYDATA function output. I had noticed duplicate values appear in the GROKDEBUGGER output when testing,
   but didn't make the correlation between what was happening in memory (please correct me if I am wrong as I am often. ).

Here is my latest, working filter. This includes a working GEOIP filter.

Code: Select all

if [type] == 'FortiLog' {
              grok {
                    match => [ 'message' , "\<%{DATA:grokpri}\>date=%{DATA:grokdate} time=%{TIME:groktime} %{GREEDYDATA:message_remainder}" ]
                    tag_on_failure => [ 'failure_grok_fortiOS' ]
                    }
   mutate {
          gsub => [ 'message_remainder','^<[0-9]*>','' ]
          }
        kv {
            source => 'message_remainder'
            }
            geoip {
                   source => 'dstip'
                   }
                      mutate {
                             remove_field => [ 'message_remainder' ]
                             }
                         }

Here is a couple screenshots of my main dashboard and a shot of a successfully parsed Fortinet message for you to review.
https://mega.nz/#!HsRDxQrS!NtxTwT_hLUkI ... v-K8zhYA3g
https://mega.nz/#!6xZTVRBb!rWIXXhCCOmYU ... JhqbIAQY7o

Again, I apologize for the delay in responding to your assistance, but I hope this information helps someone who is going through the same issues I have gone through.

Thank you for all your help.

Rodney.

[cdienger]

Thanks for the update and I think I can speak for the others when I say we don't mind the delay - especially since you came back with good news : ) Are you okay with us locking the thread at this point?

[oliverwjones]

Yes.