Can Nagios Log Server handle Splunk formatted messages?

[prhunixadmin]

Hello all,

I'd like to use my nagioslogserver to write F5 logs directly to it. The F5 company states they only support Splunk logging. Is there a way to write a text formatted messages to the log server? I would particularly like to receive the logs for ASM. But my filters aren't working as the messages come in text format. Right now i get data all globbed together.

Can someone help me with this?

Greg

[jolson]

Greg,

This should not be a problem. What I would like from your end is a few full logs that you're receiving on Nagios Log Server. Take the globbed results (probably from the 'messages' field) and post the results here - I'd like to see at least 3-4 logs to ensure that I can help you generate a proper filter. I'm also interested in seeing your current input/filter.

Thanks!


Jesse

[WillemDH]

I'm interested to see if the Splunk output works. Sending F5 logs myself to NLS. I'm working on a set of filters for F5 load balancer. Check out https://github.com/willemdh/logstash_filter_f5 It does need some work though.

[prhunixadmin]

Joel,

Here is the output from the messages field on my NLS

<134>Aug 21 15:42:59 wm-f5-rhwebprd2.us.randomhouse.com ASM:Type: Information Leakage Date: 2015-08-21 15:42:58 Dest IP: 170.171.208.125 Dest Port: 80 Geo Location: US Header: Cache-Control: no-cache\r\nConnection: Keep-Alive\r\nPragma: no-cache\r\nAccept: */*\r\nAccept-Encoding: gzip, deflate\r\nFrom: bingbot(at)microsoft.com\r\nHost: www.fodors.com\r\nUser-Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)\r\nX-Forwarded-For: 207.46.13.0\r\n\r\n Client IP: 207.46.13.0 Plicty Name: /Common/FodorWeb_Base_ASM_Policy Protocol: HTTP Query String: Reqeust: GET /world/mexico-and-central-america/costa-rica/northern-plains//feature_30069.html HTTP/1.1\r\nCache-Control: no-cache\r\nConnection: Keep-Alive\r\nPragma: no-cache\r\nAccept: */*\r\nAccept-Encoding: gzip, deflate\r\nFrom: bingbot(at)microsoft.com\r\nHost: www.fodors.com\r\nUser-Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)\r\nX-Forwarded-For: 207.46.13.0\r\n\r\n Request Status: alerted Response: Only illegal requests are logged Response Code: 500 Severity: Informational Support ID: 1352865552413476669 URI: /world/mexico-and-central-america/costa-rica/northern-plains/feature_30069.html Violaction: Illegal HTTP status in response Violation Detail:

[prhunixadmin]

Willem,

I've used some of your notes to get started. They have been very helpful. However nowhere in your documentation do you include your patterns

E.G. F5SEQ:f5_sequence and F5ID:f5_seqid. Those seem crucial to get your examples working correctly. Can you provide me your grok patterns for these.

Thanks again!

Greg

[WillemDH]

Here you go:

Code: Select all

HOSTNAMEUND \b(?:[_0-9A-Za-z][_0-9A-Za-z-]{0,62})(?:\.(?:[_0-9A-Za-z][_0-9A-Za-z-]{0,62}))*(\.?|\b)
IPNA (?:%{IPV6}|%{IPV4}|N\/A)
SCHEME (HTTPS?)
F5SEQ ([0-9]*:[0-9])
F5ID ([a-z0-9]*)

I didn't think of adding them. I'll do that asap.

What f5 'program' is the log you showed in the example. The log looks different.

[jolson]

Looks like Willem has a handle on this thread.

prhunixadmin, let us know if Willems' solution works for you. Thanks!

[prhunixadmin]

Hello,

Still having issues with this filter. I've setup the patterns and modified Wilhem's configuration to suit my needs. I've created seperate log output for f5 error messages. Here is what im seeing.

{"message":"default send string","@version":"1","@timestamp":"2015-08-26T14:13:17.682Z","type":"syslog-asm","host":"10.104.83.2","tags":["_grokparsefailure"]}
{"message":"default send string","@version":"1","@timestamp":"2015-08-26T14:13:19.519Z","type":"syslog-asm","host":"10.104.83.3","tags":["_grokparsefailure"]}
{"message":"default send string","@version":"1","@timestamp":"2015-08-26T14:13:22.686Z","type":"syslog-asm","host":"10.104.83.2","tags":["_grokparsefailure"]}
{"message":"default send string","@version":"1","@timestamp":"2015-08-26T14:13:24.524Z","type":"syslog-asm","host":"10.104.83.3","tags":["_grokparsefailure"]}

[prhunixadmin]

This also.

{"message":"<130>Aug 26 10:17:18 wm-f5-rhwebprd2.us.randomhouse.com ASM:Type: Session Hijacking Date: 2015-08-26 10:17:17 Dest IP: 170.171.208.66 Dest Port: 80 Geo Location: AE Header: Accept: image/png, image/svg+xml, image/*;q=0.8, */*;q=0.5\\r\\nReferer: http://www.randomhousebooks.com/books/5 ... t-Language: en-US,en-GB;q=0.7,en;q=0.3\\r\\nUser-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko\\r\\nAccept-Encoding: gzip, deflate\\r\\nHost: www.randomhousebooks.com\\r\\nDNT: 1\\r\\nConnection: Keep-Alive\\r\\nCookie: TS01423b3a=01b1b2a32d6a8006219eee1744ef32dbc007671527beb9cf5659b6324af0dfb562360c053b; TS01423b3a_28=01b146369e20b087a61b68bf2d38bb1156d186c9421fd20e9217311fc4f1e7950bf06e3bcaf235dbb9e48d9e8e7022591e714db43c; TS01423b3a_77=2934_8f596c441223bd38_rsb_0_rs_http%3A%2F%2Fwww.randomhousebooks.com%2Fgenres%2Fscience-fiction-fantasy%2F_rs_1_rs_1; TS01423b3a=01b1b2a32d6a8006219eee1744ef32dbc007671527beb9cf5659b6324af0dfb562360c053b; TS01423b3a_28=01b146369ed68e553d0baaefb89c88b8094174a1e1992f8408dc6782ea9c7f59f6c887a932034d3687ae3df4c105f7b20b8f4233e0; utag_main=v_id:014f6a5d4a8f003dee78ab177cac010b000200a800808$_sn:1$_ss:0$_pn:4%3Bexp-session$_st:1440600431139$ses_id:1440598542991%3Bexp-session; pid=undefined; cdi=undefined; s_fid=1836773E4773CBD7-318CC3317958B434; s_cc=true; _ga=GA1.2.1952307927.1440598545; _gat=1; s_sq=%5B%5BB%5D%5D\\r\\nX-Forwarded-For: 86.97.104.82\\r\\n\\r\\n Client IP: 86.97.104.82 Plicty Name: /Common/RHWeb_Base_ASM_Policy Protocol: HTTP Query String: Reqeust: GET /wp-content/themes/wp_rhpg/static/img/icon/facebook--dark.svg HTTP/1.1\\r\\nAccept: image/png, image/svg+xml, image/*;q=0.8, */*;q=0.5\\r\\nReferer: http://www.randomhousebooks.com/books/5 ... t-Language: en-US,en-GB;q=0.7,en;q=0.3\\r\\nUser-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko\\r\\nAccept-Encoding: gzip, deflate\\r\\nHost: www.randomhousebooks.com\\r\\nDNT: 1\\r\\nConnection: Keep-Alive\\r\\nCookie: TS01423b3a=01b1b2a32d6a8006219eee1744ef32dbc007671527beb9cf5659b6324af0dfb562360c053b; TS01423b3a_28=01b146369e20b087a61b68bf2d38bb1156d186c9421fd20e9217311fc4f1e7950bf06e3bcaf235dbb9e48d9e8e7022591e714db43c; TS01423b3a_77=2934_8f596c441223bd38_rsb_0_rs_http%3A%2F%2Fwww.randomhousebooks.com%2Fgenres%2Fscience-fiction-fantasy%2F_rs_1_rs_1; TS01423b3a=01b1b2a32d6a8006219eee1744ef32dbc007671527beb9cf5659b6324af0dfb562360c053b; TS01423b3a_28=01b146369ed68e553d0baaefb89c88b8094174a1e1992f8408dc6782ea9c7f59f6c887a932034d3687ae3df4c105f7b20b8f4233e0; utag_main=v_id:014f6a5d4a8f003dee78ab177cac010b000200a800808$_sn:1$_ss:0$_pn:4%3Bexp-session$_st:1440600431139$ses_id:1440598542991%3Bexp-session; pid=undefined; cdi=undefined; s_fid=1836773E4773CBD7-318CC3317958B434; s_cc=true; _ga=GA1.2.1952307927.1440598545; _gat=1; s_sq=%5B%5BB%5D%5D\\r\\nX-Forwarded-For: 86.97.104.82\\r\\n\\r\\n Request Status: alerted Response: Connection Reset Response Code: 0 Severity: Critical Support ID: 1352865552536458708 URI: /wp-content/themes/wp_rhpg/static/img/icon/facebook--dark.svg Violaction: ASM Cookie Hijacking Violation Detail: \r","@version":"1","@timestamp":"2015-08-26T14:17:22.331Z","host":"10.104.83.1","type":"syslog-asm","tags":["_grokparsefailure"]}

[prhunixadmin]

another:

{"message":"<131>Aug 26 10:17:18 wm-f5-rhwebprd2.us.randomhouse.com ASM:Type: Cross-site Request Forgery Date: 2015-08-26 10:17:15 Dest IP: 170.171.208.23 Dest Port: 80 Geo Location: US Header: Host: www.magictreehouse.com\\r\\nUser-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/6.1.1 Safari/537.73.11\\r\\nAccept: */*\\r\\nReferer: http://www.magictreehouse.com/iframe_sw ... t-Language: en-us\\r\\nAccept-Encoding: gzip, deflate\\r\\nCookie: TS01423b3a=01b1b2a32d380e0f8b05cdd2d9b9ea847bd18e737544f89a66d1a5d06ad621267bf5dc4bd5e3e8eca9023f975f6fb5435633bc2cc9; TS01423b3a_77=6706_10be533c919961c6_rsb_0_rs_http%3A%2F%2Fwww.magictreehouse.com%2Fiframe_swf.html%3FLibrary.swf%3Fnoop%3D1_rs_1_rs_0; PHPSESSID=a9bfc1b2e5c66b1633826883986627e2; TS01423b3a_28=01b146369e8791643a7543a01a3a2d577fa31d0da25a815ee71fdb10127a634704c994e9391f928c4911502933ca8bbd72152f8a09; s_sq=%5B%5BB%5D%5D; SC_LINKS=%5B%5BB%5D%5D; visit_referrer=www.google.com; s_cc=true\\r\\nConnection: keep-alive\\r\\nX-Forwarded-For: 71.236.230.108\\r\\n\\r\\n Client IP: 71.236.230.108 Plicty Name: /Common/RHWeb_Base_ASM_Policy Protocol: HTTP Query String: noop=1&id=125123 Reqeust: GET /passport.swf?noop=1&id=125123 HTTP/1.1\\r\\nHost: www.magictreehouse.com\\r\\nUser-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/6.1.1 Safari/537.73.11\\r\\nAccept: */*\\r\\nReferer: http://www.magictreehouse.com/iframe_sw ... t-Language: en-us\\r\\nAccept-Encoding: gzip, deflate\\r\\nCookie: TS01423b3a=01b1b2a32d380e0f8b05cdd2d9b9ea847bd18e737544f89a66d1a5d06ad621267bf5dc4bd5e3e8eca9023f975f6fb5435633bc2cc9; TS01423b3a_77=6706_10be533c919961c6_rsb_0_rs_http%3A%2F%2Fwww.magictreehouse.com%2Fiframe_swf.html%3FLibrary.swf%3Fnoop%3D1_rs_1_rs_0; PHPSESSID=a9bfc1b2e5c66b1633826883986627e2; TS01423b3a_28=01b146369e8791643a7543a01a3a2d577fa31d0da25a815ee71fdb10127a634704c994e9391f928c4911502933ca8bbd72152f8a09; s_sq=%5B%5BB%5D%5D; SC_LINKS=%5B%5BB%5D%5D; visit_referrer=www.google.com; s_cc=true\\r\\nConnection: keep-alive\\r\\nX-Forwarded-For: 71.236.230.108\\r\\n\\r\\n Request Status: alerted Response: Logging rate limit reached Response Code: 200 Severity: Error Support ID: 1352865552537032727 URI: /passport.swf Violaction: CSRF attack detected Violation Detail: \r","@version":"1","@timestamp":"2015-08-26T14:17:22.276Z","host":"10.104.83.1","type":"syslog-asm","tags":["_grokparsefailure"]}