Nagios Log Server parsing - Need Help

[bennyboy]

Hello,

I'm trying to parse a log file generated by IBM TWS. That log is not really structured. This is my first experience with grok and logstash. Can you help me recognize this well log and then make an alert with the result of a query. The first columns is the event number. As against the rest of columns is random depending on the type of event

As an example of possibility : http://www.ibm.com/support/knowledgecen ... tm?lang=en
Another good link to understand some that of log : http://www.ibm.com/support/knowledgecen ... mevent.htm

[hsmith]

When these logs are generated, are they generated as a single line, or that big log you just showed us? You're going to want to create a grok filter to help break the data down. If your logs don't have any consistent data or formatting, it's going to be nearly impossible to come up with a dynamic filter that works for you.

[bennyboy]

I choose the detect only one patern for the moment. I use grok and try to put some data to a field in Nagios Log Server.

I use rsyslog to send event.log to Nagios Log Server and I choose to name the FILE_TAG to TWSQA523. Now I try to match some data and put that data to a field in Nagios Log Server.

I create a Filter name TWS_JobEvent and I use that code.

if [program] == 'TWSQA523' {
grok {
match => { "message" => "%{NUMBER:TWSevent_number} %{HOSTNAME:TWSJobStream_WorkstationName} %{WORD:TWSJobStreamID} %{WORD:TWSJobName}"}
}
}

I generate a log and I receive it in Nagios Log Server but My 4 match don't work. If I understand good, I use %{NUMBER:TWSevent_number} to detect 101 digit in the message and put it on TWSevent_number field.


That is the example of the log in Nagios Log Server I want to parse with my filter. I want 101 in a field and other information too.

{
"_index": "logstash-2016.04.06",
"_type": "syslog",
"_id": "AVPr_KoyfjVMWWKOWOpC",
"_score": null,
"_source": {
"message": "101 SXQGBD0002 PY-CO-001-NOEL R3483_TWS0000A SXQGBD0002 3080388 5 R3483_TWS0000A jdeb7333 /jde/jdedwardsoneworld/maestro/PY/MAITRE_RUNUBE_SR\\040R3483_TWS0000A\\040CO-001 0 1 2016040600180000 0 \t+++\t 300 0 1459901776 0 705 0 61 PY-CO-001-NOEL 300 PY-CO-001-NOEL 2016040600050000 v.4 NONE NONE 4 NONE NONE NONE 0 0 0 0 0 NONE\n",
"@version": "1",
"@timestamp": "2016-04-06T14:31:57.000Z",
"type": "syslog",
"host": "172.16.1.50",
"priority": 133,
"timestamp": "Apr 6 10:31:57",
"logsource": "slqtws0523",
"program": "TWSQA523",
"severity": 5,
"facility": 16,
"facility_label": "local0",
"severity_label": "Notice"
},
"highlight": {
"program": [
"@start-highlight@TWSQA523@end-highlight@"
]
},
"sort": [
1459953117000,
1459953117000
]
}

Thank you!!!

[bennyboy]

When these logs are generated, are they generated as a single line, or that big log you just showed us? You're going to want to create a grok filter to help break the data down. If your logs don't have any consistent data or formatting, it's going to be nearly impossible to come up with a dynamic filter that works for you.

The log generale line by line.

[hsmith]

Have you tried to put your log in the grok debugger? This is a SUPER USEFUL(all caps, which makes my statement even more true) application for breaking your logs down.

https://grokdebug.herokuapp.com/

[hsmith]

Breaking down logs and making filters for you is not something we do, as it's time consuming, but I wanted to make one for you as an example so maybe it could help you better understand.

Here is the log message we are breaking down:

Code: Select all

101 SXQGBD0002 PY-CO-001-NOEL R3483_TWS0000A SXQGBD0002 3080388 5 R3483_TWS0000A jdeb7333 /jde/jdedwardsoneworld/maestro/PY/MAITRE_RUNUBE_SR\\040R3483_TWS0000A\\040CO-001 0 1 2016040600180000 0 \t+++\t 300 0 1459901776 0 705 0 61 PY-CO-001-NOEL 300 PY-CO-001-NOEL 2016040600050000 v.4 NONE NONE 4 NONE NONE NONE 0 0 0 0 0 NONE\n",

Here is the filter I created for this log message:

Code: Select all

%{POSINT:SysLogPrio} %{HOSTNAME:HostnameIThink} %{USERNAME:NotSureWhatThisOneIs} %{USERNAME:AnotherThing} %{WORD:AnotherThing2} %{POSINT:ANumber} %{POSINT:AnotherNumber} %{USERNAME:NoIdea} %{USERNAME:MaybeActuallyAUsername} %{PATH:APathIthink} %{INT:ANumber2} %{INT:ANumber3} %{INT:Timestamp} %{INT:ANumber4} %{NOTSPACE:TeesAndPlusses} %{INT:ANumber5} %{INT:ANumber6} %{INT:ANumber7} %{INT:ANumber8} %{INT:ANumber9} %{INT:ANumber10} %{INT:ANumber11} %{USERNAME:SomeThings} %{INT:ANumber12} %{USERNAME:ThisThingAgain} %{INT:AnotherTimeSTamp} %{NOTSPACE:VersionMaybe} %{GREEDYDATA:TheWordNoneALot}

Which will break it down into this:

Code: Select all

{
  "SysLogPrio": [
    [
      "101"
    ]
  ],
  "HostnameIThink": [
    [
      "SXQGBD0002"
    ]
  ],
  "NotSureWhatThisOneIs": [
    [
      "PY-CO-001-NOEL"
    ]
  ],
  "AnotherThing": [
    [
      "R3483_TWS0000A"
    ]
  ],
  "AnotherThing2": [
    [
      "SXQGBD0002"
    ]
  ],
  "ANumber": [
    [
      "3080388"
    ]
  ],
  "AnotherNumber": [
    [
      "5"
    ]
  ],
  "NoIdea": [
    [
      "R3483_TWS0000A"
    ]
  ],
  "MaybeActuallyAUsername": [
    [
      "jdeb7333"
    ]
  ],
  "APathIthink": [
    [
      "/jde/jdedwardsoneworld/maestro/PY/MAITRE_RUNUBE_SR\\\\040R3483_TWS0000A\\\\040CO-001"
    ]
  ],
  "UNIXPATH": [
    [
      "/jde/jdedwardsoneworld/maestro/PY/MAITRE_RUNUBE_SR\\\\040R3483_TWS0000A\\\\040CO-001"
    ]
  ],
  "WINPATH": [
    [
      null
    ]
  ],
  "ANumber2": [
    [
      "0"
    ]
  ],
  "ANumber3": [
    [
      "1"
    ]
  ],
  "Timestamp": [
    [
      "2016040600180000"
    ]
  ],
  "ANumber4": [
    [
      "0"
    ]
  ],
  "TeesAndPlusses": [
    [
      "\\t+++\\t"
    ]
  ],
  "ANumber5": [
    [
      "300"
    ]
  ],
  "ANumber6": [
    [
      "0"
    ]
  ],
  "ANumber7": [
    [
      "1459901776"
    ]
  ],
  "ANumber8": [
    [
      "0"
    ]
  ],
  "ANumber9": [
    [
      "705"
    ]
  ],
  "ANumber10": [
    [
      "0"
    ]
  ],
  "ANumber11": [
    [
      "61"
    ]
  ],
  "SomeThings": [
    [
      "PY-CO-001-NOEL"
    ]
  ],
  "ANumber12": [
    [
      "300"
    ]
  ],
  "ThisThingAgain": [
    [
      "PY-CO-001-NOEL"
    ]
  ],
  "AnotherTimeSTamp": [
    [
      "2016040600050000"
    ]
  ],
  "VersionMaybe": [
    [
      "v.4"
    ]
  ],
  "TheWordNoneALot": [
    [
      "NONE NONE 4 NONE NONE NONE 0 0 0 0 0 NONE\\n","
    ]
  ]
}

Please let me know if you have any questions.

[bennyboy]

Breaking down logs and making filters for you is not something we do, as it's time consuming, but I wanted to make one for you as an example so maybe it could help you better understand.

Here is the log message we are breaking down:

Code: Select all

101 SXQGBD0002 PY-CO-001-NOEL R3483_TWS0000A SXQGBD0002 3080388 5 R3483_TWS0000A jdeb7333 /jde/jdedwardsoneworld/maestro/PY/MAITRE_RUNUBE_SR\\040R3483_TWS0000A\\040CO-001 0 1 2016040600180000 0 \t+++\t 300 0 1459901776 0 705 0 61 PY-CO-001-NOEL 300 PY-CO-001-NOEL 2016040600050000 v.4 NONE NONE 4 NONE NONE NONE 0 0 0 0 0 NONE\n",

Here is the filter I created for this log message:

Code: Select all

%{POSINT:SysLogPrio} %{HOSTNAME:HostnameIThink} %{USERNAME:NotSureWhatThisOneIs} %{USERNAME:AnotherThing} %{WORD:AnotherThing2} %{POSINT:ANumber} %{POSINT:AnotherNumber} %{USERNAME:NoIdea} %{USERNAME:MaybeActuallyAUsername} %{PATH:APathIthink} %{INT:ANumber2} %{INT:ANumber3} %{INT:Timestamp} %{INT:ANumber4} %{NOTSPACE:TeesAndPlusses} %{INT:ANumber5} %{INT:ANumber6} %{INT:ANumber7} %{INT:ANumber8} %{INT:ANumber9} %{INT:ANumber10} %{INT:ANumber11} %{USERNAME:SomeThings} %{INT:ANumber12} %{USERNAME:ThisThingAgain} %{INT:AnotherTimeSTamp} %{NOTSPACE:VersionMaybe} %{GREEDYDATA:TheWordNoneALot}

Which will break it down into this:

Code: Select all

{
  "SysLogPrio": [
    [
      "101"
    ]
  ],
  "HostnameIThink": [
    [
      "SXQGBD0002"
    ]
  ],
  "NotSureWhatThisOneIs": [
    [
      "PY-CO-001-NOEL"
    ]
  ],
  "AnotherThing": [
    [
      "R3483_TWS0000A"
    ]
  ],
  "AnotherThing2": [
    [
      "SXQGBD0002"
    ]
  ],
  "ANumber": [
    [
      "3080388"
    ]
  ],
  "AnotherNumber": [
    [
      "5"
    ]
  ],
  "NoIdea": [
    [
      "R3483_TWS0000A"
    ]
  ],
  "MaybeActuallyAUsername": [
    [
      "jdeb7333"
    ]
  ],
  "APathIthink": [
    [
      "/jde/jdedwardsoneworld/maestro/PY/MAITRE_RUNUBE_SR\\\\040R3483_TWS0000A\\\\040CO-001"
    ]
  ],
  "UNIXPATH": [
    [
      "/jde/jdedwardsoneworld/maestro/PY/MAITRE_RUNUBE_SR\\\\040R3483_TWS0000A\\\\040CO-001"
    ]
  ],
  "WINPATH": [
    [
      null
    ]
  ],
  "ANumber2": [
    [
      "0"
    ]
  ],
  "ANumber3": [
    [
      "1"
    ]
  ],
  "Timestamp": [
    [
      "2016040600180000"
    ]
  ],
  "ANumber4": [
    [
      "0"
    ]
  ],
  "TeesAndPlusses": [
    [
      "\\t+++\\t"
    ]
  ],
  "ANumber5": [
    [
      "300"
    ]
  ],
  "ANumber6": [
    [
      "0"
    ]
  ],
  "ANumber7": [
    [
      "1459901776"
    ]
  ],
  "ANumber8": [
    [
      "0"
    ]
  ],
  "ANumber9": [
    [
      "705"
    ]
  ],
  "ANumber10": [
    [
      "0"
    ]
  ],
  "ANumber11": [
    [
      "61"
    ]
  ],
  "SomeThings": [
    [
      "PY-CO-001-NOEL"
    ]
  ],
  "ANumber12": [
    [
      "300"
    ]
  ],
  "ThisThingAgain": [
    [
      "PY-CO-001-NOEL"
    ]
  ],
  "AnotherTimeSTamp": [
    [
      "2016040600050000"
    ]
  ],
  "VersionMaybe": [
    [
      "v.4"
    ]
  ],
  "TheWordNoneALot": [
    [
      "NONE NONE 4 NONE NONE NONE 0 0 0 0 0 NONE\\n","
    ]
  ]
}

Please let me know if you have any questions.

Thank you very much! I chose to only detect the 4 first type. I did a test with grok debugger and here's what I suggest.

Code: Select all

\A%{POSINT:TWSEvent_number}%{SPACE}%{HOSTNAME:TWSSchedule_cpu}%{SPACE}%{USERNAME:TWSSchedule_id}%{SPACE}%{USERNAME:TWSJob_name}

Understand that sometimes the log contains one space but can also contain several space between Type.

Can you give me advice. thank you again for your help.

[hsmith]

I would change

Code: Select all

\A%{POSINT:TWSEvent_number}%{SPACE}%{HOSTNAME:TWSSchedule_cpu}%{SPACE}%{USERNAME:TWSSchedule_id}%{SPACE}%{USERNAME:TWSJob_name}

to

Code: Select all

\A%{POSINT:TWSEvent_number}% +%{HOSTNAME:TWSSchedule_cpu}% +%{USERNAME:TWSSchedule_id}% +%{USERNAME:TWSJob_name}

It accepts spaces, and the +<SPACE> will tell it one or more spaces.

Does this make sense?