Insert data with curl

[WillemDH]

Hello,

For testing purposes i'd like to find the best way to insert data as json into NLS with curl. Do I need to specify the index I'm putting it into?

This is the json output of a log that I want to grok, but as the error only comes in every 4-6 hours, it's kind of hard to test. Also because the message field is multiline the usual web grok debugggers don't seem to work..

I'm guessing Iwill have to omit the fields starting with '_'?

Code: Select all

{
  "_index": "logstash-2016.10.09",
  "_type": "eventlog",
  "_id": "AVeoRyRSfSZthwLITZsU",
  "_score": null,
  "_source": {
    "message": "Logon Failure on database \"SG Gebruikers S1 [DB1-29]\" - Windows account DOMAIN\\exch01$; mailbox /o=zdzazdazt/ou=Administratie/cn=Recipients/cn=zadzazd.\r\nError: 1245 \r\nClient Machine: exch01 \r\nClient Process: edgetransport.exe \r\nClient ProcessId: 0 \r\nClient ApplicationId: Client=Hub Transport ",
    "@version": "1",
    "@timestamp": "2016-10-09T07:10:16.680Z",
    "host": "10.54.28.110",
    "type": "eventlog",
    "category": "Logons",
    "channel": "Application",
    "eventid": 1022,
    "hostname": "exch01",
    "keywords": 36028797018963970,
    "processid": 0,
    "recordnumber": 40930624,
    "severity_label": "error",
    "severity": 4,
    "sourcemodulename": "eventlog",
    "sourcename": "MSExchangeIS Mailbox Store",
    "task": 16,
    "threadid": 0,
    "opcode": null,
    "logsource": "exch01",
  },
  "sort": [
    1475997016680,
    1475997016680
  ]
}

I found this as a method to insert json data:

Code: Select all

curl -XPOST "http://localhost:9200/indexname/typename/optionalUniqueId" -d "{ \"field\" : \"value\"}"

Some help to make this work in NLS would be appreciated.

Fyi, this is the grok filter I have now (for the message field):

Code: Select all

if [sourcename] == "MSExchangeIS Mailbox Store" {
    grok {
        match => [ "message", "(?m)%{GREEDYDATA:info1}Error: %{NUMBER:exchange_error} \nClient Machine: %{HOSTNAME:exchange_client} \nClient Process: %{HOSTNAME:exchange_processname} \nClient ProcessId: %{NUMBER:exchange_processid} \nClient ApplicationId: %{GREEDYDATA:exchange_applicationid}
" ]
    }
    mutate {
      add_tag => "mutated_msechangeis_mailbox_store"  
    }
}

But somehow I'm getting parsing failures. If I find a way to duplicate / re-insert this event with curl development on my grok filters would go faster.

Willem

[rkennedy]

Just to clarify - the JSON you posted looks like it was taken from NLS directly. Are you just trying to parse the message field properly using grok?

Do you have an example of the exact JSON you're sending from the client side for us to look at?

[WillemDH]

Just to clarify - the JSON you posted looks like it was taken from NLS directly

Correct. This is exactly what i'm trying to do.

No I don't have an example of the exact JSON I'm posting as it was sent by NxLog.

Could you just give me an example how to put a json structure in NLS with a curl statement? Please make it have a message field like this:

Code: Select all

"message": "Logon Failure on database \"SG Gebruikers S1 [DB1-29]\" - Windows account DOMAIN\\exch01$; mailbox /o=zdzazdazt/ou=Administratie/cn=Recipients/cn=zadzazd.\r\nError: 1245 \r\nClient Machine: exch01 \r\nClient Process: edgetransport.exe \r\nClient ProcessId: 0 \r\nClient ApplicationId: Client=Hub Transport "

Willem

[mcapra]

The tricky thing is that, by circumventing logstash, you lose out on any sort of filtering, timestamp reconciliation, and other sorts of things logstash does to tidy the data up before inserting it into elasticsearch.

Here's what I did against my testing machine using the elasticsearch API:

Code: Select all

curl -XPUT 'http://localhost:9200/logstash-2016.10.11/syslog/144' -d '{"message" : "fnkqrrm"}'

Which works to insert the record with a "message" into elasticsearch, but since it has no timestamp set (as well as other keys not set) I can't search it via the NLS GUI. I could delve into the depths of the elasticsearch API and find an exact way to insert into the correct index while simultaneously generating the neccecary key fields, but that's more trouble than it's worth in my opinion.

Strictly speaking, if I had an arbitrarily structured bit of JSON that I wanted to get into NLS in a meaningful way, I would recommend going through logstash. Lets say I have the following JSON structure:

Code: Select all

{
  "user": "Jerry Smith",
  "superior": "Rick Sanchez",
  "dimension": "C-137",
  "postDate": "2009-11-15T14:12:12",
  "allergies": {
    "fun": "true",
    "peanuts": "true",
    "lemon": "false"
  },
  "message": "Logon Failure on database \"SG Gebruikers S1 [DB1-29]\" - Windows account DOMAIN\\exch01$; mailbox /o=zdzazdazt/ou=Administratie/cn=Recipients/cn=zadzazd.\r\nError: 1245 \r\nClient Machine: exch01 \r\nClient Process: edgetransport.exe \r\nClient ProcessId: 0 \r\nClient ApplicationId: Client=Hub Transport"
}

Using the default JSON input rule on port 2057, I can send this to logstash programatically using something like netcat (or anything that can write TCP/UDP packets):

Code: Select all

nc -w 1 192.168.67.200 2057 <<< '{"user":"Jerry Smith","superior":"Rick Sanchez","dimension":"C-137","postDate":"2009-11-15T14:12:12","allergies":{"fun":"true","peanuts":"true","lemon":"false"},"message":"Logon Failure on database \"SG Gebruikers S1 [DB1-29]\" - Windows account DOMAIN\\exch01$; mailbox /o=zdzazdazt/ou=Administratie/cn=Recipients/cn=zadzazd.\r\nError: 1245 \r\nClient Machine: exch01 \r\nClient Process: edgetransport.exe \r\nClient ProcessId: 0 \r\nClient ApplicationId: Client=Hub Transport"}'

And I get the following record in the NLS GUI:

2016_10_11_16_09_01_Dashboard_Nagios_Log_Server.png

So in short, it's much easier to ship things to logstash rather than try to hit the elasticsearch/NLS API directly.

[WillemDH]

Ok, thanks mcapra. I'll look into your suggestions. (could take some time)

[mcapra]

Let us know if you have additional questions! I did something similar with Java Datagrams a few months back.

[nagiosnl_jorgen]

Nice example mcapra! I had to instruct netcat to use TCP instead of UDP as our NLS installation did not ship with an input "Import Files - JSON (Default)" listening on UDP. So I removed netcat option "-u"

Best regards, Jørgen van der Meulen

[mcapra]

Thanks for mentioning that Jørgen! Would definitely lead to some confusions using the default JSON input