Import archived logs

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
lukedevon
Posts: 143
Joined: Sat Mar 24, 2018 9:15 am

Import archived logs

Post by lukedevon »

Hi

I want to import archived logs, (a couple of years old or a few months old) to nls. Those logs included a "Date & Time" field. Once I imported the data, I want to draw some graphs. How can I use "Date & Time" filed as the X-axis data?

Can you please share a guide for the same ?

Thank you
Luke.
User avatar
cdienger
Support Tech
Posts: 5045
Joined: Tue Feb 07, 2017 11:26 am

Re: Import archived logs

Post by cdienger »

The default "Events Over Time" graph on the dashboard will use the date and time as the x-axis so that part is easy.

For importing the logs you can use the shipper script. Details on using the script can be found in the NLS web interface under http://nls_ip/nagioslogserver/configure/source/import.

Parsing the data may be a bit trickier. For that we would need sample logs to create a filter, but I would strongly encourage anyone to try setting up the filters themselves after reviewing https://support.nagios.com/kb/article/n ... ew-98.html. The "Logstash Example Configuration" section specifically steps through setting up custom filters.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
lukedevon
Posts: 143
Joined: Sat Mar 24, 2018 9:15 am

Re: Import archived logs

Post by lukedevon »

Hi

Can you please help me to insert this type of data to nls;

Param1 : Value1_Text
Date & Time: 2017-09-24 08:57:23.376
Param2: Integer
Param3: Value3_text
Param4: Value4_text


How can I write an input filter? Once inserted data, I should be able to get the Date & Time for the x-axis.

Thank you
Luke.
User avatar
cdienger
Support Tech
Posts: 5045
Joined: Tue Feb 07, 2017 11:26 am

Re: Import archived logs

Post by cdienger »

The shipper script and instructions use the default input "Import Files - JSON (Default)". Make sure that this is enabled under Configure > Global (All Instances) > Global Config > Inputs. The config looks like:

Code: Select all

tcp {
    type => 'import_json'
    tags => 'import_json'
    port => 2057
    codec => json
}
The next part is parsing the data. The KB linked to in the previous post goes through configuring a custom filter. It's a good exercise to go through and try to set this up, but if you need assistance with configuring a filter, then we would need a copy of the actual log to be imported.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
lukedevon
Posts: 143
Joined: Sat Mar 24, 2018 9:15 am

Re: Import archived logs

Post by lukedevon »

Hi

I have found articles and questions in the forum that some of the users have tried to inject txt based logs to nls. However, I don't need to pass my logs to syslog. I need a separate pipe to send them to nls.

https://support.nagios.com/kb/article/n ... g-727.html

https://support.nagios.com/forum/viewto ... 1532419112

what is the best way to input them to nls?

I am sorry I am unable to share the log over the forum, so I have PM the logs to you directly.

Thank you
Luke.
User avatar
cdienger
Support Tech
Posts: 5045
Joined: Tue Feb 07, 2017 11:26 am

Re: Import archived logs

Post by cdienger »

The logs can be imported using the tcp input and the multiline codec. Using one of the sample logs for example:

Code: Select all

tcp {
    type => 'import_raw'
    tags => 'import_raw'
    port => 2056
codec => multiline {
      pattern => "^SETTLEMENT"
      negate => "true"
      what => "previous"
}
}
Another similar input would need to be configured for the other logs that do not follow this pattern. Logs can be imported using a telnet command:

cat alert.txt | nc nls_ip 2056
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
lukedevon
Posts: 143
Joined: Sat Mar 24, 2018 9:15 am

Re: Import archived logs

Post by lukedevon »

Hi

I tried to follow your instructions but still, I couldn't import the logs correctly. I can see logs are been imported. In the dashboard - each having a separate row.

===================================================

I will give a sample of log data that I want to import to nls;

For example;

CompanyName: XYZ Pvt Ltd
Date & Time: 2018-09-28 00:03:47.312
Some value: xxx
Some text: abcd
So on, it has multiple lines
END:

CompanyName: ABC Pvt Ltd
Date & Time: 2018-09-28 00:02:20.312
Some value: xxx
Some text: abcdddd
So on, it has multiple lines
END:

As per the example --> starting point will be "CompanyName:" and the end point is "END:".

Cant, we make such a start and end point to import a block of data to nls?

As I understood, if the block of data properly imported, filtering is quite flexible. Can you please help me?

Thank you
Luke.
lukedevon
Posts: 143
Joined: Sat Mar 24, 2018 9:15 am

Re: Import archived logs

Post by lukedevon »

Further to the previous question;

Once imported the logs to nls, how can I get the "Date & Time:" field to the x-axis? Based on the "Date & Time: I wanna draw the rest of the graphs.

Thanks
Luke
User avatar
cdienger
Support Tech
Posts: 5045
Joined: Tue Feb 07, 2017 11:26 am

Re: Import archived logs

Post by cdienger »

My previous post had a typo. To import try:

cat alert.txt | nc nls_ip 2056

This along with an input that looks like:

Code: Select all

tcp {
    type => 'import_raw'
    tags => 'import_raw'
    port => 2056
codec => multiline {
      pattern => "^CompanyName"
      negate => "true"
      what => "previous"
}
}
should allow you to import these events.

For your follow up question, the graphs will display the date and time an event came in on the x-axis already. Are you trying to replace @timestamp with 2018-09-28 00:03:47.312 from your example? If so, I'll need to look into that one.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
lukedevon
Posts: 143
Joined: Sat Mar 24, 2018 9:15 am

Re: Import archived logs

Post by lukedevon »

Hi,

Thank you so much. It helped me to import data to nls correctly.

Now I can filter as I wish by selecting the fields in the block of data. As I stated, I need to get the "Date & Time" to X-Axis to draw some graphs. (Bar charts/Pie Charts/ and take a count of no.of occurrences of some events)

For example:-

Date & Time: 2017-10-11 06:06:27.189

I need only this part --> 2017-10-11 (YYYY-MM-DD) as the X-Axis date. How can it be done? Do we need to write GROCK filter or something like that?

Regards
Luke
Locked