Custom Log File

[77gm77]

Hi there,

I'm trying to import a custom log file to Nagios Log Server with no luck at all.

Admittedly I have no experience in doing this.

What I am trying to import:

A custom log file that contains 4 fields (generated by a script pulling down AWS CloudWatch metrics).

This log file looks like this:

Timestamp DimensionName DimensionValue CPUUtilisation
[04/Dec/2014:09:32:03] AutoScalingGroupName Prod-Tenders-Auto-Scaling-Group 0.0
[04/Dec/2014:09:33:03] AutoScalingGroupName Prod-Tenders-Auto-Scaling-Group 0.0
[04/Dec/2014:09:34:02] AutoScalingGroupName Prod-Tenders-Auto-Scaling-Group 3.73

I configured this using the "Linux File Setup" with the file tag "AWS_CLOUDWATCH", i.e.
setup-linux.sh -s 127.0.0.1 -p 5544 -f /var/blah/log -t AWS_CLOUDWATCH

These log entires do not show up at all when I view all events on the main dashboard.

I then tried configuring this as a 'Raw' input. These showed up but I could not search any of the fields (i.e. CPUUtilisation does not exist) and it was showing the _type as "import_raw"


Any pointers/tips on how to move forward would be much appreciated as we need to get this nailed befor we can implement it in our project. I have looked around quite a bit but cannot find any information/guides/documentation on what it is exactly I am trying to do (although it's highly possible I don't actually know what/where I'm supposed to be seeking this knowledge).


Cheers,
Gerard

[eloyd]

You need to write a (most likely) grok filter that separates out the text into fields. This is sorta beyond the scope of this forum, but you can get an idea of what you need to do at http://logstash.net/docs/1.4.2/filters/grok

[lgroschen]

77gm77,

Try going through the document eloyd provided as this is a great place to start. Also, check these documents out for help with filters and dashboards:

Configuring Nagios Log Server Inputs, Configuring Nagios Log Server Input Filters, Analyzing Logs With Nagios Log Server

I'll lock this post, but come back after you create your filters and let us know if you have any questions or need further assistance as we are happy to help!