NLS Log from file not working

[jolson]

I found what's causing the problem.

Code: Select all

 "Description", "Last Name", "First Name", "Node UID", "Node Name", "L ocation", "Reader", "Card Number"]

Code: Select all

<%{NUMBER:number}>%{MONTH:month} %{MONTHDAY:day} %{TIME:time} %{HOST:hostname} %{GREEDYDATA:something}: %{DATA:Partiti on}[_,]+%{DATE_EU:date2} %{HOUR:hour2}:%{MINUTE:minute2},%{DATE_EU:date3} %{HOUR:hour3}:%{MINUTE:minute3},%{DATA:Description},%{DATA:LastName},%{DATA:First Name},%{NOTSPACE:NodeUID},%{DATA:NodeName},%{DATA:Location},%{DATA:Reader},%{NUMBER:CardNumber}$

Do you see what I see?

Code: Select all

L ocation

needs to be

Code: Select all

Location

Code: Select all

DATA:Partiti on

needs to be

Code: Select all

DATA:Partition

Code: Select all

First Name

needs to be

Code: Select all

FirstName

I may have missed some. Please look over your configuration and ensure there's no whitespace that I may have missed.

[kconti]

Odd...I don't see any whitespace in my configuration. That might have just been a problem copying over to this support forum.

Code: Select all

filter {
        if [type] == 'csvinput' {
            csv {
                columns => ["Partition", "Person ID", "Node Date/Time", "Date/Time", "Description", "Last Name", "First Name", "Node UID", "Node Name", "Location", "Reader", "Card Number"]
                separator => ","
            }
          grok {
            match => [ "message", "^<%{NUMBER:number}>%{MONTH:month} %{MONTHDAY:day} %{TIME:time} %{HOST:hostname} %{GREEDYDATA:something}: %{DATA:Partition}[_,]+%{DATE_EU:date2} %{HOUR:hour2}:%{MINUTE:minute2},%{DATE_EU:date3} %{HOUR:hour3}:%{MINUTE:minute3},%{DATA:Description},%{DATA:LastName},%{DATA:FirstName},%{NOTSPACE:NodeUID},%{DATA:NodeName},%{DATA:Location},%{DATA:Reader},%{NUMBER:CardNumber}$" ]
          }
        }

[jolson]

In that case, can you follow post 10 on page 2? I included a bunch of my configuration information - could you compare your configuration to mine?

[kconti]

I matched up your configuration exatly...including the order of your inputs and filters. That still didn't work. However, I finally got the type "csvinput" to show up after I altered the /etc/rsyslog.conf

Original:
*.* @192.168.2.108:9001
*.* @192.168.2.108:5544

To this:
*.* @@192.168.2.108:9001
*.* @192.168.2.108:5544

I'm getting a few hits on the csvinput, which is definitely a step forward, but the data isn't coming over with it. I'm getting cvsinput parse and grok parse errors. I'll need to look into this more...

output_csvinput_ss.jpg

[jolson]

Can you try using the example file that you sent me to see whether or not you get the same results I am getting?

Install your example file on your server in /root/ and type:

Code: Select all

bash setup-linux.sh -s NLSIP -p 9001 -f /root/csvtest.csv -t csvtag

Attached is the file I am referring to.

[kconti]

Unfortunately, I tried that before my last post. Still having the parsing issues. I went back and even tried copying and pasting your input and filter once again to make sure I didn't accidentally mess up anything during the on and off testing.
Anything after "First Name" does not show up as a field in NLS.

Missing: Node UID, Node Name, Location, Reader, Card Number.

I went back to see if anything looked funny near the configuration for First Name and Node UID and it looked fine. I did try changing your "NOSPACE" to "DATA" and that still didn't manage to fix it. I think I just need to read up more on GROK as some parts of it just doesn't seem to make sense to me.

[jolson]

The filter that I defined for you was made very quickly, it's not optimal - could you show me a screenshot of what you are now receiving on Nagios Log Server?

I recommend reading the following post I made a little while ago for a better understanding of building your own custom filters: http://support.nagios.com/forum/viewtop ... 37&t=32221

[kconti]

Here are a couple screen shots. One of the limited fields list and the other of the current parse failed output.

output_parsefail_041615.jpg

limited hosts_ss.jpg

[kconti]

Would the parse fail if the field is empty? Some of the fields don't get populated by all...and some fields are not needed all together for our reporting such as "Person ID", "Node Date/Time", and Node UID.

[kconti]

Also, does the grok "match" need to be in the same order was the columns are listed, as they are not.