Unable to monitor log file on Windows Server

[krobertson71]

The standard Windows log file monitoring is working great.

However, I am trying to test monitoring a cognos log file. I created the log on the remote server in C:\cognos\cognos.log. I edited the nxlog.conf file according to instructions. Restart nxlog. Nothing.

No errors on remote server in nxlog.log, no errors in Logstash log file on NLS. Nothing that I can see. So here is my nxlog.conf file on the remote system and the test log I am trying to monitor.

nxlog.conf

Code: Select all

    ## See the nxlog reference manual at
    ## http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html

    ## Please set the ROOT to the folder your nxlog was installed into,
    ## otherwise it will not start.
    #define ROOT C:\Program Files\nxlog
    define ROOT C:\Program Files (x86)\nxlog
    define CERT %ROOT%\cert

    Moduledir %ROOT%\modules
    CacheDir %ROOT%\data
    Pidfile %ROOT%\data\nxlog.pid
    SpoolDir %ROOT%\data
    LogFile %ROOT%\data\nxlog.log

    # Include fileop while debugging, also enable in the output module below
    #<Extension fileop>
    #    Module xm_fileop
    #</Extension>

    <Extension json>
        Module      xm_json
    </Extension>

    <Extension syslog>
        Module xm_syslog
    </Extension>

    <Input internal>
        Module im_internal
    </Input>

    # Watch your own files
     <Input file1>
         Module   im_file
         File     'C:\cognos\cognos.log'
         SavePos  TRUE
	 Exec	  $Message = $raw_event;
     </Input>

    # Windows Event Log
    <Input eventlog>
    # Uncomment im_msvistalog for Windows Vista/2008 and later
        Module im_msvistalog
    # Custom query config to only pull the System and Application logs, excluding the Security log.
       Query    <QueryList>\
                       <Query Id="0">\
                         <Select Path="System">*</Select>\
                <Select Path="Application">*</Select>\
                        </Query>\
            </QueryList>
       
       


    # Uncomment im_mseventlog for Windows XP/2000/2003
    #   Module im_mseventlog
    </Input>

    <Output out>
        Module om_tcp
        Host nagilgp01.dcri.duke.net
        Port 3515
       
        Exec  $tmpmessage = $Message; delete($Message); rename_field("tmpmessage","message");
        Exec  $raw_event = to_json();
       
       # Uncomment for debug output
       # Exec file_write('%ROOT%\data\nxlog_output.log', $raw_event + "\n");
    </Output>

    <Route 1>
        Path internal, file1, eventlog => out
    </Route>

Test congos log file

Code: Select all

10.136.134.45:9443 4812 2014-12-05 11:04:24.709 -5 D4C8F95AB338E736FF18E41032BDEEDCC00D34DD yqwwGslqv8dGyjysjvy4yjG84lvdv8j4v88wjhC2 qq9v89wl82vssC8CClqwh4v24C9wMy9l9h9yM8j9 4488 RSVP 6039 1 Audit.RTUsage.RSVP Response BatchReportService Failure RSV-SRV-
10.136.134.45:9443 4813 2014-12-05 12:04:24.705 -5 b4C8F95AB338E736FF18E41032BDEEDCC00D34DD cqwwGslqv8dGyjysjvy4yjG84lvdv8j4v88wjhC2 qq9v89wl82vssC8CClqwh4v24C9wMy9l9h9yM8j8 4488 RSVP 6039 1 Audit.RTUsage.RSVP Response BatchReportService Failure RSV-SRV-

[krobertson71]

Turned on debugging and got this on the remote server log:

Code: Select all

2015-01-02 15:54:28 ERROR Couldn't parse Exec block at C:\Program Files (x86)\nxlog\conf\nxlog.conf:69; couldn't parse statement at line 69, character 89 in C:\Program Files (x86)\nxlog\conf\nxlog.conf; procedure 'file_write()' does not exist or takes different arguments
2015-01-02 15:54:28 ERROR module 'out' has configuration errors, not adding to route '1' at C:\Program Files (x86)\nxlog\conf\nxlog.conf:73
2015-01-02 15:54:28 ERROR route 1 is not functional without output modules, ignored at C:\Program Files (x86)\nxlog\conf\nxlog.conf:73
2015-01-02 15:54:28 WARNING no routes defined!
2015-01-02 15:54:28 WARNING not starting unused module internal
2015-01-02 15:54:28 WARNING not starting unused module file1
2015-01-02 15:54:28 WARNING not starting unused module eventlog
2015-01-02 15:54:28 WARNING not starting unused module out
2015-01-02 15:54:28 INFO nxlog-ce-2.8.1248 started

Weird to me as all Application and System logs are coming in just fine.

Actually after looking at the line it was referring to, it is not likeing the debug line itself.

[krobertson71]

Remote Host is Windows Server

Actually I am pretty sure the problem is with the hand created log file. I pointed the im_file to another log on the remote host and it came into NLS just fine, well not fine it dumped everything into the message field.

Taking the same event from above:

Code: Select all

10.136.134.45:9443 4812 2014-12-05 11:04:24.709 -5 D4C8F95AB338E736FF18E41032BDEEDCC00D34DD yqwwGslqv8dGyjysjvy4yjG84lvdv8j4v88wjhC2 qq9v89wl82vssC8CClqwh4v24C9wMy9l9h9yM8j9 4488 RSVP 6039 1 Audit.RTUsage.RSVP Response BatchReportService Failure RSV-SRV-

How would I filter this in NLS? Those random numbers/letters are actual cognost IDs. I know how filter works but in NLS where would I even start to indentify this as it's coming in under the Eventlog Input.

All the examples I read start with "If App=this" but here this is just coming in raw jason.

Basically what I am trying to figure out is:

1) Do I need to add something to the nxlog.conf file on the remote host side to add a tag or field called APP or something to add a unique identifier?

2) is there something I can do on the NLS side to achieve the same thing?

I want the event in NLS to show at least that the source of the file is Cognos and be able to have fields for the different id's and not have the time date, whis is after the ip and port in this example, be in the message field.

I have used the grok filter site and I can create a pattern that works, but not sure what to do with that pattern on the NLS side and make it associate with this event.

[scottwilkerson]

I'm not sure why your output filter is complaining about a line that is commented out, but can you just remove line 69 and restart the service

Code: Select all

# Exec file_write('%ROOT%\data\nxlog_output.log', $raw_event + "\n");

[krobertson71]

Sorry took screenshot after I tested debugging. I removed the # and restarted the agent, that is when the error about debugging was generated.

I am more concerned about my last post dealing with the message in question and how I would handle that type of event formated that way.

Please refer the post by me before this one.

[sreinhardt]

Morning Kris, I'd suggest taking a look at my IIS dashboard. It's a little different than what you are doing, as I choose to create a new output within nxlog, so that I could mangle the message a bit on the sending side, and clearly differentiate it from the standard event logs. Once those logs get to logstash, where without the filter they would also be dumped into the main message body, it knows that there is a specific port for these requests and how to tag them. This could be handled on the nxlog side of things by adding an additional tag there, and filtering input for that instead of using ports. Finally it hits my grok filter, that looks for the IIS_request type, and uses the filter setup to parse out into proper fields. This is how you will have to do it in most cases, aside from the port aspect. I choose to use ports instead of nxlog tags, as it was easier at the time due to how specific event logs incoming have to be.

http://exchange.nagios.org/directory/Ad ... rd/details

[krobertson71]

Afternoon Commi

I was working on doing it from the nxlog.conf side as well but didn't think about it sending it to another port. I can see how that would make it easier than letting it come in the default eventlog port that the config comes with as it applies a charset to the input.

Need some documentation around things like this for NLS. A lot of the users here might not be familiar with the ELK stack, or have a limited knowledge base like me, and need extra info like this.

What is up with the double \\ in your file paths. I didn't have to do that for other windows logs? Do I need to be doing this if I am not using the evenlog input?


Thanks for getting me on the right track!!

[cmerchant]

The \\ is escaping the \ in the regex expression for the input filename. Let us know if we can go ahead and close this issue. Thanks.

[krobertson71]

Yes go ahead and close it. I think I have enough information to work through it. If I get stuck along the way I will open another topic.

I am knee deep in Nagios XI deployment that I can't look at this everyday right now anyway.

[cmerchant]

We'll go ahead and close this thread. Thanks.