NXLOG - Rotating Logs Tracking / Wildcard folder

[polarbear1]

Greetings. Got a nxlog related question here that my google-fu isn't helping with.

I have a folder full of logs that have dynamic filenames that rotate out at various intervals.

Up to this point I've been using nxlog to track one specific file and I've been using the following directive:

Code: Select all

<Input mylogfile>
    Module   im_file
    File     'C:\Path\To\My\Logs\mylogfile.log'
    SavePos  TRUE
    Exec     $Message = $raw_event;
</Input>

<Input mylogfile2>
    Module   im_file
    File     'C:\Path\To\My\Logs\mylogfile2.log'
    SavePos  TRUE
    Exec     $Message = $raw_event;
</Input>

Also so on the dashboard, the SourceModuleName field reflects the name given in the <Input > tag.

If I have a wildcard situation, how do I distinguish which file the particular message came from. If I use the following directive, wouldn't all the messages found through this match appear as if they came from the same place? Is there a function (something in the Exec line, perhaps) which would capture the actual file name and send it as some field to Elasticsearch?

Code: Select all

<Input mylogs>
    Module   im_file
    File     'C:\Path\To\My\Logs\*.log'
    SavePos  TRUE
    Exec     $Message = $raw_event;
</Input>

And on a related note - how is nxlog's behavior with wildcard folders? I've read some threads I found through the googles where it would re-read old logs if no changes were made to that file - it would just continuously churn through the folder. I need to not get confused when files enter and leave the directory.

EDIT - I guess technically what I am asking is how do I fill the "program" field with nxlog from the filename while using a wildcard?

Thanks.

[rkennedy]

The im_file module actually allows you to use file_name() to decipher where it's coming from. You should be able to use something like this -

Code: Select all

    <Input file1>
        Module   im_file
        File     '%ROOT%\data\test\\*.log'
        SavePos  TRUE
        Exec                   $Message = $raw_event; \
                               $Program = file_name();
    </Input>

Which, in NLS will create a field called 'Program', and tells you where the log came from. Here's a screenshot -

nxlog-custom-field.PNG

And on a related note - how is nxlog's behavior with wildcard folders? I've read some threads I found through the googles where it would re-read old logs if no changes were made to that file - it would just continuously churn through the folder. I need to not get confused when files enter and leave the directory

I believe NXlog will pick up on anything new coming into the folder, so if you have things shifting this could cause issues with multiple logs happening. Are you just referring to log rotation?

[polarbear1]

That is pretty much exactly what I was looking for, thanks.

For Program being the whole file path is a bit wordy though and makes the dashboard look busy. Is there a way to whittle it down to just the file name? (ie. C:/My/File/Path/File.log ---> File.log)

[hsmith]

I believe you can use Regex to cut it down.

[polarbear1]

I believe you can use Regex to cut it down.

In the interest of not sending future folks down this rabbit hole, there is a more elegant solution.

Use this directive in nxlog:

Code: Select all

Exec $filename = file_basename(file_name());

That chops off the path, leaving just the "filename.log"

Also fairly certain this thread has run it's course, so feel free to lock it up.

[mcapra]

Thanks for sharing your solution! NXLog has fairly solid documentation for achieving specific formatting of the logs it sends:
https://nxlog.org/documentation/nxlog-c ... ual-v20928

Closing this up