NxLog locking file

[WillemDH]

Hello,

I'm receiving complaints about NxLog locking files it is monitoring. This is the nxlog.conf file:

Code: Select all

define ROOT C:\Program Files (x86)\nxlog
define CERT %ROOT%\cert

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

# Include fileop while debugging, also enable in the output module below
#<Extension fileop>
#	 Module xm_fileop
#</Extension>

<Extension json>
	Module xm_json
</Extension>

<Extension syslog>
	Module xm_syslog
</Extension>

<Extension multi>
	Module xm_multiline
	HeaderLine /ERROR|DEBUG|WARN|INFO|TRACE/
</Extension>

<Input internal>
	Module im_internal
</Input>

# Watch your own files
<Input file1>
	Module im_file
	File '%ROOT%\data\nxlog.log'
	SavePos TRUE
</Input>

<Input filetomcatjob>
	Module im_file
	File 'M:\Software\Apache Software Foundation\Tomcat 7.0\logs\job.log'
	SavePos TRUE
	Exec $Message = $raw_event;
	InputType multi
</Input>

<Input filetomcatportal>
	Module im_file
	File 'M:\Software\Apache Software Foundation\Tomcat 7.0\logs\portal.log'
	SavePos TRUE
	Exec $Message = $raw_event;
	InputType multi
</Input>

# Windows Event Log
<Input eventlog>
	# Uncomment im_msvistalog for Windows Vista/2008 and later
	Module im_msvistalog
	# Uncomment im_mseventlog for Windows XP/2000/2003
	# Module im_mseventlog
</Input>

<Output out>
    Module      om_tcp
    Host        ipnls
    Port        3515
	Exec $tmpmessage = $Message; delete($Message); rename_field("tmpmessage","message");
    Exec $raw_event = to_json();
    # Uncomment for debug output
    # Exec file_write('%ROOT%\data\nxlog_output.log', $raw_event + "\n");
</Output>

<Output outfiletomcatjob>
    Module      om_tcp
    Host        nlsip
    Port        5550
	Exec $tmpmessage = $Message; delete($Message); rename_field("tmpmessage","message");
    Exec $raw_event = to_json();
    # Uncomment for debug output
    # Exec file_write('%ROOT%\data\nxlog_output.log', $raw_event + "\n");
</Output>

<Output outfiletomcatportal>
    Module      om_tcp
    Host        nlsip
    Port        5551
	Exec $tmpmessage = $Message; delete($Message); rename_field("tmpmessage","message");
    Exec $raw_event = to_json();
    # Uncomment for debug output
    # Exec file_write('%ROOT%\data\nxlog_output.log', $raw_event + "\n");
</Output>

<Route 1>
    Path internal, file1, eventlog => out
</Route>

<Route 2>
    Path filetomcatjob => outfiletomcatjob
</Route>

<Route 2>
    Path filetomcatportal => outfiletomcatportal
</Route>

The application will try to rename the job.log to job.log.date , as you can see in the screenshot.

But as apparently NxLog locks the job.log file, an error is received while trying to rename the file job.log => "The action can't be completed because the file is open in nxlog."

This causes our application to stop logging it's job info... As this is a highly critical application, I have commented out the file inputs, outputs and routes for now.

Any help preventing the above issue from happening is welcome.

Edit:

Found this https://www.mail-archive.com/nxlog-ce-u ... 00261.html Seems like there are two solutions:
1) Monitor the renamed file
How would I have to rename "File 'M:\Software\Apache Software Foundation\Tomcat 7.0\logs\job.log'" so it always take the day before the current day?

Code: Select all

<Input filetomcatjob>
	Module im_file
	File 'M:\Software\Apache Software Foundation\Tomcat 7.0\logs\job.log'
	SavePos TRUE
	Exec $Message = $raw_event;
	InputType multi
</Input>

2) Scheduled import
How could I schedule this import with NxLog? I can't find any immediate example of how to do such a thing.

Grtz

Willem

[jolson]

Hi Willem,

As you've read in the mailing list:

There are read locks too, Exclusive-Mode. So reading a file can cause it from being written to. Exclusive-Mode can also cause issues on writing the file, so it is possibly the application is wanting exclusive-mode to write the logs. My suggestion is to schedule the importing of the logs, instead of actively reading them during automation runs.

Is it possible that your application desires exclusive mode access to these logs? If so, could you turn exclusive mode off?


The primary problem is that any sort of open/read may block your application from logging - this includes duplicating the logs, importing them, etc. My recommendation above is likely the most appropriate way to address your issue.

If that's not possible, you could set up log rotation within your application - have it rotate the logs with distinct filenames (jobrotated-date.log) and nxlog could look at *only* those files:

Code: Select all

    <Input filetomcatjob>
       Module im_file
       File 'M:\Software\Apache Software Foundation\Tomcat 7.0\logs\jobrotated*.log'
       SavePos TRUE
       Exec $Message = $raw_event;
       InputType multi
    </Input>

[WillemDH]

Hmm this some kind of Tomcat job. The issue arises the moment Tomcat is trying to rotate the logs I fear. So log rotation is already in place.. Monitoring the rotated file would imply we are monitoring 'old data'. As you can see in the screenshot, the file is called job.log.yyyy-MM-dd

with yyyy-MM-dd being the day before today. How can I make NxLog monitor this 'yesterday' logfile as the appended date is different every day?

[jolson]

You could do something like:

Code: Select all

<Input filetomcatjob>
  Module im_file
  File 'M:\Software\Apache Software Foundation\Tomcat 7.0\logs\job.log.????-??-??'
  SavePos TRUE
  Exec $Message = $raw_event;
  InputType multi
</Input>

This way nxlog will monitor only the rotated logs, and not the normal 'jobs.log' file - the '?' special character means any one character - so it will pick up on any date that fits the above format. Like you said, the logs would be a full day behind. Perhaps it's possible to decrease the rotation period to hourly?

Maybe the exclusive lock only happens when the log rotates - in which case you could set a scheduled task to stop nxlog a few minutes before the rotation occurs, and start nxlog after the rotation happens?