Unable to send simple logfile to NLS

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
Locked
User avatar
WillemDH
Posts: 2320
Joined: Wed Mar 20, 2013 5:49 am
Location: Ghent
Contact:
Contact WillemDH

Unable to send simple logfile to NLS

Post by WillemDH »

Hello,

I've been struggling to send a logfile to NLS with NxLog. The logfile is called job.log and is a part of Tomcat. It looks like this:

Code: Select all

DEBUG 2015-05-12 12:20:55,012 [schedulerFactoryBean_Worker-2] jobLogger - Processing instruction 1 / 1
DEBUG 2015-05-12 12:20:55,106 [schedulerFactoryBean_Worker-2] jobLogger - PERMISSIONS 8ab698f74c548787014c5613ce9112f3 : Set permissions for decree (took 63 ms)
DEBUG 2015-05-12 12:20:55,106 [schedulerFactoryBean_Worker-2] jobLogger - Job Job id:8ab6982f4d46d5c5014d4788d89802bd, type:SET_PERMISSION completed
DEBUG 2015-05-12 12:21:00,020 [schedulerFactoryBean_Worker-2] jobLogger - Job Job id:8ab6982f4d46d5c5014d4788d8a702be, type:SET_PERMISSION fetched. Execute...
INFO  2015-05-12 12:21:00,020 [schedulerFactoryBean_Worker-2] jobLogger - Return implementation of JobProcessor: setPermissionJobProcessor
DEBUG 2015-05-12 12:21:00,020 [schedulerFactoryBean_Worker-2] jobLogger - Processing instruction 1 / 1
DEBUG 2015-05-12 12:21:00,129 [schedulerFactoryBean_Worker-2] jobLogger - PERMISSIONS 8ab698f74c548787014c55dcfb5a102d : Set permissions for decree (took 78 ms)
DEBUG 2015-05-12 12:21:00,129 [schedulerFactoryBean_Worker-2] jobLogger - Job Job id:8ab6982f4d46d5c5014d4788d8a702be, type:SET_PERMISSION completed
DEBUG 2015-05-12 12:21:05,012 [schedulerFactoryBean_Worker-2] jobLogger - Job Job id:8ab6982f4d46d5c5014d4788d8a702bf, type:SET_PERMISSION fetched. Execute...
INFO  2015-05-12 12:21:05,012 [schedulerFactoryBean_Worker-2] jobLogger - Return implementation of JobProcessor: setPermissionJobProcessor
DEBUG 2015-05-12 12:21:05,012 [schedulerFactoryBean_Worker-2] jobLogger - Processing instruction 1 / 1
DEBUG 2015-05-12 12:21:05,121 [schedulerFactoryBean_Worker-2] jobLogger - PERMISSIONS 8ab698f74c548787014c55e038bf104a : Set permissions for decree (took 78 ms)
DEBUG 2015-05-12 12:21:05,121 [schedulerFactoryBean_Worker-2] jobLogger - Job Job id:8ab6982f4d46d5c5014d4788d8a702bf, type:SET_PERMISSION completed
DEBUG 2015-05-12 12:21:10,019 [schedulerFactoryBean_Worker-2] jobLogger - Job Job id:8ab6982f4d46d5c5014d4788d8a702c0, type:SET_PERMISSION fetched. Execute...
INFO  2015-05-12 12:21:10,019 [schedulerFactoryBean_Worker-2] jobLogger - Return implementation of JobProcessor: setPermissionJobProcessor
DEBUG 2015-05-12 12:21:10,019 [schedulerFactoryBean_Worker-2] jobLogger - Processing instruction 1 / 1
DEBUG 2015-05-12 12:21:10,144 [schedulerFactoryBean_Worker-2] jobLogger - PERMISSIONS 8ab698f74c4f636e014c5082dd4b0d0f : Set permissions for decree (took 78 ms)
DEBUG 2015-05-12 12:21:10,144 [schedulerFactoryBean_Worker-2] jobLogger - Job Job id:8ab6982f4d46d5c5014d4788d8a702c0, type:SET_PERMISSION completed
DEBUG 2015-05-12 12:21:15,011 [schedulerFactoryBean_Worker-2] jobLogger - Job Job id:8ab6982f4d46d5c5014d4788d8a702c1, type:SET_PERMISSION fetched. Execute...
INFO  2015-05-12 12:21:15,011 [schedulerFactoryBean_Worker-2] jobLogger - Return implementation of JobProcessor: setPermissionJobProcessor
DEBUG 2015-05-12 12:21:15,011 [schedulerFactoryBean_Worker-2] jobLogger - Processing instruction 1 / 1
DEBUG 2015-05-12 12:21:15,121 [schedulerFactoryBean_Worker-2] jobLogger - PERMISSIONS 8ab698f74c59b03c014c5aa1bf070923 : Set permissions for decree (took 63 ms)
DEBUG 2015-05-12 12:21:15,136 [schedulerFactoryBean_Worker-2] jobLogger - Job Job id:8ab6982f4d46d5c5014d4788d8a702c1, type:SET_PERMISSION completed
DEBUG 2015-05-12 12:21:20,019 [schedulerFactoryBean_Worker-2] jobLogger - Job Job id:8ab6982f4d46d5c5014d4788d8a702c2, type:SET_PERMISSION fetched. Execute...
INFO  2015-05-12 12:21:20,019 [schedulerFactoryBean_Worker-2] jobLogger - Return implementation of JobProcessor: setPermissionJobProcessor
DEBUG 2015-05-12 12:21:20,019 [schedulerFactoryBean_Worker-2] jobLogger - Processing instruction 1 / 1
DEBUG 2015-05-12 12:21:20,128 [schedulerFactoryBean_Worker-2] jobLogger - PERMISSIONS 8ab698f74c6e48b8014c6f5fcf370c26 : Set permissions for decree (took 78 ms)
DEBUG 2015-05-12 12:21:20,128 [schedulerFactoryBean_Worker-2] jobLogger - Job Job id:8ab6982f4d46d5c5014d4788d8a702c2, type:SET_PERMISSION completed
DEBUG 2015-05-12 12:21:25,011 [schedulerFactoryBean_Worker-2] jobLogger - Job Job id:8ab6982f4d46d5c5014d4788d8a702c3, type:SET_PERMISSION fetched. Execute...
INFO  2015-05-12 12:21:25,011 [schedulerFactoryBean_Worker-2] jobLogger - Return implementation of JobProcessor: setPermissionJobProcessor
DEBUG 2015-05-12 12:21:25,011 [schedulerFactoryBean_Worker-2] jobLogger - Processing instruction 1 / 1
DEBUG 2015-05-12 12:21:25,120 [schedulerFactoryBean_Worker-2] jobLogger - PERMISSIONS 8ab698f74c4a36f9014c4b16750a138e : Set permissions for decree (took 78 ms)
DEBUG 2015-05-12 12:21:25,120 [schedulerFactoryBean_Worker-2] jobLogger - Job Job id:8ab6982f4d46d5c5014d4788d8a702c3, type:SET_PERMISSION completed
DEBUG 2015-05-12 12:21:30,034 [schedulerFactoryBean_Worker-2] jobLogger - Job Job id:8ab6982f4d46d5c5014d4788d8a702c4, type:SET_PERMISSION fetched. Execute...
INFO  2015-05-12 12:21:30,034 [schedulerFactoryBean_Worker-2] jobLogger - Return implementation of JobProcessor: setPermissionJobProcessor
DEBUG 2015-05-12 12:21:30,034 [schedulerFactoryBean_Worker-2] jobLogger - Processing instruction 1 / 1
DEBUG 2015-05-12 12:21:30,143 [schedulerFactoryBean_Worker-2] jobLogger - PERMISSIONS 8ab698f74c213e16014c21ffd80c092f : Set permissions for decree (took 78 ms)
DEBUG 2015-05-12 12:21:30,143 [schedulerFactoryBean_Worker-2] jobLogger - Job Job id:8ab6982f4d46d5c5014d4788d8a702c4, type:SET_PERMISSION completed
DEBUG 2015-05-12 12:21:35,011 [schedulerFactoryBean_Worker-2] jobLogger - Job Job id:8ab6982f4d46d5c5014d4788d8a702c5, type:SET_PERMISSION fetched. Execute...
INFO  2015-05-12 12:21:35,011 [schedulerFactoryBean_Worker-2] jobLogger - Return implementation of JobProcessor: setPermissionJobProcessor
DEBUG 2015-05-12 12:21:35,011 [schedulerFactoryBean_Worker-2] jobLogger - Processing instruction 1 / 1
DEBUG 2015-05-12 12:21:35,120 [schedulerFactoryBean_Worker-2] jobLogger - PERMISSIONS 8ab698f74c31d989014c32661443090f : Set permissions for decree (took 78 ms)
DEBUG 2015-05-12 12:21:35,120 [schedulerFactoryBean_Worker-2] jobLogger - Job Job id:8ab6982f4d46d5c5014d4788d8a702c5, type:SET_PERMISSION completed
DEBUG 2015-05-12 12:21:40,018 [schedulerFactoryBean_Worker-2] jobLogger - Job Job id:8ab6982f4d46d5c5014d4788d8a702c6, type:SET_PERMISSION fetched. Execute...
INFO  2015-05-12 12:21:40,018 [schedulerFactoryBean_Worker-2] jobLogger - Return implementation of JobProcessor: setPermissionJobProcessor
DEBUG 2015-05-12 12:21:40,018 [schedulerFactoryBean_Worker-2] jobLogger - Processing instruction 1 / 1
DEBUG 2015-05-12 12:21:40,143 [schedulerFactoryBean_Worker-2] jobLogger - PERMISSIONS 8ab698f74c548787014c5590222c0a7a : Set permissions for decree (took 94 ms)
DEBUG 2015-05-12 12:21:40,143 [schedulerFactoryBean_Worker-2] jobLogger - Job Job id:8ab6982f4d46d5c5014d4788d8a702c6, type:SET_PERMISSION completed
DEBUG 2015-05-12 12:21:45,010 [schedulerFactoryBean_Worker-2] jobLogger - Job Job id:8ab6982f4d46d5c5014d4788d8a702c7, type:SET_PERMISSION fetched. Execute...
INFO  2015-05-12 12:21:45,010 [schedulerFactoryBean_Worker-2] jobLogger - Return implementation of JobProcessor: setPermissionJobProcessor
DEBUG 2015-05-12 12:21:45,010 [schedulerFactoryBean_Worker-2] jobLogger - Processing instruction 1 / 1
DEBUG 2015-05-12 12:21:45,119 [schedulerFactoryBean_Worker-2] jobLogger - PERMISSIONS 8ab698f74c0c65a9014c0d742e3a0843 : Set permissions for decree (took 78 ms)
DEBUG 2015-05-12 12:21:45,119 [schedulerFactoryBean_Worker-2] jobLogger - Job Job id:8ab6982f4d46d5c5014d4788d8a702c7, type:SET_PERMISSION completed
DEBUG 2015-05-12 12:21:50,018 [schedulerFactoryBean_Worker-2] jobLogger - Job Job id:8ab6982f4d46d5c5014d4788d8a702c8, type:SET_PERMISSION fetched. Execute...
INFO  2015-05-12 12:21:50,018 [schedulerFactoryBean_Worker-2] jobLogger - Return implementation of JobProcessor: setPermissionJobProcessor
DEBUG 2015-05-12 12:21:50,018 [schedulerFactoryBean_Worker-2] jobLogger - Processing instruction 1 / 1
DEBUG 2015-05-12 12:21:50,127 [schedulerFactoryBean_Worker-2] jobLogger - PERMISSIONS 8ab698f74c359b3f014c36c976930b5e : Set permissions for decree (took 78 ms)
DEBUG 2015-05-12 12:21:50,127 [schedulerFactoryBean_Worker-2] jobLogger - Job Job id:8ab6982f4d46d5c5014d4788d8a702c8, type:SET_PERMISSION completed
DEBUG 2015-05-12 12:21:55,010 [schedulerFactoryBean_Worker-2] jobLogger - Job Job id:8ab6982f4d46d5c5014d4788d8a702c9, type:SET_PERMISSION fetched. Execute...
INFO  2015-05-12 12:21:55,010 [schedulerFactoryBean_Worker-2] jobLogger - Return implementation of JobProcessor: setPermissionJobProcessor
DEBUG 2015-05-12 12:21:55,010 [schedulerFactoryBean_Worker-2] jobLogger - Processing instruction 1 / 1
DEBUG 2015-05-12 12:21:55,119 [schedulerFactoryBean_Worker-2] jobLogger - PERMISSIONS 8ab698f74c6917b8014c69bb84900535 : Set permissions for decree (took 78 ms)
DEBUG 2015-05-12 12:21:55,119 [schedulerFactoryBean_Worker-2] jobLogger - Job Job id:8ab6982f4d46d5c5014d4788d8a702c9, type:SET_PERMISSION completed
DEBUG 2015-05-12 12:22:00,017 [schedulerFactoryBean_Worker-2] jobLogger - Job Job id:8ab6982f4d46d5c5014d4788d8a702ca, type:SET_PERMISSION fetched. Execute...
INFO  2015-05-12 12:22:00,017 [schedulerFactoryBean_Worker-2] jobLogger - Return implementation of JobProcessor: setPermissionJobProcessor
DEBUG 2015-05-12 12:22:00,017 [schedulerFactoryBean_Worker-2] jobLogger - Processing instruction 1 / 1
DEBUG 2015-05-12 12:22:00,064 [schedulerFactoryBean_Worker-2] jobLogger - PERMISSIONS 8ab698f74c021f46014c031cef7b07cc : Set permissions for decree (took 0 ms)
DEBUG 2015-05-12 12:22:00,064 [schedulerFactoryBean_Worker-2] jobLogger - Job Job id:8ab6982f4d46d5c5014d4788d8a702ca, type:SET_PERMISSION completed
DEBUG 2015-05-12 12:22:05,025 [schedulerFactoryBean_Worker-2] jobLogger - Job Job id:8ab6982f4d46d5c5014d4788d8a702cb, type:SET_PERMISSION fetched. Execute...
INFO  2015-05-12 12:22:05,025 [schedulerFactoryBean_Worker-2] jobLogger - Return implementation of JobProcessor: setPermissionJobProcessor
DEBUG 2015-05-12 12:22:05,025 [schedulerFactoryBean_Worker-2] jobLogger - Processing instruction 1 / 1
DEBUG 2015-05-12 12:22:05,197 [schedulerFactoryBean_Worker-2] jobLogger - PERMISSIONS 8ab698f74c548787014c55a5018a0ca7 : Set permissions for decree (took 110 ms)
DEBUG 2015-05-12 12:22:05,197 [schedulerFactoryBean_Worker-2] jobLogger - Job Job id:8ab6982f4d46d5c5014d4788d8a702cb, type:SET_PERMISSION completed
DEBUG 2015-05-12 12:22:10,017 [schedulerFactoryBean_Worker-2] jobLogger - Job Job id:8ab6982f4d46d5c5014d4788d8a702cc, type:SET_PERMISSION fetched. Execute...
INFO  2015-05-12 12:22:10,017 [schedulerFactoryBean_Worker-2] jobLogger - Return implementation of JobProcessor: setPermissionJobProcessor
DEBUG 2015-05-12 12:22:10,017 [schedulerFactoryBean_Worker-2] jobLogger - Processing instruction 1 / 1
DEBUG 2015-05-12 12:22:10,126 [schedulerFactoryBean_Worker-2] jobLogger - PERMISSIONS 8ab698f74b00b9b8014b0140d7ba01a6 : Set permissions for decree (took 78 ms)
DEBUG 2015-05-12 12:22:10,126 [schedulerFactoryBean_Worker-2] jobLogger - Job Job id:8ab6982f4d46d5c5014d4788d8a702cc, type:SET_PERMISSION completed
DEBUG 2015-05-12 12:22:15,025 [schedulerFactoryBean_Worker-2] jobLogger - Job Job id:8ab6982f4d46d5c5014d4788d8a702cd, type:SET_PERMISSION fetched. Execute...
INFO  2015-05-12 12:22:15,025 [schedulerFactoryBean_Worker-2] jobLogger - Return implementation of JobProcessor: setPermissionJobProcessor
DEBUG 2015-05-12 12:22:15,025 [schedulerFactoryBean_Worker-2] jobLogger - Processing instruction 1 / 1
DEBUG 2015-05-12 12:22:15,134 [schedulerFactoryBean_Worker-2] jobLogger - PERMISSIONS 8ab698f74c31d989014c32955a5a0cf3 : Set permissions for decree (took 78 ms)
DEBUG 2015-05-12 12:22:15,134 [schedulerFactoryBean_Worker-2] jobLogger - Job Job id:8ab6982f4d46d5c5014d4788d8a702cd, type:SET_PERMISSION completed
DEBUG 2015-05-12 12:22:20,017 [schedulerFactoryBean_Worker-2] jobLogger - Job Job id:8ab6982f4d46d5c5014d4788d8a702ce, type:SET_PERMISSION fetched. Execute...
INFO  2015-05-12 12:22:20,017 [schedulerFactoryBean_Worker-2] jobLogger - Return implementation of JobProcessor: setPermissionJobProcessor
DEBUG 2015-05-12 12:22:20,032 [schedulerFactoryBean_Worker-2] jobLogger - Processing instruction 1 / 1
DEBUG 2015-05-12 12:22:20,141 [schedulerFactoryBean_Worker-2] jobLogger - PERMISSIONS 8ab698f74c4f636e014c512be72215ca : Set permissions for decree (took 78 ms)
DEBUG 2015-05-12 12:22:20,141 [schedulerFactoryBean_Worker-2] jobLogger - Job Job id:8ab6982f4d46d5c5014d4788d8a702ce, type:SET_PERMISSION completed
My nxlog.conf looks like this:

Code: Select all

## This is a sample configuration file. See the nxlog reference manual about the
## configuration options. It should be installed locally and is also available
## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html

## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.

#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog
define CERT %ROOT%\cert

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

# Include fileop while debugging, also enable in the output module below
<Extension fileop>
	 Module xm_fileop
</Extension>

<Extension json>
	Module xm_json
</Extension>

<Extension syslog>
	Module xm_syslog
</Extension>

<Input internal>
	Module im_internal
</Input>

# Watch your own files
<Input file1>
	Module im_file
	File '%ROOT%\data\nxlog.log'
	SavePos TRUE
</Input>

[b]<Input job>
	Module im_file
	File "M:\\Software\\Apache Software Foundation\\Tomcat 7.0\\logs\\job.log"
	SavePos TRUE
</Input>[/b]

# Windows Event Log
<Input eventlog>
	# Uncomment im_msvistalog for Windows Vista/2008 and later
	Module im_msvistalog
	# Uncomment im_mseventlog for Windows XP/2000/2003
	# Module im_mseventlog
</Input>

<Output out>
    Module      om_tcp
    Host        20.20.24.142
    Port        3515
	Exec $tmpmessage = $Message; delete($Message); rename_field("tmpmessage","message");
    Exec $raw_event = to_json();
    # Uncomment for debug output
    Exec file_write('%ROOT%\data\nxlog_output.log', $raw_event + "\n");
</Output>

<Route 1>
    Path internal, file1,[b] job[/b], eventlog => out
</Route>
So I tried enabling debugging, and the nxlog_output.log looks like this:

Code: Select all

{"EventReceivedTime":"2015-05-12 12:17:10","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:15","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:15","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:15","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:15","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:15","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:20","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:20","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:20","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:20","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:20","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:25","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:25","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:25","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:25","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:25","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:30","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:30","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:30","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:30","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:30","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:35","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:35","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:35","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:35","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:35","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:40","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:40","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:40","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:40","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:40","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:45","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:45","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:45","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:45","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:45","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:50","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:50","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:50","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:50","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:50","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:55","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:55","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:55","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:55","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:17:55","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:18:00","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:18:00","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:18:00","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:18:00","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:18:00","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:18:05","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:18:05","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:18:05","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:18:05","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:18:05","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:18:10","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:18:10","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:18:10","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:18:10","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:18:10","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:18:15","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:18:15","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:18:15","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:18:15","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:18:15","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:18:20","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:18:20","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:18:20","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:18:20","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:18:20","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:18:26","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:18:26","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:18:26","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:18:26","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
{"EventReceivedTime":"2015-05-12 12:18:26","SourceModuleName":"job","SourceModuleType":"im_file","message":null}
I can't find the logs in the NLS. THis is my input / filter config:

Code: Select all

# 
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Tue, 12 May 2015 12:26:51 +0200
#

#
# Global Configuration
#

input {
    tcp {
        type => 'import_json'
        tags => 'import_json'
        port => 2057
        codec => json
    }
    tcp {
        type => 'import_raw'
        tags => 'import_raw'
        port => 2056
    }
    tcp {
        type => 'eventlog'
        port => 3515
        codec => json {
            charset => 'CP1252'
        }
    }
    syslog {
        type => 'syslog'
        port => 5544
    }
    syslog {
        type => 'syslog-eternus'
        port => 1516
    }
    syslog {
        type => 'syslog-esx'
        port => 514
    }
    syslog {
        type => 'syslog-infoblox'
        port => 5545
    }
    syslog {
        type => 'syslog-linux'
        port => 5546
    }
    syslog {
        type => 'syslog-brocade'
        port => 5547
    }
    udp {
        type => 'syslog-f5'
        port => 5548
    }
    syslog {
        type => 'syslog-srx'
        port => 5549
    }
}

filter {
    if [program] == 'apache_access' {
        grok {
            match => [ 'message', '%{COMBINEDAPACHELOG}']
        }
        date {
            match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z' ]
        }
        mutate {
            replace => [ 'type', 'apache_access' ]
             convert => [ 'bytes', 'integer' ]
             convert => [ 'response', 'integer' ]
        }
    }
     
    if [program] == 'apache_error' {
        grok {
            match => [ 'message', '\[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\] \[%{WORD:class}\] \[%{WORD:originator} %{IP:clientip}\] %{GREEDYDATA:errmsg}']
        }
        mutate {
            replace => [ 'type', 'apache_error' ]
        }
    }
    if [type] == "syslog-brocade" {
        grok {
          match => [ "message", "<[\d]+>[a-z]+ [\d]+ [\d\:]+ %{IPV4:logsource}%{GREEDYDATA:program}: %{YEAR:year}\/%{MONTHNUM:month}\/%{MONTHDAY:day}-%{TIME:time}%{GREEDYDATA:data1}WWN %{IPV6:wwn}%{GREEDYDATA:data2}%{LOGLEVEL:loglevel}\, %{HOSTNAME:hostname}\, %{GREEDYDATA:info}" ]
        remove_tag => "_grokparsefailure"
        add_tag => "grokked_syslog_brocade"
        }     
      }
    
    if [type] == "syslog-f5" {
        grok {     
          break_on_match => false
          match => [ "message", "\A%{SYSLOG5424PRI}%{SYSLOGTIMESTAMP} slot1\/%{HOSTNAME:logsource} %{LOGLEVEL:severity_label} %{SYSLOGPROG}: %{GREEDYDATA:info}" ]
          add_tag => "grokked_syslog_f5"      
        }   
    }
    
    if [program] == "dcc" {
            grok {          
              patterns_dir => "/usr/local/nagioslogserver/logstash/patterns"
              match => [ "info", "%{F5SEQ:f5_sequence}: %{GREEDYDATA:info}violations: %{GREEDYDATA:f5_violations}. HTTP protocol compliance sub violations: %{GREEDYDATA:f5_http_violations}. Evasion techniques sub violations: %{GREEDYDATA:f5_evasion_violations}. Web services security sub violations: %{GREEDYDATA:f5_web_svc_violations}. Virus name: %{GREEDYDATA:f5_virusname}. Support id: %{GREEDYDATA:f5_supportid}, source ip: %{IPNA:f5_sourceip}, xff ip: %{IPNA:f5_xffip}, source port: %{NUMBER:f5_sourceport}, destination ip: %{IPNA:f5_destinationip}, destination port: %{NUMBER:f5_destinationport}, route_domain: %{NUMBER:f5_routedomain}, HTTP classifier: %{GREEDYDATA:f5_http_classifier}, scheme %{SCHEME:f5_scheme}, geographic location:%{GREEDYDATA:f5_geolocation}, request: %{GREEDYDATA:f5_request}, username:%{GREEDYDATA:f5_username}, session_id: %{GREEDYDATA:f5_sessionid}" ]          
              match => [ "info", "%{GREEDYDATA:info}" ]
              remove_tag => "grokked_syslog_f5"
              add_tag => "grokked_syslog_f5_dcc"
              overwrite => [ "info" ]
              }
        }
    
    if [type] == "eventlog" {
        mutate {
            lowercase => [ "Hostname", "EventType", "Severity" ]
            remove => [ "SourceModuleType", "EventTimeWritten", "EventTime", "EventReceivedTime", "EventType" ]
        }
    }
    
}

#
# Local Configuration
#
Thanks for any guidance in making this work.

Grtz

Willem
Nagios XI 5.8.1
https://outsideit.net
Top
jolson
Attack Rabbit
Posts: 2560
Joined: Thu Feb 12, 2015 12:40 pm

Re: Unable to send simple logfile to NLS

Post by jolson »

Willem,

After testing this on my end, this is the end-result I came up with:
2015-05-12 10_00_27-Dashboard • Nagios Log Server.png
If the above looks good to you, the configurations below (based on yours) should be good to go.

nxlog:

Code: Select all

    ## This is a sample configuration file. See the nxlog reference manual about the
    ## configuration options. It should be installed locally and is also available
    ## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html

    ## Please set the ROOT to the folder your nxlog was installed into,
    ## otherwise it will not start.

    #define ROOT C:\Program Files\nxlog
    define ROOT C:\Program Files (x86)\nxlog
    define CERT %ROOT%\cert

    Moduledir %ROOT%\modules
    CacheDir %ROOT%\data
    Pidfile %ROOT%\data\nxlog.pid
    SpoolDir %ROOT%\data
    LogFile %ROOT%\data\nxlog.log

    # Include fileop while debugging, also enable in the output module below
    <Extension fileop>
        Module xm_fileop
    </Extension>

    <Extension json>
       Module xm_json
    </Extension>

    <Extension syslog>
       Module xm_syslog
    </Extension>

    <Input internal>
       Module im_internal
    </Input>

    # Watch your own files
    <Input file1>
       Module im_file
       File '%ROOT%\data\jobs.log'
       SavePos TRUE
    </Input>

    <Input job>
    Module   im_file
    File     'D:\Software\Apache Software Foundation\Tomcat 7.0\logs\jobs.log'
    SavePos  TRUE
    Exec     $Message = $raw_event;
    </Input>

    # Windows Event Log
    <Input eventlog>
       # Uncomment im_msvistalog for Windows Vista/2008 and later
       Module im_msvistalog
       # Uncomment im_mseventlog for Windows XP/2000/2003
       # Module im_mseventlog
    </Input>

    <Output out>
        Module      om_tcp
        Host        x.x.x.x  
        Port        3515
       Exec $tmpmessage = $Message; delete($Message); rename_field("tmpmessage","message");
        Exec $raw_event = to_json();
        # Uncomment for debug output
        Exec file_write('%ROOT%\data\nxlog_output.log', $raw_event + "\n");
    </Output>

    <Route 1>
        Path internal, job, eventlog => out
    </Route>
Note: In the above configuration, I removed 'file1' from being routed, as that file does not exist on my system. You're free to add it back in if you want:

Code: Select all

Path internal, file1, job, eventlog => out
Regarding my input, I used the example config provided by our Wizard and modified it to suit the path of your file:

Code: Select all

File     'D:\Software\Apache Software Foundation\Tomcat 7.0\logs\jobs.log'
NLS input:

Code: Select all

cat /usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf

input {
tcp {
   type => 'eventlog'
   port => 3515
     codec => json {
         charset => 'CP1252'
     }
}
I did not use a custom filter for my config.

After making the above configuration changes, I restarted the nxlog service via services.msc, and I opened the 'jobs.log' file and added a line to the bottom - this is necessary because SavePos is set to TRUE. After adding this new line to the bottom of the log file, I received a result in the NLS Dashboard - this result is pictures above.

Troubleshooting: After getting the above setup, I did a

Code: Select all

tcpdump -n dst port 3515 -X
to ensure that when I add a line to my jobs.log file and save it, nxlog sends the resulting information to NLS properly. I suggest doing the same before adding a line to jobs.log.


Let me know if you have any questions Willem - thanks!
You do not have the required permissions to view the files attached to this post.
Twits Blog
Show me a man who lives alone and has a perpetually clean kitchen, and 8 times out of 9 I'll show you a man with detestable spiritual qualities.
Top
User avatar
WillemDH
Posts: 2320
Joined: Wed Mar 20, 2013 5:49 am
Location: Ghent
Contact:
Contact WillemDH

Re: Unable to send simple logfile to NLS

Post by WillemDH »

Jesse,

I followed your recommendations, thanks for the extensive examples.

So it seems like in my nxlog.conf, the only difference was:

Code: Select all

<Input job>
	Module im_file
	File 'M:\Software\Apache Software Foundation\Tomcat 7.0\logs\job.log'
	SavePos TRUE
	[b]Exec $Message = $raw_event;[/b]
</Input>
After adding 'Exec $Message = $raw_event;' , restarting nxlog service, i did a tcpdump as you proposed only to discover that it was impossible to notice anything, as I have about 18 Windows server sending logs to port 3515. So I edited the nxlog.conf to send to my other NLS node and also left out the eventlog part in route 1:

Code: Select all

<Route 1>
    Path internal, file1, job => out
</Route>
Then I did a tcpdump on the other node where I did see an entry of job.log coming in:

Code: Select all

08:47:06.973694 IP xx:xx:xx:xx.54813 > xx:xx:xx:xx.must-backplane: Flags [P.], seq 1454:1711, ack 1, win 513, length 257
        0x0000:  4500 0129 6ad6 4000 8006 4951 0a36 18af  E..)[email protected]..
        0x0010:  0a36 188d d61d 0dbb 720a 00e2 3d5e 2761  .6......r...=^'a
        0x0020:  5018 0201 c5b4 0000 7b22 4576 656e 7452  P.......{"EventR
        0x0030:  6563 6569 7665 6454 696d 6522 3a22 3230  eceivedTime":"20
        0x0040:  3135 2d30 352d 3133 2030 383a 3437 3a30  15-05-13.08:47:0
        0x0050:  3622 2c22 536f 7572 6365 4d6f 6475 6c65  6","SourceModule
        0x0060:  4e61 6d65 223a 226a 6f62 222c 2253 6f75  Name":"job","Sou
        0x0070:  7263 654d 6f64 756c 6554 7970 6522 3a22  rceModuleType":"
        0x0080:  696d 5f66 696c 6522 2c22 6d65 7373 6167  im_file","messag
        0x0090:  6522 3a22 4445 4255 4720 3230 3135 2d30  e":"DEBUG.2015-0
        0x00a0:  352d 3133 2030 383a 3436 3a33 382c 3237  5-13.08:46:38,27
        0x00b0:  3220 5b73 6368 6564 756c 6572 4661 6374  2.[schedulerFact
        0x00c0:  6f72 7942 6561 6e5f 576f 726b 6572 2d32  oryBean_Worker-2
        0x00d0:  5d20 6a6f 624c 6f67 6765 7220 2d20 4a6f  ].jobLogger.-.Jo
        0x00e0:  6220 4a6f 6220 6964 3a38 6162 3639 3832  b.Job.id:8ab6982
        0x00f0:  6634 6434 3862 3764 6230 3134 6434 6138  f4d48b7db014d4a8
        0x0100:  3866 3934 3831 6661 352c 2074 7970 653a  8f9481fa5,.type:
        0x0110:  4558 504f 5254 5f4f 4646 4c49 4e45 2066  EXPORT_OFFLINE.f
        0x0120:  6169 6c65 6422 7d0d 0a                   ailed"}..
08:53:57.581250 IP xx:xx:xx:xx.54813 > xx:xx:xx:xx.must-backplane: Flags [P.], seq 1711:1952, ack 1, win 513, length 241
        0x0000:  4500 0119 073d 4000 8006 acfa 0a36 18af  [email protected]..
        0x0010:  0a36 188d d61d 0dbb 720a 01e3 3d5e 2761  .6......r...=^'a
        0x0020:  5018 0201 4dee 0000 7b22 4576 656e 7452  P...M...{"EventR
        0x0030:  6563 6569 7665 6454 696d 6522 3a22 3230  eceivedTime":"20
        0x0040:  3135 2d30 352d 3133 2030 383a 3533 3a35  15-05-13.08:53:5
        0x0050:  3722 2c22 536f 7572 6365 4d6f 6475 6c65  7","SourceModule
        0x0060:  4e61 6d65 223a 226a 6f62 222c 2253 6f75  Name":"job","Sou
        0x0070:  7263 654d 6f64 756c 6554 7970 6522 3a22  rceModuleType":"
        0x0080:  696d 5f66 696c 6522 2c22 6d65 7373 6167  im_file","messag
        0x0090:  6522 3a22 4445 4255 4720 3230 3135 2d30  e":"DEBUG.2015-0
        0x00a0:  352d 3133 2030 323a 3037 3a33 362c 3538  5-13.02:07:36,58
        0x00b0:  3720 5b73 6368 6564 756c 6572 4661 6374  7.[schedulerFact
        0x00c0:  6f72 7942 6561 6e5f 576f 726b 6572 2d32  oryBean_Worker-2
        0x00d0:  5d20 6a6f 624c 6f67 6765 7220 2d20 5175  ].jobLogger.-.Qu
        0x00e0:  6572 7920 7265 7375 6c74 2066 6f75 6e64  ery.result.found
        0x00f0:  2069 6e20 6361 6368 652c 2072 6574 7572  .in.cache,.retur
        0x0100:  6e69 6e67 2063 6163 6865 6420 7465 6d70  ning.cached.temp
        0x0110:  6c61 7465 7322 7d0d 0a                   lates"}..
So good new I guess. But then I tried looking for the log in NLS gui and I was not able to find any word in the message. Looking for the ip address of this Tomcat server also did not yield any results. Where could this log be stuck? Is it possible the log line is not getting parsed by the default NLS filters?

EDIT: I did not manage to get it working by the method above, but I changed the nxlog.conf, so it uses a separate port, 5550

Code: Select all

<Output outfile>
    Module      om_tcp
    Host        xx.xx.xx.xx
    Port        5550
	Exec $tmpmessage = $Message; delete($Message); rename_field("tmpmessage","message");
    Exec $raw_event = to_json();
    # Uncomment for debug output
    Exec file_write('%ROOT%\data\nxlog_output.log', $raw_event + "\n");
</Output>

<Route 1>
    Path internal, file1, eventlog => out
</Route>

<Route 1>
    Path job => outfile
</Route>
Also made specific filter:

Code: Select all

syslog {
    type => 'file-tomcat-job'
    port => 5550
    codec => json {
        charset => 'CP1252'
    }
}
And now I can see the job loglilnes in NLS. Some logs appear to be multiline. I did not had to handle multiline logs yet.
They look like this sometimes:

Code: Select all

DEBUG 2015-05-13 02:07:05,840 [schedulerFactoryBean_Worker-2] jobLogger - Export AgendaPage with id 8ab698f74d089673014d0a3cb8580f59, title Nieuw reglement Samenlevingsprijs - Goedkeuring
ERROR 2015-05-13 02:07:05,918 [schedulerFactoryBean_Worker-2] jobLogger - error (io) in exporting document with title: 20150408_DO_Reglement Samenlevingsprijs m?t track changes.docx and destinationfolder C:\offline\Commissie Welzijn, Werk en Milieu\2015\5\19-05-2015\Dagorde\Openbare vergadering\Agendapunten\Departement Samenleven en Welz_\Integratiedienst\Nieuw reglement Samenlevingspr_\Bijlagen
java.io.FileNotFoundException: C:\offline\Commissie Welzijn, Werk en Milieu\2015\5\19-05-2015\Dagorde\Openbare vergadering\Agendapunten\Departement Samenleven en Welz_\Integratiedienst\Nieuw reglement Samenlevingspr_\Bijlagen\20150408_DO_Reglement Samenlevingsprijs m?t track changes.docx (The filename, directory name, or volume label syntax is incorrect)
	at java.io.FileOutputStream.open(Native Method)
	at java.io.FileOutputStream.<init>(Unknown Source)
	at java.io.FileOutputStream.<init>(Unknown Source)
	at nl.greenvalley.digipolis.portal.meeting.service.MeetingExportServiceImpl.export(MeetingExportServiceImpl.java:204)
	at nl.greenvalley.digipolis.portal.meeting.service.MeetingExportServiceImpl.exportAttachments(MeetingExportServiceImpl.java:196)
	at nl.greenvalley.digipolis.portal.meeting.service.MeetingExportServiceImpl.exportAgendaPage(MeetingExportServiceImpl.java:162)
	at nl.greenvalley.digipolis.portal.meeting.service.MeetingExportServiceImpl.buildStructure(MeetingExportServiceImpl.java:155)
	at nl.greenvalley.digipolis.portal.meeting.service.MeetingExportServiceImpl.buildStructure(MeetingExportServiceImpl.java:152)
	at nl.greenvalley.digipolis.portal.meeting.service.MeetingExportServiceImpl.buildStructure(MeetingExportServiceImpl.java:152)
	at nl.greenvalley.digipolis.portal.meeting.service.MeetingExportServiceImpl.buildStructure(MeetingExportServiceImpl.java:152)
	at nl.greenvalley.digipolis.portal.meeting.service.MeetingExportServiceImpl.buildStructure(MeetingExportServiceImpl.java:152)
EBUG 2015-05-13 02:07:27,914 [schedulerFactoryBean_Worker-2] jobLogger - Job Job id:8ab6982f4d48b7db014d4a88f9481fa5, type:EXPORT_OFFLINE fetched. Execute...
INFO  2015-05-13 02:07:27,914 [schedulerFactoryBean_Worker-2] jobLogger - Return implementation of JobProcessor: exportMeetingOffLineJobProcessor
In NLS each line is considered as a separate log, giving very strange results etc. How could I catch these multiline logs as a single logline?

EDIT: Ok, I read through the NxLog Manual. Seems I will need to use something like this:

<Extension dicom-multi>
Module xm_multiline
HeaderLine /ERROR|DEBUG|WARN|INFO/
</Extension>

As all valid loglines start with ERROR, DEBUG, INFO or WARN. I'll do some tests and see if I get this working.

EDIT 2: It seems I got the above nxlog config working. Plese leave this thread open untill I got a week or two of data. Tx

Grtz

Willem
Nagios XI 5.8.1
https://outsideit.net
Top
jolson
Attack Rabbit
Posts: 2560
Joined: Thu Feb 12, 2015 12:40 pm

Re: Unable to send simple logfile to NLS

Post by jolson »

Thanks for reporting your results back Willem, they'll be very helpful for future reference. I'll leave the thread open - let us know about any problems you might encounter. Thanks!
Twits Blog
Show me a man who lives alone and has a perpetually clean kitchen, and 8 times out of 9 I'll show you a man with detestable spiritual qualities.
Top
User avatar
WillemDH
Posts: 2320
Joined: Wed Mar 20, 2013 5:49 am
Location: Ghent
Contact:
Contact WillemDH

Re: Unable to send simple logfile to NLS

Post by WillemDH »

Jesse,

I get some very strange results with the multiline inputtype. I could swear end of last week it seemed to work. When I test now, sometimes the lines not starting with what I defined in Headerline argument

Code: Select all

<Extension multiline>
	Module xm_multiline
	HeaderLine /ERROR|DEBUG|WARN|INFO/
</Extension>

Some more example logs:

Code: Select all

DEBUG 2015-04-24 01:50:16,204 [schedulerFactoryBean_Worker-2] jobLogger - Query result found in cache, returning cached templates
WARN  2015-04-24 01:50:16,204 [schedulerFactoryBean_Worker-2] jobLogger - Multiple templates found, using first template: agenda.dmt
DEBUG 2015-05-13 02:04:58,232 [schedulerFactoryBean_Worker-2] jobLogger - Export AgendaPage with id 8ab698f74d2ca23d014d2e88b1571209, title 2015_CBS_05584 - Stopzetting overeenkomst houdende het toekennen van een subsidie aan Besiktas Gent Jeugd voor de tewerkstelling van een sportcoördinator 2015-2016 - Goedkeuring
ERROR 2015-05-13 02:04:58,356 [schedulerFactoryBean_Worker-2] jobLogger - error (io) in exporting document with title: Reglement op de subsidi?ring van de tewerkstelling van sportco?rdinatoren.pdf and destinationfolder C:\offline\College van Burgemeester en Schepenen\2015\5\13-05-2015\Dagorde\B-punten\Resul Tapmaz\Departement Cultuur Sport en V_\Sportdienst\2015_CBS_05584 - Stopzetting o_\Bijlagen
java.io.FileNotFoundException: C:\offline\College van Burgemeester en Schepenen\2015\5\13-05-2015\Dagorde\B-punten\ilui\Departement Cultuur Sport en V_\Sportdienst\2015_CBS_05584 - Stopzetting o_\Bijlagen\Reglement op de subsidi?ring van de tewerkstelling van sportco?rdinatoren.pdf (The filename, directory name, or volume label syntax is incorrect)
	at java.io.FileOutputStream.open(Native Method)
	at java.io.FileOutputStream.<init>(Unknown Source)
	at java.io.FileOutputStream.<init>(Unknown Source)
	at nl.greenvalley.digipolis.portal.meeting.service.MeetingExportServiceImpl.export(MeetingExportServiceImpl.java:204)
	at nl.greenvalley.digipolis.portal.meeting.service.MeetingExportServiceImpl.exportAttachments(MeetingExportServiceImpl.java:196)
	at nl.greenvalley.digipolis.portal.meeting.service.MeetingExportServiceImpl.exportAgendaPage(MeetingExportServiceImpl.java:162)
	at nl.greenvalley.digipolis.portal.meeting.service.MeetingExportServiceImpl.buildStructure(MeetingExportServiceImpl.java:155)
	at nl.greenvalley.digipolis.portal.meeting.service.MeetingExportServiceImpl.buildStructure(MeetingExportServiceImpl.java:152)
	at nl.greenvalley.digipolis.portal.meeting.service.MeetingExportServiceImpl.buildStructure(MeetingExportServiceImpl.java:152)
	at nl.greenvalley.digipolis.portal.meeting.service.MeetingExportServiceImpl.buildStructure(MeetingExportServiceImpl.java:152)
	at nl.greenvalley.digipolis.portal.meeting.service.MeetingExportServiceImpl.buildStructure(MeetingExportServiceImpl.java:152)
	at nl.greenvalley.digipolis.portal.meeting.service.MeetingExportServiceImpl.buildStructure(MeetingExportServiceImpl.java:152)
	at nl.greenvalley.digipolis.portal.meeting.service.MeetingExportServiceImpl.export(MeetingExportServiceImpl.java:135)
	at nl.greenvalley.digipolis.portal.meeting.service.MeetingExportServiceImpl$$FastClassBySpringCGLIB$$ffe2e27a.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:700)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:96)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:260)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:633)
	at nl.greenvalley.digipolis.portal.meeting.service.MeetingExportServiceImpl$$EnhancerBySpringCGLIB$$5e1a1098.export(<generated>)
	at nl.greenvalley.digipolis.portal.PortalFacadeImpl.exportMeetingOffline(PortalFacadeImpl.java:336)
	at nl.greenvalley.digipolis.portal.PortalFacadeImpl$$FastClassBySpringCGLIB$$e00a424e.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:700)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
	at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:51)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:96)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:260)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:633)
	at nl.greenvalley.digipolis.portal.PortalFacadeImpl$$EnhancerBySpringCGLIB$$b70a9bc1.exportMeetingOffline(<generated>)
	at nl.greenvalley.digipolis.portal.job.processor.ExportMeetingOffLineJobProcessor.doExecuteInstruction(ExportMeetingOffLineJobProcessor.java:28)
	at nl.greenvalley.digipolis.portal.job.processor.ExportMeetingOffLineJobProcessor.doExecuteInstruction(ExportMeetingOffLineJobProcessor.java:14)
	at nl.greenvalley.digipolis.portal.job.processor.BaseJobProcessor$1.doInTransaction(BaseJobProcessor.java:67)
	at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:131)
	at nl.greenvalley.digipolis.portal.job.processor.BaseJobProcessor.execute(BaseJobProcessor.java:63)
	at nl.greenvalley.digipolis.portal.job.service.JobServiceImpl.exportMeetings(JobServiceImpl.java:344)
	at nl.greenvalley.digipolis.portal.job.service.JobServiceImpl$$FastClassBySpringCGLIB$$650d710a.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:700)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:96)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:260)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:633)
	at nl.greenvalley.digipolis.portal.job.service.JobServiceImpl$$EnhancerBySpringCGLIB$$a8b3dd28.exportMeetings(<generated>)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:269)
	at org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean$MethodInvokingJob.executeInternal(MethodInvokingJobDetailFactoryBean.java:311)
	at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:113)
	at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:529)
DEBUG 2015-05-13 02:04:58,419 [schedulerFactoryBean_Worker-2] jobLogger - Export AgendaPage with id 8ab698f74c9c69f0014c9f713b241dbc, title 2015_CBS_05585 - Processen-verbaal door de toezichthoudende ambtenaren van de Dienst Milieutoezicht - Kennisneming
DEBUG 2015-04-24 01:50:16,204 [schedulerFactoryBean_Worker-2] jobLogger - Query result found in cache, returning cached templates
WARN  2015-04-24 01:50:16,204 [schedulerFactoryBean_Worker-2] jobLogger - Multiple templates found, using first template: agenda.dmt
DEBUG 2015-04-24 01:50:16,204 [schedulerFactoryBean_Worker-2] jobLogger - Query result found in cache, returning cached templates
WARN  2015-04-24 01:50:16,204 [schedulerFactoryBean_Worker-2] jobLogger - Multiple templates found, using first template: agenda.dmt
The lines beginning with java.io and ' at java.io' etc that are causing strange results. Sometimes this data is not showing at all, sometimes it creates a separate lines for each java exception line.

I would have thought using this:

Code: Select all

[code]<Extension multiline>
	Module xm_multiline
	HeaderLine /ERROR|DEBUG|WARN|INFO/
</Extension>
[/code]

Should send one log to NLS and only if the log starts with ERROR, DEBUG, WARN or INFO.

using this filter atm:

Code: Select all

if [type] == "file-tomcat-job" {   
    mutate {
        remove => [ "facility", "facility_label", "severity", "priority" ]
        add_field => { 
          "hostname" => "%{host}"
        }
        rename => [ "EventReceivedTime", "eventreceivedtime" ]
        rename => [ "SourceModuleName", "sourcemodulename" ]
        rename => [ "SourceModuleType", "sourcemoduletype" ]
    }
    dns {
        reverse => [ "hostname" ]
        action => [ "replace" ]
    }
    grok {
      match => [ "message", "%{GREEDYDATA:severity_label} %{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND} \[%{GREEDYDATA:worker}\] jobLogger - %{GREEDYDATA:message}" ]
        overwrite => [ "severity_label", "message" ]
        add_tag => "grokked_tomcat_job"      
    }
    mutate {
        lowercase => [ "severity_label" ]
    }   
    mutate {
        gsub => [ 
            "severity_label", "warn", "warning"
        ]
    } 
}
And this input:

Code: Select all

tcp {
    type => 'file-tomcat-job'
    port => 5550
    codec => json {
        charset => 'CP1252'
    }
}
This will need some more testing. To be continued...

EDIT:

I suspected the problem was in the

Code: Select all

overwrite => [ "severity_label", "message" ]
So I removed the message from the overwrite and replaced message by info:

Code: Select all

 match => [ "message", "%{GREEDYDATA:severity_label} %{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND} \[%{GREEDYDATA:worker}\] jobLogger - %{GREEDYDATA:info}" ]
Suddenly things got parsed in NLS like I expected it. Any idea what might cause this behaviour and how to catch all lines after the first? %{GREEDYDATA:info} at the end of my grok match pattern should put all lines in the info field right? I attached a screenshot that show the info field only getting partially filled.

Grtz

Willem
You do not have the required permissions to view the files attached to this post.
Nagios XI 5.8.1
https://outsideit.net
Top
jolson
Attack Rabbit
Posts: 2560
Joined: Thu Feb 12, 2015 12:40 pm

Re: Unable to send simple logfile to NLS

Post by jolson »

Suddenly things got parsed in NLS like I expected it. Any idea what might cause this behaviour and how to catch all lines after the first? %{GREEDYDATA:info} at the end of my grok match pattern should put all lines in the info field right? I attached a screenshot that show the info field only getting partially filled.
We'll likely need to enable the multi-line matching flag in regex, like so:

Code: Select all

match => [ "message", "(?m)%{GREEDYDATA:severity_label} %{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND} \[%{GREEDYDATA:worker}\] jobLogger - %{GREEDYDATA:info}" ]
Would you please insert the (?m) flag into your pattern? It should allow your last GREEDYDATA to match multiple lines.
Twits Blog
Show me a man who lives alone and has a perpetually clean kitchen, and 8 times out of 9 I'll show you a man with detestable spiritual qualities.
Top
User avatar
WillemDH
Posts: 2320
Joined: Wed Mar 20, 2013 5:49 am
Location: Ghent
Contact:
Contact WillemDH

Re: Unable to send simple logfile to NLS

Post by WillemDH »

Fantastic Jesse, seems "(?m)" did the trick. I almost finished my Tomcat portal and job logs. Please leave this thread open for some time.
Nagios XI 5.8.1
https://outsideit.net
Top
jolson
Attack Rabbit
Posts: 2560
Joined: Thu Feb 12, 2015 12:40 pm

Re: Unable to send simple logfile to NLS

Post by jolson »

Will do - I look forward to seeing what you come up with. :)
Twits Blog
Show me a man who lives alone and has a perpetually clean kitchen, and 8 times out of 9 I'll show you a man with detestable spiritual qualities.
Top
Locked

Return to “Nagios Log Server”