FTP Logs

[jgsupport]

Hi,

We are unable to see messages from our IIS FTP log files. We have nxlog-ce-2.10.2150 installed on the server.
We are seeing errors below in nxlog.log file.

Server name and ports that are below have been changed from original logs and config file to post here.


2020-08-06 12:41:27 ERROR if-else failed at line 79, character 339 in C:\Program Files\nxlog\conf\nxlog.conf. statement execution has been aborted; procedure 'parse_csv' failed at line 79, character 170 in C:\Program Files\nxlog\conf\nxlog.conf. statement execution has been aborted; couldn't parse integer: IISSERVERNAME

Our NXLOG.CONF file is below.
Line 79 has only { character show below

<Input FTP_log_SVC2>
Module im_file
File 'C:\inetpub\logs\logfiles\FTPSVC2\u_ex*.log'
SavePos TRUE
#Exec $Message = $raw_event;
Exec if $raw_event =~ /^#/ drop(); \
else \
{ \
w3c->parse_csv(); \
$EventTime = parsedate($date + " " + $time); \
$SourceName = "IIS"; \
$Message = $raw_event; \
} "------> LINE 79"

see below for full config file


## See the nxlog reference manual at
## http://nxlog.org/nxlog-docs/en/nxlog-re ... anual.html

## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.
define ROOT C:\Program Files\nxlog
#define ROOT C:\Program Files (x86)\nxlog
define CERT %ROOT%\cert

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

# Include fileop while debugging, also enable in the output module below
#<Extension fileop>
# Module xm_fileop
#</Extension>

<Extension json>
Module xm_json
</Extension>

<Extension syslog>
Module xm_syslog
</Extension>

<Extension w3c>
Module xm_csv
Fields $date, $time, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $csUser-Agent, $cs-Referer, $sc-status, $sc-substatus, $sc-win32-status, $time-taken
FieldTypes string, string, string, string, string, string, integer, string, string, string, string, integer, integer, integer, integer
Delimiter ' '
QuoteChar '"'
EscapeControl FALSE
UndefValue -
</Extension>

<Input internal>
Module im_internal
</Input>

# Watch your own files
<Input file1>
Module im_file
File '%ROOT%\data\nxlog.log'
SavePos TRUE
Exec $Message = $raw_event;
</Input>

<Input FTP_log_SVC5>
Module im_file
File 'C:\inetpub\logs\logfiles\FTPSVC5\u_ex*.log'
SavePos TRUE
#Exec $Message = $raw_event;
Exec if $raw_event =~ /^#/ drop(); \
else \
{ \
w3c->parse_csv(); \
$EventTime = parsedate($date + " " + $time); \
$SourceName = "IIS"; \
$Message = $raw_event; \
}

</Input>

<Input FTP_log_SVC2>
Module im_file
File 'C:\inetpub\logs\logfiles\FTPSVC2\u_ex*.log'
SavePos TRUE
#Exec $Message = $raw_event;
Exec if $raw_event =~ /^#/ drop(); \
else \
{ \
w3c->parse_csv(); \
$EventTime = parsedate($date + " " + $time); \
$SourceName = "IIS"; \
$Message = $raw_event; \
}

</Input>

# Windows Event Log
<Input eventlog>
# Uncomment im_msvistalog for Windows Vista/2008 and later
Module im_msvistalog

# Uncomment im_mseventlog for Windows XP/2000/2003
# Module im_mseventlog
</Input>

<Output out>
Module om_tcp
Host LOGSERVER
Port 1234

Exec $tmpmessage = $Message; delete($Message); rename_field("tmpmessage","message");
Exec $raw_event = to_json();

# Uncomment for debug output
# Exec file_write('%ROOT%\data\nxlog_output.log', $raw_event + "\n");
</Output>

<Output IIS-out>
Module om_tcp
Host LOGSERVER
Port 5678
Exec $tmpmessage = $Message; delete($Message); rename_field("tmpmessage","message");
Exec $raw_event = to_json();
# Uncomment for debug output
# Exec file_write('%ROOT%\data\nxlog_output.log', $raw_event + "\n");
</Output>

<Route 1>
Path internal, file1, eventlog => out
</Route>

<Route IIS>
Path FTP_log_SVC2, FTP_log_SVC5 => IIS-out
</Route>

[scottwilkerson]

I don't believe the syntax is correct, for this block

Code: Select all

<Input FTP_log_SVC5>
Module im_file
File 'C:\inetpub\logs\logfiles\FTPSVC5\u_ex*.log'
SavePos TRUE
#Exec $Message = $raw_event;
Exec if $raw_event =~ /^#/ drop(); \
else \
{ \
w3c->parse_csv(); \
$EventTime = parsedate($date + " " + $time); \
$SourceName = "IIS"; \
$Message = $raw_event; \
}

</Input>

<Input FTP_log_SVC2>
Module im_file
File 'C:\inetpub\logs\logfiles\FTPSVC2\u_ex*.log'
SavePos TRUE
#Exec $Message = $raw_event;
Exec if $raw_event =~ /^#/ drop(); \
else \
{ \
w3c->parse_csv(); \
$EventTime = parsedate($date + " " + $time); \
$SourceName = "IIS"; \
$Message = $raw_event; \
}

</Input>

I would suggest changing to this

Code: Select all

<Input FTP_log_SVC5>
Module im_file
File 'C:\inetpub\logs\logfiles\FTPSVC5\u_ex*.log'
SavePos TRUE
<Exec> 
    if {
        $raw_event =~ /^#/ drop();
        }
    else
    {
        w3c->parse_csv();
        $EventTime = parsedate($date + " " + $time);
        $SourceName = "IIS";
        $Message = $raw_event;
    }
</Exec>
</Input>

<Input FTP_log_SVC2>
Module im_file
File 'C:\inetpub\logs\logfiles\FTPSVC2\u_ex*.log'
SavePos TRUE
<Exec>
<Exec> 
    if {
        $raw_event =~ /^#/ drop();
        }
    else
    {
        w3c->parse_csv();
        $EventTime = parsedate($date + " " + $time);
        $SourceName = "IIS";
        $Message = $raw_event;
    }
</Exec>
</Input>

https://nxlog.co/documentation/nxlog-us ... atement_if

[jgsupport]

Thank you. What you have suggested works, I can now see the logs in the log server.
However I still have the issue where I cannot see the message from the log in the log server.
The message field is coming up as null so I have no information from the logs.
Is there any way I am able to see the messages in the FTP log on the log server?

Here is the output on log server.

{
"_index": "logstash-2020.08.06",
"_type": "IIS_requests",
"_id": "AXPFzOix4hzN-5u90BUi",
"_score": null,
"_source": {
"EventReceivedTime": "2020-08-07 08:03:27",
"SourceModuleName": "FTP_log_SVC2",
"SourceModuleType": "im_file",
"message": null,
"@version": "1",
"@timestamp": "2020-08-06T22:03:26.925Z",
"host": "1.2.3.4",
"port": 12345,
"type": "IIS_requests",
"tags": [
"IIS_requests"
]
},
"sort": [
null,
1596751406925
]
}

Is it because of this line in config? Not matching criteria? If so what would be the best to change it to?
Sorry I am not the best with coding etc..

if {
$raw_event =~ /^#/ drop();
}

I tried commenting it out completely like below but still not seeing any messages
<Input FTP_log_SVC2>
Module im_file
File 'C:\inetpub\logs\logfiles\FTPSVC2\u_ex*.log'
SavePos TRUE
<Exec>
#if {
# $raw_event =~ /^#/ drop();
# }
#else
#{
w3c->parse_csv();
$EventTime = parsedate($date + " " + $time);
$SourceName = "IIS";
$Message = $raw_event;
#}
</Exec>
</Input>

The logs I want to capture and alert on in the FTPSVC2 log file are below

2020-08-06 22:29:21 1.2.3.4 46618 - FTPSVC2 FTPSERVERNAME 5.6.7.8 21 USER ftptest 331 0 0 69 20 efccb174-1c47-46fc-8554-5b3fb0a7add9 - -
2020-08-06 22:29:21 1.2.3.4 46618 - FTPSVC2 FTPSERVERNAME 5.6.7.8 21 PASS *** 530 1326 41 53 15 efccb174-1c47-46fc-8554-5b3fb0a7add9 - An+error+occured+during+the+authentication+process.

Thanks

[scottwilkerson]

You are passing the log to w3c->parse_csv(); which will use the following

Code: Select all

<Extension w3c>
Module xm_csv
Fields $date, $time, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $csUser-Agent, $cs-Referer, $sc-status, $sc-substatus, $sc-win32-status, $time-taken
FieldTypes string, string, string, string, string, string, integer, string, string, string, string, integer, integer, integer, integer
Delimiter ' '
QuoteChar '"'
EscapeControl FALSE
UndefValue -
</Extension>

The order of these Fields and FieldTypes doesn't match the fields in your log

Code: Select all

2020-08-06 22:29:21 1.2.3.4 46618 - FTPSVC2 FTPSERVERNAME 5.6.7.8 21 PASS *** 530 1326 41 53 15 efccb174-1c47-46fc-8554-5b3fb0a7add9 - An+error+occured+during+the+authentication+process.

I would suggest making a new extension to parse the FTP logs that has fields/values that match the data