Log messages being duplicated
Posted: Wed Nov 26, 2014 1:32 pm
I am working on adding tomcat logs to NLS, and for some reason, every line put into catalina.out is being duplicated in NLS.
Here is a breakdown of what is happening. Tomcat is NOT running. I am manually adding lines to the log:
> echo "Kajigger" >> /var/tomcat7/logs/catalina.out
I then look in NLS, and I see that line in there twice. (It is ok that it is failing to grok parse.. this is just an example with a simple message, same thing happens with messages that do properly parse)
This does not happen with other logs from the same server. I can echo a message into apache_access and it will only show up in NLS once.
Here is the filter I have setup for tomcat logs:
Here is the rsyslog config on the server:
Not sure what is going on here that is causing this. Any help would be appreciated
Here is a breakdown of what is happening. Tomcat is NOT running. I am manually adding lines to the log:
> echo "Kajigger" >> /var/tomcat7/logs/catalina.out
I then look in NLS, and I see that line in there twice. (It is ok that it is failing to grok parse.. this is just an example with a simple message, same thing happens with messages that do properly parse)
Code: Select all
{
"_index": "logstash-2014.11.26",
"_type": "tomcat",
"_id": "XVT3PYJaSiq5_hO1GCYYDw",
"_score": null,
"_source": {
"message": "Kajigger",
"@version": "1",
"@timestamp": "2014-11-26T17:52:05.000Z",
"type": "tomcat",
"host": "X.X.X.X",
"priority": 133,
"timestamp": "Nov 26 12:52:05",
"logsource": "tibet",
"program": "tomcat",
"severity": 5,
"facility": 16,
"facility_label": "local0",
"severity_label": "Notice",
"tags": [
"_grokparsefailure"
]
},
"highlight": {
"message": [
"@start-highlight@Kajigger@end-highlight@"
]
},
"sort": [
1417024325000,
1417024325000
]
}
{
"_index": "logstash-2014.11.26",
"_type": "tomcat",
"_id": "_01Fq5OASY6zWt4xDkDWCQ",
"_score": null,
"_source": {
"message": "Kajigger",
"@version": "1",
"@timestamp": "2014-11-26T17:52:05.000Z",
"type": "tomcat",
"host": "X.X.X.X",
"priority": 133,
"timestamp": "Nov 26 12:52:05",
"logsource": "tibet",
"program": "tomcat",
"severity": 5,
"facility": 16,
"facility_label": "local0",
"severity_label": "Notice",
"tags": [
"_grokparsefailure"
]
},
"highlight": {
"message": [
"@start-highlight@Kajigger@end-highlight@"
]
},
"sort": [
1417024325000,
1417024325000
]
}
Here is the filter I have setup for tomcat logs:
Code: Select all
if [program] == 'tomcat' {
grok {
match => [ 'message', '%{TIMESTAMP_ISO8601:timestamp}\-\[%{WORD:class}-%{LOGLEVEL:loglevel}\] \- afterSuccessfulAuthentication for Uauser\[%{INT:uauserid}\] Email\[%{NOTSPACE:email}\] IP\[%{IP:clientip}(?:%{SPACE},%{SPACE}(?:%{IP}|unknown))*\]' ]
match => [ 'message', '%{TIMESTAMP_ISO8601:timestamp}\-\[%{WORD:class}-%{LOGLEVEL:loglevel}\] \- Finished executing request \[%{NOTSPACE:request}\] for \[%{IP:clientip}(?:%{SPACE},%{SPACE}(?:%{IP}|unknown))*\]' ]
match => [ 'message', '%{TIMESTAMP_ISO8601:timestamp}\-\[%{WORD:class}-%{LOGLEVEL:loglevel}\] \- Got email activity event\=\[%{WORD:email_event}\] email\=\[%{NOTSPACE:email}\] category\=\[%{DATA:email_category}\]' ]
match => [ 'message', '%{TIMESTAMP_ISO8601:timestamp}\-\[%{WORD:class}-%{LOGLEVEL:loglevel}\] \- Sending e-mail to \[%{NOTSPACE:email}\]' ]
match => [ 'message', '%{TIMESTAMP_ISO8601:timestamp}\-\[%{WORD:class}-%{LOGLEVEL:loglevel}\] \- ' ]
match => [ 'message', '%{TIMESTAMP_ISO8601:timestamp}\: \[(?<garbage_collect>(?:Full )?GC) %{INT}K\-\>%{INT}K\(%{INT}K\)\, %{BASE10NUM} secs\]' ]
overwrite => [ 'timestamp' ]
}
date {
match => [ 'timestamp', 'yyyy-MM-dd HH:mm:ss,SSS', 'ISO8601' ]
}
mutate {
replace => [ 'type', 'tomcat' ]
}
}
Code: Select all
$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/lib/rsyslog
# Input for tomcat
$InputFileName /var/tomcat7/logs/catalina.out
$InputFileTag tomcat:
$InputFileStateFile nls-state-tomcat # Must be unique for each file being polled
# Uncomment the folowing line to override the default severity for messages
# from this file.
#$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor
# Forward to Nagios Log Server and then discard, otherwise these messages
# will end up in the syslog file (/var/log/messages) unless there are other
# overriding rules.
if $programname == 'tomcat' then @@XXXX:5544
if $programname == 'tomcat' then ~