Nagios Support Forum

Support for Nagios products and services
https://support.nagios.com/forum/

Logstash failing to include 1 message type in index

https://support.nagios.com/forum/viewtopic.php?t=35790

Page 1 of 1

Logstash failing to include 1 message type in index

Posted: Wed Nov 18, 2015 3:09 pm
by Jklre
Hey guys,

I noticed an issue with logstash dropping messages with the error "failed action with response of 400, dropping action:" This seems to be just this one particular error code which I think is strange. The text from this message works fine GROK debugger. It appears it has always been dropping this message we just haven't noticed it. Any Ideas on why this may be failing? Some googling only finds that it may be an issue with the name of the indexes? nothing conclusive though https://github.com/elastic/logstash/issues/3131

thanks you.

inputs

Code: Select all

input {
    syslog {
        type => 'syslog'
        port => 514
    }
    tcp {
        type => 'eventlog'
        port => 3515
        codec => json {
            charset => 'CP1252'
        }
    }
    tcp {
        type => 'import_raw'
        tags => 'import_raw'
        port => 2056
    }
    tcp {
        type => 'import_json'
        tags => 'import_json'
        port => 2057
        codec => json
    }
}
filters

Code: Select all

# 
# Logstash Configuration File
#
# Global filters
#

filter {
    if [program] == 'apache_access' {
        grok {
            match => [ 'message', '%{COMBINEDAPACHELOG}']
        }
        date {
            match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z' ]
        }
        mutate {
            replace => [ 'type', 'apache_access' ]
             convert => [ 'bytes', 'integer' ]
             convert => [ 'response', 'integer' ]
        }
    }
     
    
    if [type] == 'syslog' {
        grok {
            patterns_dir => "/usr/local/nagioslogserver/logstash/patterns/."
            break_on_match => true
            match => [ 'message', '%{SYSLOG5424PRI}%{MITCH_HOST:Mitch_host}%{GREEDYDATA:syslog_message}%%{CSTTERROR:CSTTError}%%%{MITCHERRORNUM:Mitch_ErrorNum}%%%{MITCHPRIORITY:Mitch_Priority}%%%{MITCH_ORG_HOST:Mitch_org_host}%{MITCHPROG:Mitch_prog}%{MITCHTIME:Mitchell_time}%{MITCHMESSAGE:Mitch_Message}%{MITCHMESSAGE2:Mitch_Message1}' ]
    add_tag => "Mitchell_Jboss_Log"
    add_field => {
       "CSTTLINK" => "http://cstt-prod.mitchell.com/Home/ErrorLogDetails?error_log_id=%{CSTTError}"
            }
        }
    }
    if [type] == 'syslog' {
        grok {
            patterns_dir => "/usr/local/nagioslogserver/logstash/patterns/."
            match => [ 'message', '%{SYSLOG5424PRI}Message forwarded from %{MITCH_IBM_HOST:Mitch_host}:%{GREEDYDATA:Mitch_Message}' ]
            add_tag => "Mitchell_IBM"
        }
    }
    if [type] == 'syslog' {
        grok {
            patterns_dir => "/usr/local/nagioslogserver/logstash/patterns/."
            match => [ 'message', '%{SYSLOG5424PRI}%{HEARTBEAT:Mitch_Message}' ]
            add_tag => "Mitchell_Heartbeat"
        }
    }
    if [type] == 'syslog' {
        grok {
            patterns_dir => "/usr/local/nagioslogserver/logstash/patterns/."
            match => [ 'message', '%{SYSLOG5424PRI}%{CISCOTIMESTAMP} %{MITCH_IBM_HOST:Mitch_host}%{GREEDYDATA:Mitch_Message}' ]
            add_tag => "Mitchell_MQChannelState"
        }
    }
    if [type] == 'syslog' {
        grok {
            patterns_dir => "/usr/local/nagioslogserver/logstash/patterns/."
            match => [ 'message', '(MQSeries:.Queue (above|below) depth threshold..QMgr=)%{MQserver:MQserver}.Que=%{MQNAME:MQname}' ]
            add_tag => "Mitchell_MQueueEvent"
        }
    }
}

#
# Local filters
#
Error Message

Code: Select all

{:timestamp=>"2015-11-18T05:42:03.267000-0800", :message=>"failed action with response of 400, dropping action: [\"index\", {:_id=>nil, :_index=>\"logstash-2015.11.18\", :_type=>\"syslog\", :_routing=>nil}, #<LogStash::Event:0x2f79dfd1 @metadata={\"retry_count\"=>0}, @accessors=#<LogStash::Util::Accessors:0x7d65bf59 @store={\"message\"=>\"<11>pjbc11n4lxv jboss_jbc11: node4: [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %%-1%%55005%%NONFATAL%%pjbc11n4lxv.mitchell.com%%CORESERVICES%%ERRORLOG_SERVICE%%Nov 18, 2015 5:42:02 AM%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%[jboss_jbc11] XML failed validation.\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2015-11-18T13:42:02.776Z\", \"type\"=>\"syslog\", \"host\"=>\"172.22.1.68\", \"tags\"=>[\"_grokparsefailure_sysloginput\", \"Mitchell_Jboss_Log\", \"_grokparsefailure\"], \"priority\"=>0, \"severity\"=>0, \"facility\"=>0, \"facility_label\"=>\"kernel\", \"severity_label\"=>\"Emergency\", \"syslog5424_pri\"=>\"11\", \"Mitch_host\"=>\"pjbc11n4lxv jboss_jbc11: node4:\", \"syslog_message\"=>\" [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %\", \"CSTTError\"=>\"-1\", \"Mitch_ErrorNum\"=>\"55005\", \"Mitch_Priority\"=>\"NONFATAL\", \"Mitch_org_host\"=>\"pjbc11n4lxv.mitchell.com\", \"Mitch_prog\"=>\"%%CORESERVICES%%ERRORLOG_SERVICE%%\", \"Mitchell_time\"=>\"Nov 18, 2015 5:42:02 AM\", \"Mitch_Message\"=>\"%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%\", \"Mitch_Message1\"=>\"[jboss_jbc11] XML failed validation.\\n\", \"CSTTLINK\"=>\"http://cstt-prod.mitchell.com/Home/ErrorLogDetails?error_log_id=-1\"}, @lut={\"type\"=>[{\"message\"=>\"<11>pjbc11n4lxv jboss_jbc11: node4: [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %%-1%%55005%%NONFATAL%%pjbc11n4lxv.mitchell.com%%CORESERVICES%%ERRORLOG_SERVICE%%Nov 18, 2015 5:42:02 AM%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%[jboss_jbc11] XML failed validation.\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2015-11-18T13:42:02.776Z\", \"type\"=>\"syslog\", \"host\"=>\"172.22.1.68\", \"tags\"=>[\"_grokparsefailure_sysloginput\", \"Mitchell_Jboss_Log\", \"_grokparsefailure\"], \"priority\"=>0, \"severity\"=>0, \"facility\"=>0, \"facility_label\"=>\"kernel\", \"severity_label\"=>\"Emergency\", \"syslog5424_pri\"=>\"11\", \"Mitch_host\"=>\"pjbc11n4lxv jboss_jbc11: node4:\", \"syslog_message\"=>\" [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %\", \"CSTTError\"=>\"-1\", \"Mitch_ErrorNum\"=>\"55005\", \"Mitch_Priority\"=>\"NONFATAL\", \"Mitch_org_host\"=>\"pjbc11n4lxv.mitchell.com\", \"Mitch_prog\"=>\"%%CORESERVICES%%ERRORLOG_SERVICE%%\", \"Mitchell_time\"=>\"Nov 18, 2015 5:42:02 AM\", \"Mitch_Message\"=>\"%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%\", \"Mitch_Message1\"=>\"[jboss_jbc11] XML failed validation.\\n\", \"CSTTLINK\"=>\"http://cstt-prod.mitchell.com/Home/ErrorLogDetails?error_log_id=-1\"}, \"type\"], \"host\"=>[{\"message\"=>\"<11>pjbc11n4lxv jboss_jbc11: node4: [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %%-1%%55005%%NONFATAL%%pjbc11n4lxv.mitchell.com%%CORESERVICES%%ERRORLOG_SERVICE%%Nov 18, 2015 5:42:02 AM%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%[jboss_jbc11] XML failed validation.\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2015-11-18T13:42:02.776Z\", \"type\"=>\"syslog\", \"host\"=>\"172.22.1.68\", \"tags\"=>[\"_grokparsefailure_sysloginput\", \"Mitchell_Jboss_Log\", \"_grokparsefailure\"], \"priority\"=>0, \"severity\"=>0, \"facility\"=>0, \"facility_label\"=>\"kernel\", \"severity_label\"=>\"Emergency\", \"syslog5424_pri\"=>\"11\", \"Mitch_host\"=>\"pjbc11n4lxv jboss_jbc11: node4:\", \"syslog_message\"=>\" [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %\", \"CSTTError\"=>\"-1\", \"Mitch_ErrorNum\"=>\"55005\", \"Mitch_Priority\"=>\"NONFATAL\", \"Mitch_org_host\"=>\"pjbc11n4lxv.mitchell.com\", \"Mitch_prog\"=>\"%%CORESERVICES%%ERRORLOG_SERVICE%%\", \"Mitchell_time\"=>\"Nov 18, 2015 5:42:02 AM\", \"Mitch_Message\"=>\"%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%\", \"Mitch_Message1\"=>\"[jboss_jbc11] XML failed validation.\\n\", \"CSTTLINK\"=>\"http://cstt-prod.mitchell.com/Home/ErrorLogDetails?error_log_id=-1\"}, \"host\"], \"message\"=>[{\"message\"=>\"<11>pjbc11n4lxv jboss_jbc11: node4: [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %%-1%%55005%%NONFATAL%%pjbc11n4lxv.mitchell.com%%CORESERVICES%%ERRORLOG_SERVICE%%Nov 18, 2015 5:42:02 AM%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%[jboss_jbc11] XML failed validation.\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2015-11-18T13:42:02.776Z\", \"type\"=>\"syslog\", \"host\"=>\"172.22.1.68\", \"tags\"=>[\"_grokparsefailure_sysloginput\", \"Mitchell_Jboss_Log\", \"_grokparsefailure\"], \"priority\"=>0, \"severity\"=>0, \"facility\"=>0, \"facility_label\"=>\"kernel\", \"severity_label\"=>\"Emergency\", \"syslog5424_pri\"=>\"11\", \"Mitch_host\"=>\"pjbc11n4lxv jboss_jbc11: node4:\", \"syslog_message\"=>\" [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %\", \"CSTTError\"=>\"-1\", \"Mitch_ErrorNum\"=>\"55005\", \"Mitch_Priority\"=>\"NONFATAL\", \"Mitch_org_host\"=>\"pjbc11n4lxv.mitchell.com\", \"Mitch_prog\"=>\"%%CORESERVICES%%ERRORLOG_SERVICE%%\", \"Mitchell_time\"=>\"Nov 18, 2015 5:42:02 AM\", \"Mitch_Message\"=>\"%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%\", \"Mitch_Message1\"=>\"[jboss_jbc11] XML failed validation.\\n\", \"CSTTLINK\"=>\"http://cstt-prod.mitchell.com/Home/ErrorLogDetails?error_log_id=-1\"}, \"message\"], \"tags\"=>[{\"message\"=>\"<11>pjbc11n4lxv jboss_jbc11: node4: [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %%-1%%55005%%NONFATAL%%pjbc11n4lxv.mitchell.com%%CORESERVICES%%ERRORLOG_SERVICE%%Nov 18, 2015 5:42:02 AM%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%[jboss_jbc11] XML failed validation.\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2015-11-18T13:42:02.776Z\", \"type\"=>\"syslog\", \"host\"=>\"172.22.1.68\", \"tags\"=>[\"_grokparsefailure_sysloginput\", \"Mitchell_Jboss_Log\", \"_grokparsefailure\"], \"priority\"=>0, \"severity\"=>0, \"facility\"=>0, \"facility_label\"=>\"kernel\", \"severity_label\"=>\"Emergency\", \"syslog5424_pri\"=>\"11\", \"Mitch_host\"=>\"pjbc11n4lxv jboss_jbc11: node4:\", \"syslog_message\"=>\" [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %\", \"CSTTError\"=>\"-1\", \"Mitch_ErrorNum\"=>\"55005\", \"Mitch_Priority\"=>\"NONFATAL\", \"Mitch_org_host\"=>\"pjbc11n4lxv.mitchell.com\", \"Mitch_prog\"=>\"%%CORESERVICES%%ERRORLOG_SERVICE%%\", \"Mitchell_time\"=>\"Nov 18, 2015 5:42:02 AM\", \"Mitch_Message\"=>\"%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%\", \"Mitch_Message1\"=>\"[jboss_jbc11] XML failed validation.\\n\", \"CSTTLINK\"=>\"http://cstt-prod.mitchell.com/Home/ErrorLogDetails?error_log_id=-1\"}, \"tags\"], \"priority\"=>[{\"message\"=>\"<11>pjbc11n4lxv jboss_jbc11: node4: [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %%-1%%55005%%NONFATAL%%pjbc11n4lxv.mitchell.com%%CORESERVICES%%ERRORLOG_SERVICE%%Nov 18, 2015 5:42:02 AM%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%[jboss_jbc11] XML failed validation.\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2015-11-18T13:42:02.776Z\", \"type\"=>\"syslog\", \"host\"=>\"172.22.1.68\", \"tags\"=>[\"_grokparsefailure_sysloginput\", \"Mitchell_Jboss_Log\", \"_grokparsefailure\"], \"priority\"=>0, \"severity\"=>0, \"facility\"=>0, \"facility_label\"=>\"kernel\", \"severity_label\"=>\"Emergency\", \"syslog5424_pri\"=>\"11\", \"Mitch_host\"=>\"pjbc11n4lxv jboss_jbc11: node4:\", \"syslog_message\"=>\" [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %\", \"CSTTError\"=>\"-1\", \"Mitch_ErrorNum\"=>\"55005\", \"Mitch_Priority\"=>\"NONFATAL\", \"Mitch_org_host\"=>\"pjbc11n4lxv.mitchell.com\", \"Mitch_prog\"=>\"%%CORESERVICES%%ERRORLOG_SERVICE%%\", \"Mitchell_time\"=>\"Nov 18, 2015 5:42:02 AM\", \"Mitch_Message\"=>\"%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%\", \"Mitch_Message1\"=>\"[jboss_jbc11] XML failed validation.\\n\", \"CSTTLINK\"=>\"http://cstt-prod.mitchell.com/Home/ErrorLogDetails?error_log_id=-1\"}, \"priority\"], \"severity\"=>[{\"message\"=>\"<11>pjbc11n4lxv jboss_jbc11: node4: [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %%-1%%55005%%NONFATAL%%pjbc11n4lxv.mitchell.com%%CORESERVICES%%ERRORLOG_SERVICE%%Nov 18, 2015 5:42:02 AM%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%[jboss_jbc11] XML failed validation.\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2015-11-18T13:42:02.776Z\", \"type\"=>\"syslog\", \"host\"=>\"172.22.1.68\", \"tags\"=>[\"_grokparsefailure_sysloginput\", \"Mitchell_Jboss_Log\", \"_grokparsefailure\"], \"priority\"=>0, \"severity\"=>0, \"facility\"=>0, \"facility_label\"=>\"kernel\", \"severity_label\"=>\"Emergency\", \"syslog5424_pri\"=>\"11\", \"Mitch_host\"=>\"pjbc11n4lxv jboss_jbc11: node4:\", \"syslog_message\"=>\" [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %\", \"CSTTError\"=>\"-1\", \"Mitch_ErrorNum\"=>\"55005\", \"Mitch_Priority\"=>\"NONFATAL\", \"Mitch_org_host\"=>\"pjbc11n4lxv.mitchell.com\", \"Mitch_prog\"=>\"%%CORESERVICES%%ERRORLOG_SERVICE%%\", \"Mitchell_time\"=>\"Nov 18, 2015 5:42:02 AM\", \"Mitch_Message\"=>\"%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%\", \"Mitch_Message1\"=>\"[jboss_jbc11] XML failed validation.\\n\", \"CSTTLINK\"=>\"http://cstt-prod.mitchell.com/Home/ErrorLogDetails?error_log_id=-1\"}, \"severity\"], \"facility\"=>[{\"message\"=>\"<11>pjbc11n4lxv jboss_jbc11: node4: [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %%-1%%55005%%NONFATAL%%pjbc11n4lxv.mitchell.com%%CORESERVICES%%ERRORLOG_SERVICE%%Nov 18, 2015 5:42:02 AM%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%[jboss_jbc11] XML failed validation.\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2015-11-18T13:42:02.776Z\", \"type\"=>\"syslog\", \"host\"=>\"172.22.1.68\", \"tags\"=>[\"_grokparsefailure_sysloginput\", \"Mitchell_Jboss_Log\", \"_grokparsefailure\"], \"priority\"=>0, \"severity\"=>0, \"facility\"=>0, \"facility_label\"=>\"kernel\", \"severity_label\"=>\"Emergency\", \"syslog5424_pri\"=>\"11\", \"Mitch_host\"=>\"pjbc11n4lxv jboss_jbc11: node4:\", \"syslog_message\"=>\" [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %\", \"CSTTError\"=>\"-1\", \"Mitch_ErrorNum\"=>\"55005\", \"Mitch_Priority\"=>\"NONFATAL\", \"Mitch_org_host\"=>\"pjbc11n4lxv.mitchell.com\", \"Mitch_prog\"=>\"%%CORESERVICES%%ERRORLOG_SERVICE%%\", \"Mitchell_time\"=>\"Nov 18, 2015 5:42:02 AM\", \"Mitch_Message\"=>\"%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%\", \"Mitch_Message1\"=>\"[jboss_jbc11] XML failed validation.\\n\", \"CSTTLINK\"=>\"http://cstt-prod.mitchell.com/Home/ErrorLogDetails?error_log_id=-1\"}, \"facility\"], \"timestamp8601\"=>[{\"message\"=>\"<11>pjbc11n4lxv jboss_jbc11: node4: [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %%-1%%55005%%NONFATAL%%pjbc11n4lxv.mitchell.com%%CORESERVICES%%ERRORLOG_SERVICE%%Nov 18, 2015 5:42:02 AM%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%[jboss_jbc11] XML failed validation.\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2015-11-18T13:42:02.776Z\", \"type\"=>\"syslog\", \"host\"=>\"172.22.1.68\", \"tags\"=>[\"_grokparsefailure_sysloginput\", \"Mitchell_Jboss_Log\", \"_grokparsefailure\"], \"priority\"=>0, \"severity\"=>0, \"facility\"=>0, \"facility_label\"=>\"kernel\", \"severity_label\"=>\"Emergency\", \"syslog5424_pri\"=>\"11\", \"Mitch_host\"=>\"pjbc11n4lxv jboss_jbc11: node4:\", \"syslog_message\"=>\" [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %\", \"CSTTError\"=>\"-1\", \"Mitch_ErrorNum\"=>\"55005\", \"Mitch_Priority\"=>\"NONFATAL\", \"Mitch_org_host\"=>\"pjbc11n4lxv.mitchell.com\", \"Mitch_prog\"=>\"%%CORESERVICES%%ERRORLOG_SERVICE%%\", \"Mitchell_time\"=>\"Nov 18, 2015 5:42:02 AM\", \"Mitch_Message\"=>\"%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%\", \"Mitch_Message1\"=>\"[jboss_jbc11] XML failed validation.\\n\", \"CSTTLINK\"=>\"http://cstt-prod.mitchell.com/Home/ErrorLogDetails?error_log_id=-1\"}, \"timestamp8601\"], \"timestamp\"=>[{\"message\"=>\"<11>pjbc11n4lxv jboss_jbc11: node4: [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %%-1%%55005%%NONFATAL%%pjbc11n4lxv.mitchell.com%%CORESERVICES%%ERRORLOG_SERVICE%%Nov 18, 2015 5:42:02 AM%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%[jboss_jbc11] XML failed validation.\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2015-11-18T13:42:02.776Z\", \"type\"=>\"syslog\", \"host\"=>\"172.22.1.68\", \"tags\"=>[\"_grokparsefailure_sysloginput\", \"Mitchell_Jboss_Log\", \"_grokparsefailure\"], \"priority\"=>0, \"severity\"=>0, \"facility\"=>0, \"facility_label\"=>\"kernel\", \"severity_label\"=>\"Emergency\", \"syslog5424_pri\"=>\"11\", \"Mitch_host\"=>\"pjbc11n4lxv jboss_jbc11: node4:\", \"syslog_message\"=>\" [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %\", \"CSTTError\"=>\"-1\", \"Mitch_ErrorNum\"=>\"55005\", \"Mitch_Priority\"=>\"NONFATAL\", \"Mitch_org_host\"=>\"pjbc11n4lxv.mitchell.com\", \"Mitch_prog\"=>\"%%CORESERVICES%%ERRORLOG_SERVICE%%\", \"Mitchell_time\"=>\"Nov 18, 2015 5:42:02 AM\", \"Mitch_Message\"=>\"%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%\", \"Mitch_Message1\"=>\"[jboss_jbc11] XML failed validation.\\n\", \"CSTTLINK\"=>\"http://cstt-prod.mitchell.com/Home/ErrorLogDetails?error_log_id=-1\"}, \"timestamp\"], \"facility_label\"=>[{\"message\"=>\"<11>pjbc11n4lxv jboss_jbc11: node4: [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %%-1%%55005%%NONFATAL%%pjbc11n4lxv.mitchell.com%%CORESERVICES%%ERRORLOG_SERVICE%%Nov 18, 2015 5:42:02 AM%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%[jboss_jbc11] XML failed validation.\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2015-11-18T13:42:02.776Z\", \"type\"=>\"syslog\", \"host\"=>\"172.22.1.68\", \"tags\"=>[\"_grokparsefailure_sysloginput\", \"Mitchell_Jboss_Log\", \"_grokparsefailure\"], \"priority\"=>0, \"severity\"=>0, \"facility\"=>0, \"facility_label\"=>\"kernel\", \"severity_label\"=>\"Emergency\", \"syslog5424_pri\"=>\"11\", \"Mitch_host\"=>\"pjbc11n4lxv jboss_jbc11: node4:\", \"syslog_message\"=>\" [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %\", \"CSTTError\"=>\"-1\", \"Mitch_ErrorNum\"=>\"55005\", \"Mitch_Priority\"=>\"NONFATAL\", \"Mitch_org_host\"=>\"pjbc11n4lxv.mitchell.com\", \"Mitch_prog\"=>\"%%CORESERVICES%%ERRORLOG_SERVICE%%\", \"Mitchell_time\"=>\"Nov 18, 2015 5:42:02 AM\", \"Mitch_Message\"=>\"%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%\", \"Mitch_Message1\"=>\"[jboss_jbc11] XML failed validation.\\n\", \"CSTTLINK\"=>\"http://cstt-prod.mitchell.com/Home/ErrorLogDetails?error_log_id=-1\"}, \"facility_label\"], \"severity_label\"=>[{\"message\"=>\"<11>pjbc11n4lxv jboss_jbc11: node4: [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %%-1%%55005%%NONFATAL%%pjbc11n4lxv.mitchell.com%%CORESERVICES%%ERRORLOG_SERVICE%%Nov 18, 2015 5:42:02 AM%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%[jboss_jbc11] XML failed validation.\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2015-11-18T13:42:02.776Z\", \"type\"=>\"syslog\", \"host\"=>\"172.22.1.68\", \"tags\"=>[\"_grokparsefailure_sysloginput\", \"Mitchell_Jboss_Log\", \"_grokparsefailure\"], \"priority\"=>0, \"severity\"=>0, \"facility\"=>0, \"facility_label\"=>\"kernel\", \"severity_label\"=>\"Emergency\", \"syslog5424_pri\"=>\"11\", \"Mitch_host\"=>\"pjbc11n4lxv jboss_jbc11: node4:\", \"syslog_message\"=>\" [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %\", \"CSTTError\"=>\"-1\", \"Mitch_ErrorNum\"=>\"55005\", \"Mitch_Priority\"=>\"NONFATAL\", \"Mitch_org_host\"=>\"pjbc11n4lxv.mitchell.com\", \"Mitch_prog\"=>\"%%CORESERVICES%%ERRORLOG_SERVICE%%\", \"Mitchell_time\"=>\"Nov 18, 2015 5:42:02 AM\", \"Mitch_Message\"=>\"%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%\", \"Mitch_Message1\"=>\"[jboss_jbc11] XML failed validation.\\n\", \"CSTTLINK\"=>\"http://cstt-prod.mitchell.com/Home/ErrorLogDetails?error_log_id=-1\"}, \"severity_label\"], \"[program]\"=>[{\"message\"=>\"<11>pjbc11n4lxv jboss_jbc11: node4: [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %%-1%%55005%%NONFATAL%%pjbc11n4lxv.mitchell.com%%CORESERVICES%%ERRORLOG_SERVICE%%Nov 18, 2015 5:42:02 AM%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%[jboss_jbc11] XML failed validation.\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2015-11-18T13:42:02.776Z\", \"type\"=>\"syslog\", \"host\"=>\"172.22.1.68\", \"tags\"=>[\"_grokparsefailure_sysloginput\", \"Mitchell_Jboss_Log\", \"_grokparsefailure\"], \"priority\"=>0, \"severity\"=>0, \"facility\"=>0, \"facility_label\"=>\"kernel\", \"severity_label\"=>\"Emergency\", \"syslog5424_pri\"=>\"11\", \"Mitch_host\"=>\"pjbc11n4lxv jboss_jbc11: node4:\", \"syslog_message\"=>\" [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %\", \"CSTTError\"=>\"-1\", \"Mitch_ErrorNum\"=>\"55005\", \"Mitch_Priority\"=>\"NONFATAL\", \"Mitch_org_host\"=>\"pjbc11n4lxv.mitchell.com\", \"Mitch_prog\"=>\"%%CORESERVICES%%ERRORLOG_SERVICE%%\", \"Mitchell_time\"=>\"Nov 18, 2015 5:42:02 AM\", \"Mitch_Message\"=>\"%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%\", \"Mitch_Message1\"=>\"[jboss_jbc11] XML failed validation.\\n\", \"CSTTLINK\"=>\"http://cstt-prod.mitchell.com/Home/ErrorLogDetails?error_log_id=-1\"}, \"program\"], \"[type]\"=>[{\"message\"=>\"<11>pjbc11n4lxv jboss_jbc11: node4: [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %%-1%%55005%%NONFATAL%%pjbc11n4lxv.mitchell.com%%CORESERVICES%%ERRORLOG_SERVICE%%Nov 18, 2015 5:42:02 AM%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%[jboss_jbc11] XML failed validation.\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2015-11-18T13:42:02.776Z\", \"type\"=>\"syslog\", \"host\"=>\"172.22.1.68\", \"tags\"=>[\"_grokparsefailure_sysloginput\", \"Mitchell_Jboss_Log\", \"_grokparsefailure\"], \"priority\"=>0, \"severity\"=>0, \"facility\"=>0, \"facility_label\"=>\"kernel\", \"severity_label\"=>\"Emergency\", \"syslog5424_pri\"=>\"11\", \"Mitch_host\"=>\"pjbc11n4lxv jboss_jbc11: node4:\", \"syslog_message\"=>\" [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %\", \"CSTTError\"=>\"-1\", \"Mitch_ErrorNum\"=>\"55005\", \"Mitch_Priority\"=>\"NONFATAL\", \"Mitch_org_host\"=>\"pjbc11n4lxv.mitchell.com\", \"Mitch_prog\"=>\"%%CORESERVICES%%ERRORLOG_SERVICE%%\", \"Mitchell_time\"=>\"Nov 18, 2015 5:42:02 AM\", \"Mitch_Message\"=>\"%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%\", \"Mitch_Message1\"=>\"[jboss_jbc11] XML failed validation.\\n\", \"CSTTLINK\"=>\"http://cstt-prod.mitchell.com/Home/ErrorLogDetails?error_log_id=-1\"}, \"type\"], \"syslog5424_pri\"=>[{\"message\"=>\"<11>pjbc11n4lxv jboss_jbc11: node4: [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %%-1%%55005%%NONFATAL%%pjbc11n4lxv.mitchell.com%%CORESERVICES%%ERRORLOG_SERVICE%%Nov 18, 2015 5:42:02 AM%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%[jboss_jbc11] XML failed validation.\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2015-11-18T13:42:02.776Z\", \"type\"=>\"syslog\", \"host\"=>\"172.22.1.68\", \"tags\"=>[\"_grokparsefailure_sysloginput\", \"Mitchell_Jboss_Log\", \"_grokparsefailure\"], \"priority\"=>0, \"severity\"=>0, \"facility\"=>0, \"facility_label\"=>\"kernel\", \"severity_label\"=>\"Emergency\", \"syslog5424_pri\"=>\"11\", \"Mitch_host\"=>\"pjbc11n4lxv jboss_jbc11: node4:\", \"syslog_message\"=>\" [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %\", \"CSTTError\"=>\"-1\", \"Mitch_ErrorNum\"=>\"55005\", \"Mitch_Priority\"=>\"NONFATAL\", \"Mitch_org_host\"=>\"pjbc11n4lxv.mitchell.com\", \"Mitch_prog\"=>\"%%CORESERVICES%%ERRORLOG_SERVICE%%\", \"Mitchell_time\"=>\"Nov 18, 2015 5:42:02 AM\", \"Mitch_Message\"=>\"%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%\", \"Mitch_Message1\"=>\"[jboss_jbc11] XML failed validation.\\n\", \"CSTTLINK\"=>\"http://cstt-prod.mitchell.com/Home/ErrorLogDetails?error_log_id=-1\"}, \"syslog5424_pri\"], \"Mitch_host\"=>[{\"message\"=>\"<11>pjbc11n4lxv jboss_jbc11: node4: [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %%-1%%55005%%NONFATAL%%pjbc11n4lxv.mitchell.com%%CORESERVICES%%ERRORLOG_SERVICE%%Nov 18, 2015 5:42:02 AM%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%[jboss_jbc11] XML failed validation.\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2015-11-18T13:42:02.776Z\", \"type\"=>\"syslog\", \"host\"=>\"172.22.1.68\", \"tags\"=>[\"_grokparsefailure_sysloginput\", \"Mitchell_Jboss_Log\", \"_grokparsefailure\"], \"priority\"=>0, \"severity\"=>0, \"facility\"=>0, \"facility_label\"=>\"kernel\", \"severity_label\"=>\"Emergency\", \"syslog5424_pri\"=>\"11\", \"Mitch_host\"=>\"pjbc11n4lxv jboss_jbc11: node4:\", \"syslog_message\"=>\" [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %\", \"CSTTError\"=>\"-1\", \"Mitch_ErrorNum\"=>\"55005\", \"Mitch_Priority\"=>\"NONFATAL\", \"Mitch_org_host\"=>\"pjbc11n4lxv.mitchell.com\", \"Mitch_prog\"=>\"%%CORESERVICES%%ERRORLOG_SERVICE%%\", \"Mitchell_time\"=>\"Nov 18, 2015 5:42:02 AM\", \"Mitch_Message\"=>\"%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%\", \"Mitch_Message1\"=>\"[jboss_jbc11] XML failed validation.\\n\", \"CSTTLINK\"=>\"http://cstt-prod.mitchell.com/Home/ErrorLogDetails?error_log_id=-1\"}, \"Mitch_host\"], \"syslog_message\"=>[{\"message\"=>\"<11>pjbc11n4lxv jboss_jbc11: node4: [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %%-1%%55005%%NONFATAL%%pjbc11n4lxv.mitchell.com%%CORESERVICES%%ERRORLOG_SERVICE%%Nov 18, 2015 5:42:02 AM%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%[jboss_jbc11] XML failed validation.\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2015-11-18T13:42:02.776Z\", \"type\"=>\"syslog\", \"host\"=>\"172.22.1.68\", \"tags\"=>[\"_grokparsefailure_sysloginput\", \"Mitchell_Jboss_Log\", \"_grokparsefailure\"], \"priority\"=>0, \"severity\"=>0, \"facility\"=>0, \"facility_label\"=>\"kernel\", \"severity_label\"=>\"Emergency\", \"syslog5424_pri\"=>\"11\", \"Mitch_host\"=>\"pjbc11n4lxv jboss_jbc11: node4:\", \"syslog_message\"=>\" [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %\", \"CSTTError\"=>\"-1\", \"Mitch_ErrorNum\"=>\"55005\", \"Mitch_Priority\"=>\"NONFATAL\", \"Mitch_org_host\"=>\"pjbc11n4lxv.mitchell.com\", \"Mitch_prog\"=>\"%%CORESERVICES%%ERRORLOG_SERVICE%%\", \"Mitchell_time\"=>\"Nov 18, 2015 5:42:02 AM\", \"Mitch_Message\"=>\"%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%\", \"Mitch_Message1\"=>\"[jboss_jbc11] XML failed validation.\\n\", \"CSTTLINK\"=>\"http://cstt-prod.mitchell.com/Home/ErrorLogDetails?error_log_id=-1\"}, \"syslog_message\"], \"CSTTError\"=>[{\"message\"=>\"<11>pjbc11n4lxv jboss_jbc11: node4: [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %%-1%%55005%%NONFATAL%%pjbc11n4lxv.mitchell.com%%CORESERVICES%%ERRORLOG_SERVICE%%Nov 18, 2015 5:42:02 AM%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%[jboss_jbc11] XML failed validation.\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2015-11-18T13:42:02.776Z\", \"type\"=>\"syslog\", \"host\"=>\"172.22.1.68\", \"tags\"=>[\"_grokparsefailure_sysloginput\", \"Mitchell_Jboss_Log\", \"_grokparsefailure\"], \"priority\"=>0, \"severity\"=>0, \"facility\"=>0, \"facility_label\"=>\"kernel\", \"severity_label\"=>\"Emergency\", \"syslog5424_pri\"=>\"11\", \"Mitch_host\"=>\"pjbc11n4lxv jboss_jbc11: node4:\", \"syslog_message\"=>\" [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %\", \"CSTTError\"=>\"-1\", \"Mitch_ErrorNum\"=>\"55005\", \"Mitch_Priority\"=>\"NONFATAL\", \"Mitch_org_host\"=>\"pjbc11n4lxv.mitchell.com\", \"Mitch_prog\"=>\"%%CORESERVICES%%ERRORLOG_SERVICE%%\", \"Mitchell_time\"=>\"Nov 18, 2015 5:42:02 AM\", \"Mitch_Message\"=>\"%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%\", \"Mitch_Message1\"=>\"[jboss_jbc11] XML failed validation.\\n\", \"CSTTLINK\"=>\"http://cstt-prod.mitchell.com/Home/ErrorLogDetails?error_log_id=-1\"}, \"CSTTError\"], \"Mitch_ErrorNum\"=>[{\"message\"=>\"<11>pjbc11n4lxv jboss_jbc11: node4: [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %%-1%%55005%%NONFATAL%%pjbc11n4lxv.mitchell.com%%CORESERVICES%%ERRORLOG_SERVICE%%Nov 18, 2015 5:42:02 AM%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%[jboss_jbc11] XML failed validation.\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2015-11-18T13:42:02.776Z\", \"type\"=>\"syslog\", \"host\"=>\"172.22.1.68\", \"tags\"=>[\"_grokparsefailure_sysloginput\", \"Mitchell_Jboss_Log\", \"_grokparsefailure\"], \"priority\"=>0, \"severity\"=>0, \"facility\"=>0, \"facility_label\"=>\"kernel\", \"severity_label\"=>\"Emergency\", \"syslog5424_pri\"=>\"11\", \"Mitch_host\"=>\"pjbc11n4lxv jboss_jbc11: node4:\", \"syslog_message\"=>\" [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %\", \"CSTTError\"=>\"-1\", \"Mitch_ErrorNum\"=>\"55005\", \"Mitch_Priority\"=>\"NONFATAL\", \"Mitch_org_host\"=>\"pjbc11n4lxv.mitchell.com\", \"Mitch_prog\"=>\"%%CORESERVICES%%ERRORLOG_SERVICE%%\", \"Mitchell_time\"=>\"Nov 18, 2015 5:42:02 AM\", \"Mitch_Message\"=>\"%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%\", \"Mitch_Message1\"=>\"[jboss_jbc11] XML failed validation.\\n\", \"CSTTLINK\"=>\"http://cstt-prod.mitchell.com/Home/ErrorLogDetails?error_log_id=-1\"}, \"Mitch_ErrorNum\"], \"Mitch_Priority\"=>[{\"message\"=>\"<11>pjbc11n4lxv jboss_jbc11: node4: [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %%-1%%55005%%NONFATAL%%pjbc11n4lxv.mitchell.com%%CORESERVICES%%ERRORLOG_SERVICE%%Nov 18, 2015 5:42:02 AM%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%[jboss_jbc11] XML failed validation.\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2015-11-18T13:42:02.776Z\", \"type\"=>\"syslog\", \"host\"=>\"172.22.1.68\", \"tags\"=>[\"_grokparsefailure_sysloginput\", \"Mitchell_Jboss_Log\", \"_grokparsefailure\"], \"priority\"=>0, \"severity\"=>0, \"facility\"=>0, \"facility_label\"=>\"kernel\", \"severity_label\"=>\"Emergency\", \"syslog5424_pri\"=>\"11\", \"Mitch_host\"=>\"pjbc11n4lxv jboss_jbc11: node4:\", \"syslog_message\"=>\" [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %\", \"CSTTError\"=>\"-1\", \"Mitch_ErrorNum\"=>\"55005\", \"Mitch_Priority\"=>\"NONFATAL\", \"Mitch_org_host\"=>\"pjbc11n4lxv.mitchell.com\", \"Mitch_prog\"=>\"%%CORESERVICES%%ERRORLOG_SERVICE%%\", \"Mitchell_time\"=>\"Nov 18, 2015 5:42:02 AM\", \"Mitch_Message\"=>\"%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%\", \"Mitch_Message1\"=>\"[jboss_jbc11] XML failed validation.\\n\", \"CSTTLINK\"=>\"http://cstt-prod.mitchell.com/Home/ErrorLogDetails?error_log_id=-1\"}, \"Mitch_Priority\"], \"Mitch_org_host\"=>[{\"message\"=>\"<11>pjbc11n4lxv jboss_jbc11: node4: [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %%-1%%55005%%NONFATAL%%pjbc11n4lxv.mitchell.com%%CORESERVICES%%ERRORLOG_SERVICE%%Nov 18, 2015 5:42:02 AM%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%[jboss_jbc11] XML failed validation.\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2015-11-18T13:42:02.776Z\", \"type\"=>\"syslog\", \"host\"=>\"172.22.1.68\", \"tags\"=>[\"_grokparsefailure_sysloginput\", \"Mitchell_Jboss_Log\", \"_grokparsefailure\"], \"priority\"=>0, \"severity\"=>0, \"facility\"=>0, \"facility_label\"=>\"kernel\", \"severity_label\"=>\"Emergency\", \"syslog5424_pri\"=>\"11\", \"Mitch_host\"=>\"pjbc11n4lxv jboss_jbc11: node4:\", \"syslog_message\"=>\" [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %\", \"CSTTError\"=>\"-1\", \"Mitch_ErrorNum\"=>\"55005\", \"Mitch_Priority\"=>\"NONFATAL\", \"Mitch_org_host\"=>\"pjbc11n4lxv.mitchell.com\", \"Mitch_prog\"=>\"%%CORESERVICES%%ERRORLOG_SERVICE%%\", \"Mitchell_time\"=>\"Nov 18, 2015 5:42:02 AM\", \"Mitch_Message\"=>\"%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%\", \"Mitch_Message1\"=>\"[jboss_jbc11] XML failed validation.\\n\", \"CSTTLINK\"=>\"http://cstt-prod.mitchell.com/Home/ErrorLogDetails?error_log_id=-1\"}, \"Mitch_org_host\"], \"Mitch_prog\"=>[{\"message\"=>\"<11>pjbc11n4lxv jboss_jbc11: node4: [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %%-1%%55005%%NONFATAL%%pjbc11n4lxv.mitchell.com%%CORESERVICES%%ERRORLOG_SERVICE%%Nov 18, 2015 5:42:02 AM%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%[jboss_jbc11] XML failed validation.\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2015-11-18T13:42:02.776Z\", \"type\"=>\"syslog\", \"host\"=>\"172.22.1.68\", \"tags\"=>[\"_grokparsefailure_sysloginput\", \"Mitchell_Jboss_Log\", \"_grokparsefailure\"], \"priority\"=>0, \"severity\"=>0, \"facility\"=>0, \"facility_label\"=>\"kernel\", \"severity_label\"=>\"Emergency\", \"syslog5424_pri\"=>\"11\", \"Mitch_host\"=>\"pjbc11n4lxv jboss_jbc11: node4:\", \"syslog_message\"=>\" [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %\", \"CSTTError\"=>\"-1\", \"Mitch_ErrorNum\"=>\"55005\", \"Mitch_Priority\"=>\"NONFATAL\", \"Mitch_org_host\"=>\"pjbc11n4lxv.mitchell.com\", \"Mitch_prog\"=>\"%%CORESERVICES%%ERRORLOG_SERVICE%%\", \"Mitchell_time\"=>\"Nov 18, 2015 5:42:02 AM\", \"Mitch_Message\"=>\"%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%\", \"Mitch_Message1\"=>\"[jboss_jbc11] XML failed validation.\\n\", \"CSTTLINK\"=>\"http://cstt-prod.mitchell.com/Home/ErrorLogDetails?error_log_id=-1\"}, \"Mitch_prog\"], \"Mitchell_time\"=>[{\"message\"=>\"<11>pjbc11n4lxv jboss_jbc11: node4: [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %%-1%%55005%%NONFATAL%%pjbc11n4lxv.mitchell.com%%CORESERVICES%%ERRORLOG_SERVICE%%Nov 18, 2015 5:42:02 AM%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%[jboss_jbc11] XML failed validation.\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2015-11-18T13:42:02.776Z\", \"type\"=>\"syslog\", \"host\"=>\"172.22.1.68\", \"tags\"=>[\"_grokparsefailure_sysloginput\", \"Mitchell_Jboss_Log\", \"_grokparsefailure\"], \"priority\"=>0, \"severity\"=>0, \"facility\"=>0, \"facility_label\"=>\"kernel\", \"severity_label\"=>\"Emergency\", \"syslog5424_pri\"=>\"11\", \"Mitch_host\"=>\"pjbc11n4lxv jboss_jbc11: node4:\", \"syslog_message\"=>\" [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %\", \"CSTTError\"=>\"-1\", \"Mitch_ErrorNum\"=>\"55005\", \"Mitch_Priority\"=>\"NONFATAL\", \"Mitch_org_host\"=>\"pjbc11n4lxv.mitchell.com\", \"Mitch_prog\"=>\"%%CORESERVICES%%ERRORLOG_SERVICE%%\", \"Mitchell_time\"=>\"Nov 18, 2015 5:42:02 AM\", \"Mitch_Message\"=>\"%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%\", \"Mitch_Message1\"=>\"[jboss_jbc11] XML failed validation.\\n\", \"CSTTLINK\"=>\"http://cstt-prod.mitchell.com/Home/ErrorLogDetails?error_log_id=-1\"}, \"Mitchell_time\"], \"Mitch_Message\"=>[{\"message\"=>\"<11>pjbc11n4lxv jboss_jbc11: node4: [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %%-1%%55005%%NONFATAL%%pjbc11n4lxv.mitchell.com%%CORESERVICES%%ERRORLOG_SERVICE%%Nov 18, 2015 5:42:02 AM%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%[jboss_jbc11] XML failed validation.\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2015-11-18T13:42:02.776Z\", \"type\"=>\"syslog\", \"host\"=>\"172.22.1.68\", \"tags\"=>[\"_grokparsefailure_sysloginput\", \"Mitchell_Jboss_Log\", \"_grokparsefailure\"], \"priority\"=>0, \"severity\"=>0, \"facility\"=>0, \"facility_label\"=>\"kernel\", \"severity_label\"=>\"Emergency\", \"syslog5424_pri\"=>\"11\", \"Mitch_host\"=>\"pjbc11n4lxv jboss_jbc11: node4:\", \"syslog_message\"=>\" [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %\", \"CSTTError\"=>\"-1\", \"Mitch_ErrorNum\"=>\"55005\", \"Mitch_Priority\"=>\"NONFATAL\", \"Mitch_org_host\"=>\"pjbc11n4lxv.mitchell.com\", \"Mitch_prog\"=>\"%%CORESERVICES%%ERRORLOG_SERVICE%%\", \"Mitchell_time\"=>\"Nov 18, 2015 5:42:02 AM\", \"Mitch_Message\"=>\"%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%\", \"Mitch_Message1\"=>\"[jboss_jbc11] XML failed validation.\\n\", \"CSTTLINK\"=>\"http://cstt-prod.mitchell.com/Home/ErrorLogDetails?error_log_id=-1\"}, \"Mitch_Message\"], \"Mitch_Message1\"=>[{\"message\"=>\"<11>pjbc11n4lxv jboss_jbc11: node4: [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %%-1%%55005%%NONFATAL%%pjbc11n4lxv.mitchell.com%%CORESERVICES%%ERRORLOG_SERVICE%%Nov 18, 2015 5:42:02 AM%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%[jboss_jbc11] XML failed validation.\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2015-11-18T13:42:02.776Z\", \"type\"=>\"syslog\", \"host\"=>\"172.22.1.68\", \"tags\"=>[\"_grokparsefailure_sysloginput\", \"Mitchell_Jboss_Log\", \"_grokparsefailure\"], \"priority\"=>0, \"severity\"=>0, \"facility\"=>0, \"facility_label\"=>\"kernel\", \"severity_label\"=>\"Emergency\", \"syslog5424_pri\"=>\"11\", \"Mitch_host\"=>\"pjbc11n4lxv jboss_jbc11: node4:\", \"syslog_message\"=>\" [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %\", \"CSTTError\"=>\"-1\", \"Mitch_ErrorNum\"=>\"55005\", \"Mitch_Priority\"=>\"NONFATAL\", \"Mitch_org_host\"=>\"pjbc11n4lxv.mitchell.com\", \"Mitch_prog\"=>\"%%CORESERVICES%%ERRORLOG_SERVICE%%\", \"Mitchell_time\"=>\"Nov 18, 2015 5:42:02 AM\", \"Mitch_Message\"=>\"%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%\", \"Mitch_Message1\"=>\"[jboss_jbc11] XML failed validation.\\n\", \"CSTTLINK\"=>\"http://cstt-prod.mitchell.com/Home/ErrorLogDetails?error_log_id=-1\"}, \"Mitch_Message1\"], \"CSTTLINK\"=>[{\"message\"=>\"<11>pjbc11n4lxv jboss_jbc11: node4: [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %%-1%%55005%%NONFATAL%%pjbc11n4lxv.mitchell.com%%CORESERVICES%%ERRORLOG_SERVICE%%Nov 18, 2015 5:42:02 AM%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%[jboss_jbc11] XML failed validation.\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2015-11-18T13:42:02.776Z\", \"type\"=>\"syslog\", \"host\"=>\"172.22.1.68\", \"tags\"=>[\"_grokparsefailure_sysloginput\", \"Mitchell_Jboss_Log\", \"_grokparsefailure\"], \"priority\"=>0, \"severity\"=>0, \"facility\"=>0, \"facility_label\"=>\"kernel\", \"severity_label\"=>\"Emergency\", \"syslog5424_pri\"=>\"11\", \"Mitch_host\"=>\"pjbc11n4lxv jboss_jbc11: node4:\", \"syslog_message\"=>\" [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %\", \"CSTTError\"=>\"-1\", \"Mitch_ErrorNum\"=>\"55005\", \"Mitch_Priority\"=>\"NONFATAL\", \"Mitch_org_host\"=>\"pjbc11n4lxv.mitchell.com\", \"Mitch_prog\"=>\"%%CORESERVICES%%ERRORLOG_SERVICE%%\", \"Mitchell_time\"=>\"Nov 18, 2015 5:42:02 AM\", \"Mitch_Message\"=>\"%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%\", \"Mitch_Message1\"=>\"[jboss_jbc11] XML failed validation.\\n\", \"CSTTLINK\"=>\"http://cstt-prod.mitchell.com/Home/ErrorLogDetails?error_log_id=-1\"}, \"CSTTLINK\"]}>, @data={\"message\"=>\"<11>pjbc11n4lxv jboss_jbc11: node4: [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %%-1%%55005%%NONFATAL%%pjbc11n4lxv.mitchell.com%%CORESERVICES%%ERRORLOG_SERVICE%%Nov 18, 2015 5:42:02 AM%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%[jboss_jbc11] XML failed validation.\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2015-11-18T13:42:02.776Z\", \"type\"=>\"syslog\", \"host\"=>\"172.22.1.68\", \"tags\"=>[\"_grokparsefailure_sysloginput\", \"Mitchell_Jboss_Log\", \"_grokparsefailure\"], \"priority\"=>0, \"severity\"=>0, \"facility\"=>0, \"facility_label\"=>\"kernel\", \"severity_label\"=>\"Emergency\", \"syslog5424_pri\"=>\"11\", \"Mitch_host\"=>\"pjbc11n4lxv jboss_jbc11: node4:\", \"syslog_message\"=>\" [ID 4628 Nov 18, 2015 5:42:02 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %\", \"CSTTError\"=>\"-1\", \"Mitch_ErrorNum\"=>\"55005\", \"Mitch_Priority\"=>\"NONFATAL\", \"Mitch_org_host\"=>\"pjbc11n4lxv.mitchell.com\", \"Mitch_prog\"=>\"%%CORESERVICES%%ERRORLOG_SERVICE%%\", \"Mitchell_time\"=>\"Nov 18, 2015 5:42:02 AM\", \"Mitch_Message\"=>\"%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%\", \"Mitch_Message1\"=>\"[jboss_jbc11] XML failed validation.\\n\", \"CSTTLINK\"=>\"http://cstt-prod.mitchell.com/Home/ErrorLogDetails?error_log_id=-1\"}, @metadata_accessors=#<LogStash::Util::Accessors:0x221154b1 @store={\"retry_count\"=>0}, @lut={}>, @cancelled=false>]", :level=>:warn}

Re: Logstash failing to include 1 message type in index

Posted: Wed Nov 18, 2015 5:24 pm
by Jklre
Looks like I found whats causing this failure. Looks like elastic search doe snot like a specific date format we are using. GROK parses it but elasticsearch does not like it and chokes. the date format it doe snot like is "Nov 18, 2015 3:52:56 AM" I imagine its choking on the AM - PM portion. Possible solution is to mutate the date into a format elastic search will accept. Or take the error ID and replace it.

Code: Select all

2015-11-18 03:52:56,080][DEBUG][action.bulk              ] [41a07432-8d31-4259-a3d5-9ba9c0379bad] [logstash-2015.11.18][0] failed to execute bulk item (index) index {[logstash-2015.11.18][syslog][AVEacMIPkqP4pGFSpxHx], source[{"message":"<11>pjbc11n2lxv jboss_jbc11: node2: [ID 1953 Nov 18, 2015 3:52:56 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %%-1%%55005%%NONFATAL%%pjbc11n2lxv.mitchell.com%%CORESERVICES%%ERRORLOG_SERVICE%%Nov 18, 2015 3:52:56 AM%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%[jboss_jbc11] Error Instering 'Internal' Error: Exception creating BO: ORA-06502: PL/SQL: numeric or value error: character string buffer too small-ORA-06512: at \"ERR.ERR_LOG_PKG\", line 482-ORA-12899: value too large for column \"ERR\".\"ERR_LOG\".\"CORR_ID\" (actual: 58, maximum: 50)-ORA-06512: at line 1- (java.sql.SQLException: ORA-06502: PL/SQL: numeric or value error: character string buffer too small-ORA-06512: at \"ERR.ERR_LOG_PKG\", line 482-ORA-12899: value too large for column \"ERR\".\"ERR_LOG\".\"CORR_ID\" (actual: 58, maximum: 50)-ORA-06512: at line 1-); Origianl Error Description: XML failed validation.\n","@version":"1","@timestamp":"2015-11-18T11:52:56.073Z","type":"syslog","host":"172.22.1.68","tags":["_grokparsefailure_sysloginput","Mitchell_Jboss_Log","_grokparsefailure"],"priority":0,"severity":0,"facility":0,"facility_label":"kernel","severity_label":"Emergency","syslog5424_pri":"11","Mitch_host":"pjbc11n2lxv jboss_jbc11: node2:","syslog_message":" [ID 1953 Nov 18, 2015 3:52:56 AM com.mitchell.services.core.errorlog.server.SyslogWriter write#012SEVERE: %","CSTTError":"-1","Mitch_ErrorNum":"55005","Mitch_Priority":"NONFATAL","Mitch_org_host":"pjbc11n2lxv.mitchell.com","Mitch_prog":"%%CORESERVICES%%ERRORLOG_SERVICE%%","Mitchell_time":"Nov 18, 2015 3:52:56 AM","Mitch_Message":"%%com.mitchell.services.core.errorlog.server.ErrorLogMessageHandler%%handleTextMessage%%%%%%%%","Mitch_Message1":"[jboss_jbc11] Error Instering 'Internal' Error: Exception creating BO: ORA-06502: PL/SQL: numeric or value error: character string buffer too small-ORA-06512: at \"ERR.ERR_LOG_PKG\", line 482-ORA-12899: value too large for column \"ERR\".\"ERR_LOG\".\"CORR_ID\" (actual: 58, maximum: 50)-ORA-06512: at line 1- (java.sql.SQLException: ORA-06502: PL/SQL: numeric or value error: character string buffer too small-ORA-06512: at \"ERR.ERR_LOG_PKG\", line 482-ORA-12899: value too large for column \"ERR\".\"ERR_LOG\".\"CORR_ID\" (actual: 58, maximum: 50)-ORA-06512: at line 1-); Origianl Error Description: XML failed validation.\n","CSTTLINK":"http://cstt-prod.mitchell.com/Home/ErrorLogDetails?error_log_id=-1"}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [Mitchell_time]
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:411)
	at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:706)
	at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:497)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:544)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:493)
	at org.elasticsearch.index.shard.IndexShard.prepareCreate(IndexShard.java:466)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:418)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:148)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase.performOnPrimary(TransportShardReplicationOperationAction.java:574)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase$1.doRun(TransportShardReplicationOperationAction.java:440)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.mapper.MapperParsingException: failed to parse date field [Nov 18, 2015 3:52:56 AM], tried both date format [dateOptionalTime], and timestamp number with locale []
	at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:617)
	at org.elasticsearch.index.mapper.core.DateFieldMapper.innerParseCreateField(DateFieldMapper.java:535)
	at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:239)
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:401)
	... 13 more
Caused by: java.lang.IllegalArgumentException: Invalid format: "Nov 18, 2015 3:52:56 AM"
	at org.elasticsearch.common.joda.time.format.DateTimeParserBucket.doParseMillis(DateTimeParserBucket.java:187)
	at org.elasticsearch.common.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:780)
	at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:612)
	... 16 more

Re: Logstash failing to include 1 message type in index

Posted: Wed Nov 18, 2015 6:05 pm
by jolson
You can use the 'date' filter to explicitly match the date you're talking about. It should accept joda time format.

https://www.elastic.co/guide/en/logstas ... -date.html
http://joda-time.sourceforge.net/apidoc ... ormat.html

"Nov 18, 2015 3:52:56 AM" might be matched by something like:

Code: Select all

date {
        match => [ "timefield", "MMM dd, YYYY hh:mm:ss aa" ]
    }

Re: Logstash failing to include 1 message type in index

Posted: Thu Nov 19, 2015 5:21 pm
by Jklre
jolson wrote:You can use the 'date' filter to explicitly match the date you're talking about. It should accept joda time format.

https://www.elastic.co/guide/en/logstas ... -date.html
http://joda-time.sourceforge.net/apidoc ... ormat.html
"Nov 18, 2015 3:52:56 AM"

"Nov 18, 2015 3:52:56 AM" might be matched by something like:

Code: Select all

date {
        match => [ "timefield", "MMM dd, YYYY hh:mm:ss aa" ]
    }

Thanks for the info.

I ended up using this to resolve this issue. I had to have one with 1 h and another with hh since the hours send from this were only a single digit example 1:00 instead of 01:00

Code: Select all

    
date {
            match => [ "Mitchell_time",
            "MMM dd, YYYY h:mm:ss aa",
            "MMM dd, YYYY hh:mm:ss aa",
            "ISO8601" ]
            target => "Mitchell_time"
        }
Thank you.

Re: Logstash failing to include 1 message type in index

Posted: Thu Nov 19, 2015 5:43 pm
by jolson
No problem - I'll lock this up. Glad you got it working. :)
All times are UTC-05:00
Page 1 of 1

Powered by phpBB® Forum Software © phpBB Limited