failed action with response of 400, dropping action

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
Locked
CFT6Server
Posts: 506
Joined: Wed Apr 15, 2015 4:21 pm

failed action with response of 400, dropping action

Post by CFT6Server »

I am seeing quiet a bit of these in our logstash log. Looks like this is coming from the local log server host.... Any ideas?

Code: Select all

{:timestamp=>"2016-01-28T09:56:02.408000-0800", :message=>"failed action with response of 400, dropping action: [\"index\", {:_id=>nil, :_index=>\"logstash-2016.01.28\", :_type=>\"syslog\", :_routing=>nil}, #<LogStash::Event:0x7e4cbf91 @metadata={\"retry_count\"=>0}, @accessors=#<LogStash::Util::Accessors:0xd7a2bec @store={\"message\"=>\"  nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-01-28T17:56:01.000Z\", \"type\"=>\"syslog\", \"host\"=>\"0:0:0:0:0:0:0:1\", \"priority\"=>85, \"timestamp\"=>\"Jan 28 09:56:01\", \"logsource\"=>\"fptnaglsp1\", \"program\"=>\"sudo\", \"severity\"=>5, \"facility\"=>10, \"facility_label\"=>\"security/authorization\", \"severity_label\"=>\"Notice\"}, @lut={\"type\"=>[{\"message\"=>\"  nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-01-28T17:56:01.000Z\", \"type\"=>\"syslog\", \"host\"=>\"0:0:0:0:0:0:0:1\", \"priority\"=>85, \"timestamp\"=>\"Jan 28 09:56:01\", \"logsource\"=>\"fptnaglsp1\", \"program\"=>\"sudo\", \"severity\"=>5, \"facility\"=>10, \"facility_label\"=>\"security/authorization\", \"severity_label\"=>\"Notice\"}, \"type\"], \"host\"=>[{\"message\"=>\"  nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-01-28T17:56:01.000Z\", \"type\"=>\"syslog\", \"host\"=>\"0:0:0:0:0:0:0:1\", \"priority\"=>85, \"timestamp\"=>\"Jan 28 09:56:01\", \"logsource\"=>\"fptnaglsp1\", \"program\"=>\"sudo\", \"severity\"=>5, \"facility\"=>10, \"facility_label\"=>\"security/authorization\", \"severity_label\"=>\"Notice\"}, \"host\"], \"message\"=>[{\"message\"=>\"  nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-01-28T17:56:01.000Z\", \"type\"=>\"syslog\", \"host\"=>\"0:0:0:0:0:0:0:1\", \"priority\"=>85, \"timestamp\"=>\"Jan 28 09:56:01\", \"logsource\"=>\"fptnaglsp1\", \"program\"=>\"sudo\", \"severity\"=>5, \"facility\"=>10, \"facility_label\"=>\"security/authorization\", \"severity_label\"=>\"Notice\"}, \"message\"], \"priority\"=>[{\"message\"=>\"  nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-01-28T17:56:01.000Z\", \"type\"=>\"syslog\", \"host\"=>\"0:0:0:0:0:0:0:1\", \"priority\"=>85, \"timestamp\"=>\"Jan 28 09:56:01\", \"logsource\"=>\"fptnaglsp1\", \"program\"=>\"sudo\", \"severity\"=>5, \"facility\"=>10, \"facility_label\"=>\"security/authorization\", \"severity_label\"=>\"Notice\"}, \"priority\"], \"timestamp\"=>[{\"message\"=>\"  nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-01-28T17:56:01.000Z\", \"type\"=>\"syslog\", \"host\"=>\"0:0:0:0:0:0:0:1\", \"priority\"=>85, \"timestamp\"=>\"Jan 28 09:56:01\", \"logsource\"=>\"fptnaglsp1\", \"program\"=>\"sudo\", \"severity\"=>5, \"facility\"=>10, \"facility_label\"=>\"security/authorization\", \"severity_label\"=>\"Notice\"}, \"timestamp\"], \"logsource\"=>[{\"message\"=>\"  nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-01-28T17:56:01.000Z\", \"type\"=>\"syslog\", \"host\"=>\"0:0:0:0:0:0:0:1\", \"priority\"=>85, \"timestamp\"=>\"Jan 28 09:56:01\", \"logsource\"=>\"fptnaglsp1\", \"program\"=>\"sudo\", \"severity\"=>5, \"facility\"=>10, \"facility_label\"=>\"security/authorization\", \"severity_label\"=>\"Notice\"}, \"logsource\"], \"program\"=>[{\"message\"=>\"  nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-01-28T17:56:01.000Z\", \"type\"=>\"syslog\", \"host\"=>\"0:0:0:0:0:0:0:1\", \"priority\"=>85, \"timestamp\"=>\"Jan 28 09:56:01\", \"logsource\"=>\"fptnaglsp1\", \"program\"=>\"sudo\", \"severity\"=>5, \"facility\"=>10, \"facility_label\"=>\"security/authorization\", \"severity_label\"=>\"Notice\"}, \"program\"], \"tags\"=>[{\"message\"=>\"  nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-01-28T17:56:01.000Z\", \"type\"=>\"syslog\", \"host\"=>\"0:0:0:0:0:0:0:1\", \"priority\"=>85, \"timestamp\"=>\"Jan 28 09:56:01\", \"logsource\"=>\"fptnaglsp1\", \"program\"=>\"sudo\", \"severity\"=>5, \"facility\"=>10, \"facility_label\"=>\"security/authorization\", \"severity_label\"=>\"Notice\"}, \"tags\"], \"severity\"=>[{\"message\"=>\"  nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-01-28T17:56:01.000Z\", \"type\"=>\"syslog\", \"host\"=>\"0:0:0:0:0:0:0:1\", \"priority\"=>85, \"timestamp\"=>\"Jan 28 09:56:01\", \"logsource\"=>\"fptnaglsp1\", \"program\"=>\"sudo\", \"severity\"=>5, \"facility\"=>10, \"facility_label\"=>\"security/authorization\", \"severity_label\"=>\"Notice\"}, \"severity\"], \"facility\"=>[{\"message\"=>\"  nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-01-28T17:56:01.000Z\", \"type\"=>\"syslog\", \"host\"=>\"0:0:0:0:0:0:0:1\", \"priority\"=>85, \"timestamp\"=>\"Jan 28 09:56:01\", \"logsource\"=>\"fptnaglsp1\", \"program\"=>\"sudo\", \"severity\"=>5, \"facility\"=>10, \"facility_label\"=>\"security/authorization\", \"severity_label\"=>\"Notice\"}, \"facility\"], \"timestamp8601\"=>[{\"message\"=>\"  nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-01-28T17:56:01.000Z\", \"type\"=>\"syslog\", \"host\"=>\"0:0:0:0:0:0:0:1\", \"priority\"=>85, \"timestamp\"=>\"Jan 28 09:56:01\", \"logsource\"=>\"fptnaglsp1\", \"program\"=>\"sudo\", \"severity\"=>5, \"facility\"=>10, \"facility_label\"=>\"security/authorization\", \"severity_label\"=>\"Notice\"}, \"timestamp8601\"], \"@timestamp\"=>[{\"message\"=>\"  nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-01-28T17:56:01.000Z\", \"type\"=>\"syslog\", \"host\"=>\"0:0:0:0:0:0:0:1\", \"priority\"=>85, \"timestamp\"=>\"Jan 28 09:56:01\", \"logsource\"=>\"fptnaglsp1\", \"program\"=>\"sudo\", \"severity\"=>5, \"facility\"=>10, \"facility_label\"=>\"security/authorization\", \"severity_label\"=>\"Notice\"}, \"@timestamp\"], \"facility_label\"=>[{\"message\"=>\"  nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-01-28T17:56:01.000Z\", \"type\"=>\"syslog\", \"host\"=>\"0:0:0:0:0:0:0:1\", \"priority\"=>85, \"timestamp\"=>\"Jan 28 09:56:01\", \"logsource\"=>\"fptnaglsp1\", \"program\"=>\"sudo\", \"severity\"=>5, \"facility\"=>10, \"facility_label\"=>\"security/authorization\", \"severity_label\"=>\"Notice\"}, \"facility_label\"], \"severity_label\"=>[{\"message\"=>\"  nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-01-28T17:56:01.000Z\", \"type\"=>\"syslog\", \"host\"=>\"0:0:0:0:0:0:0:1\", \"priority\"=>85, \"timestamp\"=>\"Jan 28 09:56:01\", \"logsource\"=>\"fptnaglsp1\", \"program\"=>\"sudo\", \"severity\"=>5, \"facility\"=>10, \"facility_label\"=>\"security/authorization\", \"severity_label\"=>\"Notice\"}, \"severity_label\"], \"[program]\"=>[{\"message\"=>\"  nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-01-28T17:56:01.000Z\", \"type\"=>\"syslog\", \"host\"=>\"0:0:0:0:0:0:0:1\", \"priority\"=>85, \"timestamp\"=>\"Jan 28 09:56:01\", \"logsource\"=>\"fptnaglsp1\", \"program\"=>\"sudo\", \"severity\"=>5, \"facility\"=>10, \"facility_label\"=>\"security/authorization\", \"severity_label\"=>\"Notice\"}, \"program\"], \"[host]\"=>[{\"message\"=>\"  nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-01-28T17:56:01.000Z\", \"type\"=>\"syslog\", \"host\"=>\"0:0:0:0:0:0:0:1\", \"priority\"=>85, \"timestamp\"=>\"Jan 28 09:56:01\", \"logsource\"=>\"fptnaglsp1\", \"program\"=>\"sudo\", \"severity\"=>5, \"facility\"=>10, \"facility_label\"=>\"security/authorization\", \"severity_label\"=>\"Notice\"}, \"host\"], \"[type]\"=>[{\"message\"=>\"  nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-01-28T17:56:01.000Z\", \"type\"=>\"syslog\", \"host\"=>\"0:0:0:0:0:0:0:1\", \"priority\"=>85, \"timestamp\"=>\"Jan 28 09:56:01\", \"logsource\"=>\"fptnaglsp1\", \"program\"=>\"sudo\", \"severity\"=>5, \"facility\"=>10, \"facility_label\"=>\"security/authorization\", \"severity_label\"=>\"Notice\"}, \"type\"], \"[prog]\"=>[{\"message\"=>\"  nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-01-28T17:56:01.000Z\", \"type\"=>\"syslog\", \"host\"=>\"0:0:0:0:0:0:0:1\", \"priority\"=>85, \"timestamp\"=>\"Jan 28 09:56:01\", \"logsource\"=>\"fptnaglsp1\", \"program\"=>\"sudo\", \"severity\"=>5, \"facility\"=>10, \"facility_label\"=>\"security/authorization\", \"severity_label\"=>\"Notice\"}, \"prog\"]}>, @data={\"message\"=>\"  nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-01-28T17:56:01.000Z\", \"type\"=>\"syslog\", \"host\"=>\"0:0:0:0:0:0:0:1\", \"priority\"=>85, \"timestamp\"=>\"Jan 28 09:56:01\", \"logsource\"=>\"fptnaglsp1\", \"program\"=>\"sudo\", \"severity\"=>5, \"facility\"=>10, \"facility_label\"=>\"security/authorization\", \"severity_label\"=>\"Notice\"}, @metadata_accessors=#<LogStash::Util::Accessors:0x759f0a7e @store={\"retry_count\"=>0}, @lut={}>, @cancelled=false>]", :level=>:warn}
Top
User avatar
hsmith
Agent Smith
Posts: 3539
Joined: Thu Jul 30, 2015 11:09 am
Location: 127.0.0.1
Contact:
Contact hsmith

Re: failed action with response of 400, dropping action

Post by hsmith »

Can we see the output of the following commands?

Code: Select all

cat /etc/hosts
cat /usr/local/nagioslogserver/logstash/etc/conf.d/*
cat /etc/rsyslog.d/nagioslogserver.conf
Former Nagios Employee.
me.
Top
CFT6Server
Posts: 506
Joined: Wed Apr 15, 2015 4:21 pm

Re: failed action with response of 400, dropping action

Post by CFT6Server »

Code: Select all

127.0.0.1       <SERVER> localhost localhost.localdomain
::1             <SERVER> localhost localhost.localdomain

Code: Select all

# cat /usr/local/nagioslogserver/logstash/etc/conf.d/*
#
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Thu, 28 Jan 2016 09:55:17 -0800
#

#
# Global inputs
#

input {
    syslog {
        type => 'syslog'
        port => 5544
    }
    tcp {
        type => 'eventlog'
        port => 3515
        codec => json
    }
    tcp {
        type => 'import_raw'
        tags => 'import_raw'
        port => 2056
    }
    tcp {
        type => 'import_json'
        tags => 'import_json'
        port => 2057
        codec => json
    }
    syslog {
        type => 'syslog'
        port => 514
    }
}

#
# Local inputs
#


#
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Thu, 28 Jan 2016 09:55:17 -0800
#

#
# Global filters
#

filter {
    if [program] == 'apache_access' {
        grok {
            match => [ 'message', '%{COMBINEDAPACHELOG}']
        }
        date {
            match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z' ]
        }
        mutate {
            replace => [ 'type', 'apache_access' ]
             convert => [ 'bytes', 'integer' ]
             convert => [ 'response', 'integer' ]
        }
    }

    if [program] == 'apache_error' {
        grok {
            match => [ 'message', '\[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\] \[%{WORD:class}\] \[%{WORD:originator} %{IP:clientip}\] %{GREEDYDATA:errmsg}']
        }
        mutate {
            replace => [ 'type', 'apache_error' ]
        }
    }
    if ([host] == '10.62.11.74' or [host] == '10.62.11.78' or [host] == '10.62.11.202' or [host] == '10.62.11.195') {
        grok {
            match => [
                'message', '(?<esxi_thread_id>[0-9A-Z]{8,8}) %{DATA:esxi_loglevel} \'%{DATA:esxi_service}\' %{DATA}\] \[%{DATA}::%{DATA:esxi_sub_service}\] %{GREEDYDATA:esxi_message}',
                        'message', '(?<esxi_thread_id>[0-9A-Z]{8,8}) %{DATA:esxi_loglevel} \'%{DATA:esxi_service}\' %{DATA}\] \[%{DATA:esxi_sub_service}\] %{GREEDYDATA:esxi_message}',
                        'message', '(?<esxi_thread_id>[0-9A-Z]{8,8}) %{DATA:esxi_loglevel} \'%{DATA:esxi_service}\'\ (.*)] %{GREEDYDATA:esxi_message}',
                        'message', '(?<esxi_thread_id>[0-9A-Z]{8,8}) %{DATA:esxi_loglevel} \'%{DATA:esxi_service}\'] %{GREEDYDATA:esxi_message}',
                        'message', '%{GREEDYDATA:esxi_message}'
                        ]
                        add_tag => ['VMware']
                }
        mutate {
                replace => [ 'message', '%{esxi_message}']
                remove_field => ['esxi_message']
                }
        }
    if [type] == "eventlog" {
       grok {
          match => ["Hostname", "%{WORD:Hostname}"]
          overwrite => ["Hostname"]
        }
        date {
           match => ["EventTime", "YYYY-MM-dd HH:mm:ss"]
          }
    }
    if [type] == "dmz-eventlog" {
       grok {
          match => ["Hostname", "%{WORD:Hostname}"]
          overwrite => ["Hostname"]
          add_tag => ["Event Log"]
        }
    }

}

#
# Local filters
#


#
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Thu, 28 Jan 2016 09:55:17 -0800
#

#
# Required output for Nagios Log Server
#

output {
    elasticsearch {
        cluster => '406279af-a1e6-4c0b-8432-2eb3c337e012'
        host => 'localhost'
        document_type => '%{type}'
        node_name => 'd45faaf2-e8d3-45e8-b6cf-cdffd3c35495'
        protocol => 'transport'
        workers => 4
    }
}

#
# Global outputs
#



#
# Local outputs
#

Code: Select all

# cat /etc/rsyslog.d/nagioslogserver.conf
# ### begin forwarding rule ###
#
# NAGIOS LOG SERVER
#
$WorkDirectory /var/lib/rsyslog    # where to place spool files
$ActionQueueFileName fwdRule1      # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g        # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on      # save messages to disk on shutdown
$ActionQueueType LinkedList        # run asynchronously
$ActionResumeRetryCount -1         # infinite retries if host is down
*.* @@localhost:5544
#
# ### end of the forwarding rule ###[root@fptnaglsp1 ~]#
Top
jolson
Attack Rabbit
Posts: 2560
Joined: Thu Feb 12, 2015 12:40 pm

Re: failed action with response of 400, dropping action

Post by jolson »

In your rsyslog configuration file, try changing this:

Code: Select all

*.* @@localhost:5544
To this:

Code: Select all

*.* @@localhost:2057
And restart rsyslog:

Code: Select all

service rsyslogd restart
Any luck with this procedure? I'm wondering if the syslog input isn't messing with things.
Twits Blog
Show me a man who lives alone and has a perpetually clean kitchen, and 8 times out of 9 I'll show you a man with detestable spiritual qualities.
Top
CFT6Server
Posts: 506
Joined: Wed Apr 15, 2015 4:21 pm

Re: failed action with response of 400, dropping action

Post by CFT6Server »

I think this is working now. I will monitor the logs for next little while to see if it comes back.
Top
User avatar
hsmith
Agent Smith
Posts: 3539
Joined: Thu Jul 30, 2015 11:09 am
Location: 127.0.0.1
Contact:
Contact hsmith

Re: failed action with response of 400, dropping action

Post by hsmith »

Let us know! Glad it's looking better though :)
Former Nagios Employee.
me.
Top
Locked

Return to “Nagios Log Server”