Nagios Support Forum

Support for Nagios products and services
https://support.nagios.com/forum/

failed action with response of 400, dropping action

https://support.nagios.com/forum/viewtopic.php?t=36792

Page 1 of 1

failed action with response of 400, dropping action

Posted: Thu Jan 28, 2016 12:58 pm
by CFT6Server
I am seeing quiet a bit of these in our logstash log. Looks like this is coming from the local log server host.... Any ideas?

Code: Select all

{:timestamp=>"2016-01-28T09:56:02.408000-0800", :message=>"failed action with response of 400, dropping action: [\"index\", {:_id=>nil, :_index=>\"logstash-2016.01.28\", :_type=>\"syslog\", :_routing=>nil}, #<LogStash::Event:0x7e4cbf91 @metadata={\"retry_count\"=>0}, @accessors=#<LogStash::Util::Accessors:0xd7a2bec @store={\"message\"=>\"  nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-01-28T17:56:01.000Z\", \"type\"=>\"syslog\", \"host\"=>\"0:0:0:0:0:0:0:1\", \"priority\"=>85, \"timestamp\"=>\"Jan 28 09:56:01\", \"logsource\"=>\"fptnaglsp1\", \"program\"=>\"sudo\", \"severity\"=>5, \"facility\"=>10, \"facility_label\"=>\"security/authorization\", \"severity_label\"=>\"Notice\"}, @lut={\"type\"=>[{\"message\"=>\"  nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-01-28T17:56:01.000Z\", \"type\"=>\"syslog\", \"host\"=>\"0:0:0:0:0:0:0:1\", \"priority\"=>85, \"timestamp\"=>\"Jan 28 09:56:01\", \"logsource\"=>\"fptnaglsp1\", \"program\"=>\"sudo\", \"severity\"=>5, \"facility\"=>10, \"facility_label\"=>\"security/authorization\", \"severity_label\"=>\"Notice\"}, \"type\"], \"host\"=>[{\"message\"=>\"  nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-01-28T17:56:01.000Z\", \"type\"=>\"syslog\", \"host\"=>\"0:0:0:0:0:0:0:1\", \"priority\"=>85, \"timestamp\"=>\"Jan 28 09:56:01\", \"logsource\"=>\"fptnaglsp1\", \"program\"=>\"sudo\", \"severity\"=>5, \"facility\"=>10, \"facility_label\"=>\"security/authorization\", \"severity_label\"=>\"Notice\"}, \"host\"], \"message\"=>[{\"message\"=>\"  nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-01-28T17:56:01.000Z\", \"type\"=>\"syslog\", \"host\"=>\"0:0:0:0:0:0:0:1\", \"priority\"=>85, \"timestamp\"=>\"Jan 28 09:56:01\", \"logsource\"=>\"fptnaglsp1\", \"program\"=>\"sudo\", \"severity\"=>5, \"facility\"=>10, \"facility_label\"=>\"security/authorization\", \"severity_label\"=>\"Notice\"}, \"message\"], \"priority\"=>[{\"message\"=>\"  nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-01-28T17:56:01.000Z\", \"type\"=>\"syslog\", \"host\"=>\"0:0:0:0:0:0:0:1\", \"priority\"=>85, \"timestamp\"=>\"Jan 28 09:56:01\", \"logsource\"=>\"fptnaglsp1\", \"program\"=>\"sudo\", \"severity\"=>5, \"facility\"=>10, \"facility_label\"=>\"security/authorization\", \"severity_label\"=>\"Notice\"}, \"priority\"], \"timestamp\"=>[{\"message\"=>\"  nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-01-28T17:56:01.000Z\", \"type\"=>\"syslog\", \"host\"=>\"0:0:0:0:0:0:0:1\", \"priority\"=>85, \"timestamp\"=>\"Jan 28 09:56:01\", \"logsource\"=>\"fptnaglsp1\", \"program\"=>\"sudo\", \"severity\"=>5, \"facility\"=>10, \"facility_label\"=>\"security/authorization\", \"severity_label\"=>\"Notice\"}, \"timestamp\"], \"logsource\"=>[{\"message\"=>\"  nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-01-28T17:56:01.000Z\", \"type\"=>\"syslog\", \"host\"=>\"0:0:0:0:0:0:0:1\", \"priority\"=>85, \"timestamp\"=>\"Jan 28 09:56:01\", \"logsource\"=>\"fptnaglsp1\", \"program\"=>\"sudo\", \"severity\"=>5, \"facility\"=>10, \"facility_label\"=>\"security/authorization\", \"severity_label\"=>\"Notice\"}, \"logsource\"], \"program\"=>[{\"message\"=>\"  nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-01-28T17:56:01.000Z\", \"type\"=>\"syslog\", \"host\"=>\"0:0:0:0:0:0:0:1\", \"priority\"=>85, \"timestamp\"=>\"Jan 28 09:56:01\", \"logsource\"=>\"fptnaglsp1\", \"program\"=>\"sudo\", \"severity\"=>5, \"facility\"=>10, \"facility_label\"=>\"security/authorization\", \"severity_label\"=>\"Notice\"}, \"program\"], \"tags\"=>[{\"message\"=>\"  nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-01-28T17:56:01.000Z\", \"type\"=>\"syslog\", \"host\"=>\"0:0:0:0:0:0:0:1\", \"priority\"=>85, \"timestamp\"=>\"Jan 28 09:56:01\", \"logsource\"=>\"fptnaglsp1\", \"program\"=>\"sudo\", \"severity\"=>5, \"facility\"=>10, \"facility_label\"=>\"security/authorization\", \"severity_label\"=>\"Notice\"}, \"tags\"], \"severity\"=>[{\"message\"=>\"  nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-01-28T17:56:01.000Z\", \"type\"=>\"syslog\", \"host\"=>\"0:0:0:0:0:0:0:1\", \"priority\"=>85, \"timestamp\"=>\"Jan 28 09:56:01\", \"logsource\"=>\"fptnaglsp1\", \"program\"=>\"sudo\", \"severity\"=>5, \"facility\"=>10, \"facility_label\"=>\"security/authorization\", \"severity_label\"=>\"Notice\"}, \"severity\"], \"facility\"=>[{\"message\"=>\"  nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-01-28T17:56:01.000Z\", \"type\"=>\"syslog\", \"host\"=>\"0:0:0:0:0:0:0:1\", \"priority\"=>85, \"timestamp\"=>\"Jan 28 09:56:01\", \"logsource\"=>\"fptnaglsp1\", \"program\"=>\"sudo\", \"severity\"=>5, \"facility\"=>10, \"facility_label\"=>\"security/authorization\", \"severity_label\"=>\"Notice\"}, \"facility\"], \"timestamp8601\"=>[{\"message\"=>\"  nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-01-28T17:56:01.000Z\", \"type\"=>\"syslog\", \"host\"=>\"0:0:0:0:0:0:0:1\", \"priority\"=>85, \"timestamp\"=>\"Jan 28 09:56:01\", \"logsource\"=>\"fptnaglsp1\", \"program\"=>\"sudo\", \"severity\"=>5, \"facility\"=>10, \"facility_label\"=>\"security/authorization\", \"severity_label\"=>\"Notice\"}, \"timestamp8601\"], \"@timestamp\"=>[{\"message\"=>\"  nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-01-28T17:56:01.000Z\", \"type\"=>\"syslog\", \"host\"=>\"0:0:0:0:0:0:0:1\", \"priority\"=>85, \"timestamp\"=>\"Jan 28 09:56:01\", \"logsource\"=>\"fptnaglsp1\", \"program\"=>\"sudo\", \"severity\"=>5, \"facility\"=>10, \"facility_label\"=>\"security/authorization\", \"severity_label\"=>\"Notice\"}, \"@timestamp\"], \"facility_label\"=>[{\"message\"=>\"  nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-01-28T17:56:01.000Z\", \"type\"=>\"syslog\", \"host\"=>\"0:0:0:0:0:0:0:1\", \"priority\"=>85, \"timestamp\"=>\"Jan 28 09:56:01\", \"logsource\"=>\"fptnaglsp1\", \"program\"=>\"sudo\", \"severity\"=>5, \"facility\"=>10, \"facility_label\"=>\"security/authorization\", \"severity_label\"=>\"Notice\"}, \"facility_label\"], \"severity_label\"=>[{\"message\"=>\"  nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-01-28T17:56:01.000Z\", \"type\"=>\"syslog\", \"host\"=>\"0:0:0:0:0:0:0:1\", \"priority\"=>85, \"timestamp\"=>\"Jan 28 09:56:01\", \"logsource\"=>\"fptnaglsp1\", \"program\"=>\"sudo\", \"severity\"=>5, \"facility\"=>10, \"facility_label\"=>\"security/authorization\", \"severity_label\"=>\"Notice\"}, \"severity_label\"], \"[program]\"=>[{\"message\"=>\"  nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-01-28T17:56:01.000Z\", \"type\"=>\"syslog\", \"host\"=>\"0:0:0:0:0:0:0:1\", \"priority\"=>85, \"timestamp\"=>\"Jan 28 09:56:01\", \"logsource\"=>\"fptnaglsp1\", \"program\"=>\"sudo\", \"severity\"=>5, \"facility\"=>10, \"facility_label\"=>\"security/authorization\", \"severity_label\"=>\"Notice\"}, \"program\"], \"[host]\"=>[{\"message\"=>\"  nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-01-28T17:56:01.000Z\", \"type\"=>\"syslog\", \"host\"=>\"0:0:0:0:0:0:0:1\", \"priority\"=>85, \"timestamp\"=>\"Jan 28 09:56:01\", \"logsource\"=>\"fptnaglsp1\", \"program\"=>\"sudo\", \"severity\"=>5, \"facility\"=>10, \"facility_label\"=>\"security/authorization\", \"severity_label\"=>\"Notice\"}, \"host\"], \"[type]\"=>[{\"message\"=>\"  nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-01-28T17:56:01.000Z\", \"type\"=>\"syslog\", \"host\"=>\"0:0:0:0:0:0:0:1\", \"priority\"=>85, \"timestamp\"=>\"Jan 28 09:56:01\", \"logsource\"=>\"fptnaglsp1\", \"program\"=>\"sudo\", \"severity\"=>5, \"facility\"=>10, \"facility_label\"=>\"security/authorization\", \"severity_label\"=>\"Notice\"}, \"type\"], \"[prog]\"=>[{\"message\"=>\"  nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-01-28T17:56:01.000Z\", \"type\"=>\"syslog\", \"host\"=>\"0:0:0:0:0:0:0:1\", \"priority\"=>85, \"timestamp\"=>\"Jan 28 09:56:01\", \"logsource\"=>\"fptnaglsp1\", \"program\"=>\"sudo\", \"severity\"=>5, \"facility\"=>10, \"facility_label\"=>\"security/authorization\", \"severity_label\"=>\"Notice\"}, \"prog\"]}>, @data={\"message\"=>\"  nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status\\n\", \"@version\"=>\"1\", \"@timestamp\"=>\"2016-01-28T17:56:01.000Z\", \"type\"=>\"syslog\", \"host\"=>\"0:0:0:0:0:0:0:1\", \"priority\"=>85, \"timestamp\"=>\"Jan 28 09:56:01\", \"logsource\"=>\"fptnaglsp1\", \"program\"=>\"sudo\", \"severity\"=>5, \"facility\"=>10, \"facility_label\"=>\"security/authorization\", \"severity_label\"=>\"Notice\"}, @metadata_accessors=#<LogStash::Util::Accessors:0x759f0a7e @store={\"retry_count\"=>0}, @lut={}>, @cancelled=false>]", :level=>:warn}

Re: failed action with response of 400, dropping action

Posted: Thu Jan 28, 2016 5:30 pm
by hsmith
Can we see the output of the following commands?

Code: Select all

cat /etc/hosts
cat /usr/local/nagioslogserver/logstash/etc/conf.d/*
cat /etc/rsyslog.d/nagioslogserver.conf

Re: failed action with response of 400, dropping action

Posted: Fri Jan 29, 2016 1:09 pm
by CFT6Server

Code: Select all

127.0.0.1       <SERVER> localhost localhost.localdomain
::1             <SERVER> localhost localhost.localdomain

Code: Select all

# cat /usr/local/nagioslogserver/logstash/etc/conf.d/*
#
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Thu, 28 Jan 2016 09:55:17 -0800
#

#
# Global inputs
#

input {
    syslog {
        type => 'syslog'
        port => 5544
    }
    tcp {
        type => 'eventlog'
        port => 3515
        codec => json
    }
    tcp {
        type => 'import_raw'
        tags => 'import_raw'
        port => 2056
    }
    tcp {
        type => 'import_json'
        tags => 'import_json'
        port => 2057
        codec => json
    }
    syslog {
        type => 'syslog'
        port => 514
    }
}

#
# Local inputs
#


#
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Thu, 28 Jan 2016 09:55:17 -0800
#

#
# Global filters
#

filter {
    if [program] == 'apache_access' {
        grok {
            match => [ 'message', '%{COMBINEDAPACHELOG}']
        }
        date {
            match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z' ]
        }
        mutate {
            replace => [ 'type', 'apache_access' ]
             convert => [ 'bytes', 'integer' ]
             convert => [ 'response', 'integer' ]
        }
    }

    if [program] == 'apache_error' {
        grok {
            match => [ 'message', '\[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\] \[%{WORD:class}\] \[%{WORD:originator} %{IP:clientip}\] %{GREEDYDATA:errmsg}']
        }
        mutate {
            replace => [ 'type', 'apache_error' ]
        }
    }
    if ([host] == '10.62.11.74' or [host] == '10.62.11.78' or [host] == '10.62.11.202' or [host] == '10.62.11.195') {
        grok {
            match => [
                'message', '(?<esxi_thread_id>[0-9A-Z]{8,8}) %{DATA:esxi_loglevel} \'%{DATA:esxi_service}\' %{DATA}\] \[%{DATA}::%{DATA:esxi_sub_service}\] %{GREEDYDATA:esxi_message}',
                        'message', '(?<esxi_thread_id>[0-9A-Z]{8,8}) %{DATA:esxi_loglevel} \'%{DATA:esxi_service}\' %{DATA}\] \[%{DATA:esxi_sub_service}\] %{GREEDYDATA:esxi_message}',
                        'message', '(?<esxi_thread_id>[0-9A-Z]{8,8}) %{DATA:esxi_loglevel} \'%{DATA:esxi_service}\'\ (.*)] %{GREEDYDATA:esxi_message}',
                        'message', '(?<esxi_thread_id>[0-9A-Z]{8,8}) %{DATA:esxi_loglevel} \'%{DATA:esxi_service}\'] %{GREEDYDATA:esxi_message}',
                        'message', '%{GREEDYDATA:esxi_message}'
                        ]
                        add_tag => ['VMware']
                }
        mutate {
                replace => [ 'message', '%{esxi_message}']
                remove_field => ['esxi_message']
                }
        }
    if [type] == "eventlog" {
       grok {
          match => ["Hostname", "%{WORD:Hostname}"]
          overwrite => ["Hostname"]
        }
        date {
           match => ["EventTime", "YYYY-MM-dd HH:mm:ss"]
          }
    }
    if [type] == "dmz-eventlog" {
       grok {
          match => ["Hostname", "%{WORD:Hostname}"]
          overwrite => ["Hostname"]
          add_tag => ["Event Log"]
        }
    }

}

#
# Local filters
#


#
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Thu, 28 Jan 2016 09:55:17 -0800
#

#
# Required output for Nagios Log Server
#

output {
    elasticsearch {
        cluster => '406279af-a1e6-4c0b-8432-2eb3c337e012'
        host => 'localhost'
        document_type => '%{type}'
        node_name => 'd45faaf2-e8d3-45e8-b6cf-cdffd3c35495'
        protocol => 'transport'
        workers => 4
    }
}

#
# Global outputs
#



#
# Local outputs
#

Code: Select all

# cat /etc/rsyslog.d/nagioslogserver.conf
# ### begin forwarding rule ###
#
# NAGIOS LOG SERVER
#
$WorkDirectory /var/lib/rsyslog    # where to place spool files
$ActionQueueFileName fwdRule1      # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g        # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on      # save messages to disk on shutdown
$ActionQueueType LinkedList        # run asynchronously
$ActionResumeRetryCount -1         # infinite retries if host is down
*.* @@localhost:5544
#
# ### end of the forwarding rule ###[root@fptnaglsp1 ~]#

Re: failed action with response of 400, dropping action

Posted: Fri Jan 29, 2016 3:18 pm
by jolson
In your rsyslog configuration file, try changing this:

Code: Select all

*.* @@localhost:5544
To this:

Code: Select all

*.* @@localhost:2057
And restart rsyslog:

Code: Select all

service rsyslogd restart
Any luck with this procedure? I'm wondering if the syslog input isn't messing with things.

Re: failed action with response of 400, dropping action

Posted: Fri Jan 29, 2016 5:19 pm
by CFT6Server
I think this is working now. I will monitor the logs for next little while to see if it comes back.

Re: failed action with response of 400, dropping action

Posted: Mon Feb 01, 2016 10:39 am
by hsmith
Let us know! Glad it's looking better though :)
All times are UTC-05:00
Page 1 of 1

Powered by phpBB® Forum Software © phpBB Limited