Better Apache Dashboard stopped working

GhostRider2110 · Post by **GhostRider2110** » Fri Jan 20, 2017 11:06 am

Sorry for the delay. Start of session had me having to focus on other tasks, but back to this.

avandemore wrote:Ghostrider,

What happens if you try the solution listed here:

https://support.nagios.com/forum/viewto ... 93#p192160

Do you mean increasing LS_OPEN_FILES? I'll do that and restart.

Maybe this will help. I have both Apache Dashboard and Better Apache Dashboard on the system. I have not looked at the "default" Apache Dashboard. Attaching are screen shots. Looks like same thing happening

See-ya
Mitch

GhostRider2110 · Post by **GhostRider2110** » Fri Jan 20, 2017 12:56 pm

I notice something ....

When I look at the "All Events" panel, there are 7 columns, but only 3 get filled in. There seems to not be a field for "logsource" "verb" "response" "request".
Even if I expand "all" types, those don't show up???

Post by **mcapra** » Fri Jan 20, 2017 1:14 pm

Can you expand one of those events (with the missing columns) in the GUI and share it's entire contents? We could be seeing a filter failure or something more sinister, but seeing what fields are/aren't set would be helpful.

Like so:

2017_01_20_12_14_37_Dashboard_Nagios_Log_Server.png

GhostRider2110 · Post by **GhostRider2110** » Fri Jan 20, 2017 1:23 pm

Is this what you are looking for?

GhostRider2110 · Post by **GhostRider2110** » Fri Jan 20, 2017 1:25 pm

Here is the Fields list..Current and All.

I don't see any of those columns that are not displaying any data, listed in the fields list?

Post by **mcapra** » Fri Jan 20, 2017 1:49 pm

What I would like to see is the expanded event itself being stored on the back-end if possible, not the detected fields. This might contain key information in the tags field, the message field, etc.

GhostRider2110 · Post by **GhostRider2110** » Fri Jan 20, 2017 2:04 pm

Please refresh my memory on how I would do that?

Post by **mcapra** » Fri Jan 20, 2017 2:07 pm

Ah, you did it in the first post and I just overlooked it

https://support.nagios.com/forum/downlo ... &mode=view

It looks as if the default apache filter set is not correctly being applied to this apache event. Can you share the output of the following command executed from the CLI of (any one of) your Nagios Log Server machine:

Code: Select all

grep '' /usr/local/nagioslogserver/logstash/etc/conf.d/*

GhostRider2110 · Post by **GhostRider2110** » Fri Jan 20, 2017 2:13 pm

Code: Select all

[root@IGAnagioslog bin]# cd /usr/local/nagioslogserver/logstash/etc/conf.d/
[root@IGAnagioslog conf.d]# grep '' *
000_inputs.conf:# 
000_inputs.conf:# Logstash Configuration File
000_inputs.conf:# Dynamically created by Nagios Log Server
000_inputs.conf:#
000_inputs.conf:# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
000_inputs.conf:#
000_inputs.conf:# Created Fri, 20 Jan 2017 14:10:38 -0500
000_inputs.conf:#
000_inputs.conf:
000_inputs.conf:#
000_inputs.conf:# Global inputs
000_inputs.conf:#
000_inputs.conf:
000_inputs.conf:input {
000_inputs.conf:    tcp {
000_inputs.conf:        type => 'eventlog'
000_inputs.conf:        port => 3515
000_inputs.conf:        codec => json {
000_inputs.conf:            charset => 'CP1252'
000_inputs.conf:        }
000_inputs.conf:    }
000_inputs.conf:    tcp {
000_inputs.conf:        type => 'import_raw'
000_inputs.conf:        tags => 'import_raw'
000_inputs.conf:        port => 2056
000_inputs.conf:    }
000_inputs.conf:    tcp {
000_inputs.conf:        type => 'import_json'
000_inputs.conf:        tags => 'import_json'
000_inputs.conf:        port => 2057
000_inputs.conf:        codec => json
000_inputs.conf:    }
000_inputs.conf:    syslog {
000_inputs.conf:        type => 'syslog'
000_inputs.conf:        port => 514
000_inputs.conf:    }
000_inputs.conf:    syslog {
000_inputs.conf:        type => 'asa'
000_inputs.conf:        port => 6544
000_inputs.conf:    }
000_inputs.conf:    tcp {
000_inputs.conf:        port => 5544
000_inputs.conf:        type => 'syslog'
000_inputs.conf:    }
000_inputs.conf:    udp {
000_inputs.conf:        port => 5544
000_inputs.conf:        type => 'syslog'
000_inputs.conf:    
000_inputs.conf:    }
000_inputs.conf:    tcp {
000_inputs.conf:        type => 'apache_access'
000_inputs.conf:        port => 5581
000_inputs.conf:    }
000_inputs.conf:    udp {
000_inputs.conf:        type => 'apache_access'
000_inputs.conf:        port => 5581
000_inputs.conf:    }
000_inputs.conf:}
000_inputs.conf:
000_inputs.conf:#
000_inputs.conf:# Local inputs
000_inputs.conf:#
000_inputs.conf:
000_inputs.conf:
500_filters.conf:# 
500_filters.conf:# Logstash Configuration File
500_filters.conf:# Dynamically created by Nagios Log Server
500_filters.conf:#
500_filters.conf:# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
500_filters.conf:#
500_filters.conf:# Created Fri, 20 Jan 2017 14:10:38 -0500
500_filters.conf:#
500_filters.conf:
500_filters.conf:#
500_filters.conf:# Global filters
500_filters.conf:#
500_filters.conf:
500_filters.conf:filter {
500_filters.conf:    if [program] == 'apache_access' {
500_filters.conf:        grok {
500_filters.conf:            match => [ 'message', '%{COMBINEDAPACHELOG}']
500_filters.conf:        }
500_filters.conf:        date {
500_filters.conf:            match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z' ]
500_filters.conf:        }
500_filters.conf:        mutate {
500_filters.conf:            replace => [ 'type', 'apache_access' ]
500_filters.conf:             convert => [ 'bytes', 'integer' ]
500_filters.conf:             convert => [ 'response', 'integer' ]
500_filters.conf:        }
500_filters.conf:    }
500_filters.conf:     
500_filters.conf:    if [program] == 'apache_error' {
500_filters.conf:        grok {
500_filters.conf:            match => [ 'message', '\[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\] \[%{WORD:class}\] \[%{WORD:originator} %{IP:clientip}\] %{GREEDYDATA:errmsg}']
500_filters.conf:        }
500_filters.conf:        mutate {
500_filters.conf:            replace => [ 'type', 'apache_error' ]
500_filters.conf:        }
500_filters.conf:    }
500_filters.conf:    if [program] == 'TrexSyncPubRep' {
500_filters.conf:    mutate {
500_filters.conf:    replace => [ 'type', 'TrexSyncPubRep' ]
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:    if [type] == 'asa' {
500_filters.conf:    grok{
500_filters.conf:    match => ['message', '%{SYSLOG5424PRI}%%{WORD:LogType}-%{INT:LogSeverity}-%{INT:LogMessageNumber}: Group = %{IPORHOST:Group}, Username = %{IPORHOST:username}, IP = %{IP:IPAddress}, Session disconnected. Session Type: %{WORD:SessionType}, Duration: %{CUSTOM1:DurationDays=[0-9]?}%{CUSTOM2=d? ?}%{INT:DurationHours:int}h:%{INT:DurationMinutes:int}m:%{INT:DurationSeconds:int}s, Bytes xmt: %{INT:BytesTransmitted:int}, Bytes rcv: %{INT:BytesReceived:int}, Reason: %{GREEDYDATA:Reason}']
500_filters.conf:    }
500_filters.conf:    geoip {
500_filters.conf:      source => "IPAddress"
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:    if [program] == 'apache_access' {
500_filters.conf:        geoip {
500_filters.conf:            source => 'clientip'
500_filters.conf:        }
500_filters.conf:    }
500_filters.conf:    if [program] == 'TrexSyncRep' {
500_filters.conf:    mutate {
500_filters.conf:    replace => [ 'type', 'TrexSyncRep' ]
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:    if [program] == 'Jupiter_log' {
500_filters.conf:    mutate {
500_filters.conf:    replace => [ 'type', 'Jupiter' ]
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:    if [program] == 'diablo_in1_video_management' {
500_filters.conf:    mutate {
500_filters.conf:    replace => [ 'type', 'diablo' ]
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:    if [program] == 'PUB_API_ACCESS' {
500_filters.conf:    mutate {
500_filters.conf:    replace => [ 'type', 'APIaccess' ]
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:    if [program] == 'sudo' {
500_filters.conf:    mutate {
500_filters.conf:    replace => [ 'type', 'sudo' ]
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:    if [program] == 'opt_lrms_logs_cmgopher' {
500_filters.conf:    mutate {
500_filters.conf:    replace => [ 'type', 'CMGopher_LRMS' ]
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:    if [program] == 'lrms_user_mgmt' {
500_filters.conf:    mutate {
500_filters.conf:    replace => [ 'type', 'User_Mgmt_LRMS' ]
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:    if [program] == 'opt_lrms_logs_uam' {
500_filters.conf:    mutate {
500_filters.conf:    replace => [ 'type', 'UAMGopher_LRMS' ]
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:    if [program] == 'cm_log' {
500_filters.conf:    mutate {
500_filters.conf:    replace => [ 'type', 'CM_LOG' ]
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:    if [program] == 'Epsy_log' {
500_filters.conf:    mutate {
500_filters.conf:    replace => [ 'type', 'Epsy_log' ]
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:    if [program] == 'Wowzastream_access' {
500_filters.conf:    mutate {
500_filters.conf:    replace => [ 'type', 'wowzastream' ]
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:    if [program] == 'Wowzastream_error' {
500_filters.conf:    mutate {
500_filters.conf:    replace => [ 'type', 'wowzastream' ]
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:    if [program] == 'lrms_logs' {
500_filters.conf:    mutate {
500_filters.conf:    replace => [ 'type', 'LRMS_LOG' ]
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:    if [program] == 'ca_logs' {
500_filters.conf:    mutate {
500_filters.conf:    replace => [ 'type', 'CA_LOG' ]
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:    if [program] == 'datacleanup_logs' {
500_filters.conf:    mutate {
500_filters.conf:    replace => [ 'type', 'DATACLEANUP_LOG' ]
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:    if [program] == 'draftingrequest_logs' {
500_filters.conf:    mutate {
500_filters.conf:    replace => [ 'type', 'DRAFTINGREQUEST_LOG' ]
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:    if [program] == 'cmgopher_log' {
500_filters.conf:    mutate {
500_filters.conf:    replace => [ 'type', 'CMGHPHER_LOG' ]
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:    if [program] == 'lm_log' {
500_filters.conf:    mutate {
500_filters.conf:    replace => [ 'type', 'LM_LOG' ]
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:    if [type] == "syslog" {
500_filters.conf:       grok {
500_filters.conf:           match => { "message" => "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:}
500_filters.conf:
500_filters.conf:#
500_filters.conf:# Local filters
500_filters.conf:#
500_filters.conf:
500_filters.conf:
999_outputs.conf:# 
999_outputs.conf:# Logstash Configuration File
999_outputs.conf:# Dynamically created by Nagios Log Server
999_outputs.conf:#
999_outputs.conf:# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
999_outputs.conf:#
999_outputs.conf:# Created Fri, 20 Jan 2017 14:10:38 -0500
999_outputs.conf:#
999_outputs.conf:
999_outputs.conf:#
999_outputs.conf:# Required output for Nagios Log Server
999_outputs.conf:#
999_outputs.conf:
999_outputs.conf:output {
999_outputs.conf:    elasticsearch {
999_outputs.conf:        cluster => '25e0abdc-5b56-4815-adcb-4239555d0899'
999_outputs.conf:        host => 'localhost'
999_outputs.conf:        document_type => '%{type}'
999_outputs.conf:        node_name => 'bb8f313e-98b6-4e1d-8ac4-19e6421ac511'
999_outputs.conf:        protocol => 'transport'
999_outputs.conf:        workers => 4
999_outputs.conf:    }
999_outputs.conf:}
999_outputs.conf:
999_outputs.conf:#
999_outputs.conf:# Global outputs
999_outputs.conf:#
999_outputs.conf:
999_outputs.conf:
999_outputs.conf:
999_outputs.conf:#
999_outputs.conf:# Local outputs
999_outputs.conf:#
999_outputs.conf:
999_outputs.conf:
[root@IGAnagioslog conf.d]#

Post by **mcapra** » Fri Jan 20, 2017 2:29 pm

It looks as if your apache logs don't have the program field set, which is what our default apache filter looks for:

Code: Select all

if [program] == 'apache_access' {
        grok {
            match => [ 'message', '%{COMBINEDAPACHELOG}']
        }
        date {
            match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z' ]
        }
        mutate {
            replace => [ 'type', 'apache_access' ]
             convert => [ 'bytes', 'integer' ]
             convert => [ 'response', 'integer' ]
        }
    }

A simple modification of this filter to match if [type] == 'apache_access' should start tagging your events correctly moving forward.

Nagios Support Forum

Better Apache Dashboard stopped working

Re: Better Apache Dashboard stopped working

Re: Better Apache Dashboard stopped working

Re: Better Apache Dashboard stopped working

Re: Better Apache Dashboard stopped working

Re: Better Apache Dashboard stopped working

Re: Better Apache Dashboard stopped working

Re: Better Apache Dashboard stopped working

Re: Better Apache Dashboard stopped working

Re: Better Apache Dashboard stopped working

Re: Better Apache Dashboard stopped working