Better Apache Dashboard stopped working

[GhostRider2110]

I'm not sure when, but sometime after one of the last couple updates, the Better Apache Dashboard, stopped working. I had not looked at it in a while so I really can't pin down when. We had some network changes to our external servers which caused the logs sending to be blocked and had them shut down until we got the work around in place. Now that it is and logs are being sent to the NLS again, I'm not getting anything from the BA Dashboard but the blank tiles just begging to have data in them.

2 node cluster, 1.4.2 One is the VM from Nagios the other is scratch build on RHEL 7. Both indicate they are running nomally.

I have checked the /usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf file and the is there on both systems:

Code: Select all

if [program] == 'apache_access' {
    grok {
        match => [ 'message', '%{COMBINEDAPACHELOG}']
    }
    date {
        match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z' ]
    }
    mutate {
        replace => [ 'type', 'apache_access' ]
         convert => [ 'bytes', 'integer' ]
         convert => [ 'response', 'integer' ]
    }
}
 
if [program] == 'apache_error' {
    grok {
        match => [ 'message', '\[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\] \[%{WORD:class}\] \[%{WORD:originator} %{IP:clientip}\] %{GREEDYDATA:errmsg}']
    }
    mutate {
        replace => [ 'type', 'apache_error' ]
    }
}

I believe the problem is that the replace type to apache_access is not working. When I look at one of the log entries I see the _type is still syslog. Here is the message entry:

Code: Select all

133>Sep 15 09:13:59 igarh7webcache01 apache_access: 5.9.6.51 - - [15/Sep/2016:09:13:57 -0400] "GET /legislative/2014/bills/house/1113 HTTP/1.1" 200 9419 "-" "Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler)"

But the syslog_program is showing apache_access.

Any help would be greatly appreciated.

See-ya
Mitch

[mcapra]

Does adjusting the filter to say

Code: Select all

if [syslog_program] == 'apache_access'

Change anything?

If not, can we see the syslog/rsyslog configuration this machine is using to send the apache logs?

[GhostRider2110]

Made the change, save& apply no change.

Here is from 90-nagioslogserver_var_log_httpd_access_log.conf

Code: Select all

$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/lib/rsyslog

# Input for apache_access
$InputFileName /var/log/httpd/access_log
$InputFileTag apache_access:
$InputFileStateFile nls-state-var_log_httpd_access_log # Must be unique for each file being polled
# Uncomment the folowing line to override the default severity for messages
# from this file.
#$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor

# Forward to Nagios Log Server and then discard, otherwise these messages
# will end up in the syslog file (/var/log/messages) unless there are other
# overriding rules.
if $programname == 'apache_access' then @@iganagioslog.iga.local:5544
if $programname == 'apache_access' then ~

[mcapra]

Are you able to share the contents of /var/log/logstash/logstash.log?

I would also try sending the events over the port used for raw tcp/udp inputs. If one doesn't exist, you can define an input like this:

Code: Select all

tcp {
    type => 'apache_access'
    port => 5581
}
udp {
    type => 'apache_access'
    port => 5581
}

And alter the last line of your configuration file to use the newly defined port for those raw inputs:

Code: Select all

if $programname == 'apache_access' then @@iganagioslog.iga.local:5581

[GhostRider2110]

Ok I have made those changes and confirmed the logs are still coming in. Attached is the logfile.

See-ya
Mitch

[mcapra]

Looks like logstash is forcibly closing connections. Which host is attempting to send the apache logs?

Can you restart the logstash service on both of your nodes and share the output of the command?

Also how are these logs reaching your NLS environment? Do you have a load balancer, rrdns, single node gets all the apache logs, etc?

[GhostRider2110]

Sorry was on vacation...

Currently all logs are going to one of the nodes. Looking at the IP addresses, several different systems are showing that closed error, yet there are logs coming in from those systems.
See screen shot attached.


Restart of logstash on node 1:

[root@IGAnagioslog ~]# service logstash restart
Restarting Logstash Daemon: [ OK ]
WARNING: Default JAVA_OPTS will be overridden by the JAVA_OPTS defined in the environment. Environment JAVA_OPTS are -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp
[ OK ]
[root@IGAnagioslog ~]# Oct 07, 2016 10:28:33 AM org.elasticsearch.plugins.PluginsService <init>
INFO: [bb8f313e-98b6-4e1d-8ac4-19e6421ac511] loaded [], sites []
Oct 07, 2016 10:28:35 AM org.elasticsearch.plugins.PluginsService <init>
INFO: [bb8f313e-98b6-4e1d-8ac4-19e6421ac511] loaded [], sites []
Oct 07, 2016 10:28:35 AM org.elasticsearch.plugins.PluginsService <init>
INFO: [bb8f313e-98b6-4e1d-8ac4-19e6421ac511] loaded [], sites []
Oct 07, 2016 10:28:35 AM org.elasticsearch.plugins.PluginsService <init>
INFO: [bb8f313e-98b6-4e1d-8ac4-19e6421ac511] loaded [], sites []
Oct 07, 2016 10:28:35 AM org.elasticsearch.plugins.PluginsService <init>
INFO: [bb8f313e-98b6-4e1d-8ac4-19e6421ac511] loaded [], sites []

See-ya
Mitch

[rkennedy]

Also how are these logs reaching your NLS environment? Do you have a load balancer, rrdns, single node gets all the apache logs, etc?

Could you answer these questions?

Could you post your /var/log/logstash/logstash.log for us to look at once again?

[GhostRider2110]

Sorry thought I did... Not using load balancer, All sending direct to node1.

Attaching logstash.log file from node 1. Logstatsh.log file from node 2 is currently empty.

See-ya
Mitch

[avandemore]

Ghostrider,

What happens if you try the solution listed here:

https://support.nagios.com/forum/viewto ... 93#p192160