Better Apache Dashboard stopped working

[GhostRider2110]

It looks as if your apache logs don't have the program field set, which is what our default apache filter looks for:

Code: Select all

if [program] == 'apache_access' {
        grok {
            match => [ 'message', '%{COMBINEDAPACHELOG}']
        }
        date {
            match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z' ]
        }
        mutate {
            replace => [ 'type', 'apache_access' ]
             convert => [ 'bytes', 'integer' ]
             convert => [ 'response', 'integer' ]
        }
    }

A simple modification of this filter to match if [type] == 'apache_access' should start tagging your events correctly moving forward.

So I should change the first line from

Code: Select all

if [program] == 'apache_access' {

to

Code: Select all

if [type] == 'apache_access

?

[mcapra]

Based on how you're receiving the apache logs, you should be able to just swap program with type in this case and apply the new configuration. Though, I can't account for every possible message that may get sent to port 5581 and typed as apache_access.

[GhostRider2110]

Just for grins, I did switch "program" to "type" and seemed to get more working.

I'm not seeing "verb" "response" "request" but still no "logsource"

I'm just wondering why this would have changed. It was working for a long while, and then just stopped. I nor anyone change that filter??

[mcapra]

How were/are these logs being sent to NLS? If it's via rsyslog/nxlog/syslog-ng, it might have something to do with the configuration or changes to the agent itself. I'd like to look at the configuration being used if possible.

The stock apache filter we provide also might not be applicable to your logs. If you could provide some samplings of your access log, I can look over it.

[GhostRider2110]

How were/are these logs being sent to NLS? If it's via rsyslog/nxlog/syslog-ng, it might have something to do with the configuration or changes to the agent itself. I'd like to look at the configuration being used if possible.

The stock apache filter we provide also might not be applicable to your logs. If you could provide some samplings of your access log, I can look over it.

Using rsyslog to send from the clients.

Code: Select all

[lsaadmin@igarh7webcache01 rsyslog.d]$ cat 90-nagioslogserver_var_log_httpd_access_log.conf 
$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/lib/rsyslog

# Input for apache_access
$InputFileName /var/log/httpd/access_log
$InputFileTag apache_access:
$InputFileStateFile nls-state-var_log_httpd_access_log # Must be unique for each file being polled
# Uncomment the folowing line to override the default severity for messages
# from this file.
#$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor

# Forward to Nagios Log Server and then discard, otherwise these messages
# will end up in the syslog file (/var/log/messages) unless there are other
# overriding rules.
if $programname == 'apache_access' then @@iganagioslog.iga.local:5581
if $programname == 'apache_access' then ~
[lsaadmin@igarh7webcache01 rsyslog.d]$ 
[lsaadmin@igarh7webcache01 rsyslog.d]$ 
[lsaadmin@igarh7webcache01 rsyslog.d]$ cat 90-nagioslogserver_var_log_httpd_error_log.conf 
$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/lib/rsyslog

# Input for apache_error
$InputFileName /var/log/httpd/error_log
$InputFileTag apache_error:
$InputFileStateFile nls-state-var_log_httpd_error_log # Must be unique for each file being polled
# Uncomment the folowing line to override the default severity for messages
# from this file.
#$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor

# Forward to Nagios Log Server and then discard, otherwise these messages
# will end up in the syslog file (/var/log/messages) unless there are other
# overriding rules.
if $programname == 'apache_error' then @@iganagioslog.iga.local:5544
if $programname == 'apache_error' then ~
[lsaadmin@igarh7webcache01 rsyslog.d]$ 

Code: Select all

access_log
70.196.18.85 - - [20/Jan/2017:15:29:17 -0500] "GET /static-documents/9/9/8/6/998621ac/8.png HTTP/1.1" 200 436845 "https://iga.in.gov/documents/998621ac" "Mozilla/5.0 (iPhone; CPU iPhone OS 10_2 like Mac OS X) AppleWebKit/602.3.12 (KHTML, like Gecko) Version/10.0 Mobile/14C92 Safari/602.1"
66.244.112.201 - - [20/Jan/2017:15:29:17 -0500] "GET /static-documents/d/7/0/b/d70bc9c3/210.png HTTP/1.1" 200 8484 "http://iga.in.gov/legislative/2014/publications/handbooks/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"
157.55.39.191 - - [20/Jan/2017:15:29:18 -0500] "GET /legislative/2017/portraits/legislator_ryan_dvorak_697/ HTTP/1.1" 200 31124 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
69.27.60.73 - - [20/Jan/2017:15:29:18 -0500] "GET /static-documents/e/a/d/b/eadbab4a/TITLE29_title29.pdf HTTP/1.1" 206 16384 "-" "Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko"
68.32.101.170 - - [20/Jan/2017:15:29:18 -0500] "GET /static-documents/5/2/0/b/520b05dc/TITLE32_title32.pdf HTTP/1.1" 206 24576 "https://iga.in.gov/static-documents/5/2/0/b/520b05dc/TITLE32_title32.pdf" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 Edge/14.14393"
69.1.140.14 - - [20/Jan/2017:15:29:18 -0500] "GET /static-documents/2/6/5/b/265bcadb/2.svgz HTTP/1.1" 200 11345 "http://iga.in.gov/legislative/laws/2016/ic/titles/035/articles/042/chapters/002/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"
69.1.140.14 - - [20/Jan/2017:15:29:18 -0500] "GET /static-documents/2/6/5/b/265bcadb/2.png HTTP/1.1" 200 337885 "http://iga.in.gov/legislative/laws/2016/ic/titles/035/articles/042/chapters/002/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"
10.8.53.21 - - [20/Jan/2017:15:29:18 -0500] "GET /" 200 55777 "-" "-"
68.32.101.170 - - [20/Jan/2017:15:29:18 -0500] "GET /static-documents/5/2/0/b/520b05dc/TITLE32_title32.pdf HTTP/1.1" 206 4268032 "https://iga.in.gov/static-documents/5/2/0/b/520b05dc/TITLE32_title32.pdf" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 Edge/14.14393"
184.71.250.14 - - [20/Jan/2017:15:29:19 -0500] "GET /static/img/ico_caret.gif HTTP/1.1" 304 - "https://iga.in.gov/static/css/bootstrap.css" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"

If you need more of a sample, let me know.

Thanks!

[mcapra]

Everything checks out so far. My best guess is that the rsyslog agent was updated as part of a regular package update (via yum/apt) which may have caused some deprecated configuration conventions to be discarded (namely the one responsible for setting the program field).

Can you share the output of rsyslogd -version on the machine sending the apache logs?

[GhostRider2110]

We use standard RHEL 7. The current version:

[lsaadmin@igarh7webcache01 ~]$ rsyslogd -version
rsyslogd 7.4.7, compiled with:
FEATURE_REGEXP: Yes
FEATURE_LARGEFILE: No
GSSAPI Kerberos 5 support: Yes
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
Runtime Instrumentation (slow code): No
uuid support: Yes

See http://www.rsyslog.com for more information.

We are further along:
Better Apache Dashboard is now actually showing the geoip info and the response codes. Bandwidth and Events are still not there...

BetterApache-partworking-1.png

BetterApache-parkworking-2.png

The default Apache Dashboard is showing even more. Everything except Log Sources:

ApacheDash-working-1.png

[mcapra]

The logsource field is typically set by the default syslog grok pattern. Since we're not feeding it into the syslog input on port 5544, the logsource field isn't being set. The field doesn't really serve a purpose though if the machine sending the apache logs (to NLS) is the same machine holding the apache logs.

As far as the bandwidth panel not showing up on the Better Apache Dashboard, I would suggest importing the dashboard again and seeing if that reproduces all the panels. Generally a panel won't go missing when something isn't configured correctly; It'll holler with a lot of red text.

[GhostRider2110]

We do use logsource, since we have both an internal and external setup with multiple load balanced front end servers. That way we can keep track of possible issues within one server. Also there are several back end servers behind the caching servers.

I'll have to go back and remember how I imported that dashboard.. LOL been a while since I imported any into the log server. Will let you know.

[dwhitfield]

Will let you know.

Sounds great. We'll be here.