trial issue, setup multiple sources, none showing up

[ucemike]

Interesting. Let's take a look at your logstash configuration. Please run the following on the CLI and report the output to us:

Code: Select all

cat /usr/local/nagioslogserver/logstash/etc/conf.d/*

This should show us all of your logstash inputs, filters, and outputs - I would like to verify that they look correct.

Code: Select all

cat /usr/local/nagioslogserver/logstash/etc/conf.d/*
#
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Wed, 06 May 2015 17:19:56 -0400
#

#
# Global inputs
#

input {
    syslog {
        type => 'syslog'
        port => 5544
    }
    tcp {
        type => 'eventlog'
        port => 3515
        codec => json {
            charset => 'CP1252'
        }
    }
    tcp {
        type => 'import_raw'
        tags => 'import_raw'
        port => 2056
    }
    tcp {
        type => 'import_json'
        tags => 'import_json'
        port => 2057
        codec => json
    }
}

#
# Local inputs
#


#
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Wed, 06 May 2015 17:19:56 -0400
#

#
# Global filters
#

filter {
    if [program] == 'apache_access' {
        grok {
            match => [ 'message', '%{COMBINEDAPACHELOG}']
        }
        date {
            match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z' ]
        }
        mutate {
            replace => [ 'type', 'apache_access' ]
             convert => [ 'bytes', 'integer' ]
             convert => [ 'response', 'integer' ]
        }
    }

    if [program] == 'apache_error' {
        grok {
            match => [ 'message', '\[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\] \[%{WORD:class}\] \[%{WORD:originator} %{IP:clientip}\] %{GREEDYDATA:errmsg}']
        }
        mutate {
            replace => [ 'type', 'apache_error' ]
        }
    }
}

#
# Local filters
#


#
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Wed, 06 May 2015 17:19:56 -0400
#

#
# Required output for Nagios Log Server
#

output {
    elasticsearch {
        cluster => 'c012c664-0f59-41f2-9acb-c0ad2dd5f318'
        host => 'localhost'
        index_type => '%{type}'
        node_name => '7189e740-df68-4ea3-8bba-f430fe37d1eb'
        protocol => 'transport'
        workers => 4
    }
}

#
# Global outputs
#



#
# Local outputs
#

How many nodes are in this cluster - is this a single node?
I'd like to see the output of this to ensure your output is configured properly:

Code: Select all

cat /usr/local/nagioslogserver/var/cluster_uuid

Code: Select all

 cat /usr/local/nagioslogserver/var/cluster_uuid
c012c664-0f59-41f2-9acb-c0ad2dd5f318

Also, let's look at one of your clients rsyslog configurations to ensure it's put together correctly:

Code: Select all

cat /etc/rsyslog.conf
cat /etc/rsyslog.d/*log*

Code: Select all

cat /etc/rsyslog.conf
# rsyslog v5 configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imklog   # provides kernel logging support (previously done by rklogd)
$ModLoad immark   # provides --MARK-- message capability

# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514

# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514


#### GLOBAL DIRECTIVES ####

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf


#### RULES ####

local0.info;mark.debug                                  /var/log/local0/info
local1.info;mark.debug                                  /var/log/local1/info
local2.info;mark.debug                                  /var/log/local2/info
local3.info;mark.debug                                  /var/log/local3/info
local4.info;mark.debug                                  /var/log/local4/info
local5.info;mark.debug                                  /var/log/local5/info
local6.info;mark.debug                                  /var/log/local6/info
local7.info;mark.debug                                  /var/log/local7/info

local0,mark.debug                                       /var/log/local0/debug
local1,mark.debug                                       /var/log/local1/debug
local2,mark.debug                                       /var/log/local2/debug
local3,mark.debug                                       /var/log/local3/debug
local4,mark.debug                                       /var/log/local4/debug
local6,mark.debug                                       /var/log/local6/debug
local5,mark.debug                                       /var/log/local5/debug
local7,mark.debug                                       /var/log/local7/debug

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog


# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 *

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log

*.debug                                                 @10.1.2.X
& @10.1.1.X

# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/lib/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList   # run asynchronously
#$ActionResumeRetryCount -1    # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###

Code: Select all

 cat /etc/rsyslog.d/*log*
### Begin forwarding rule for Nagios Log Server                           NAGIOSLOGSERVER
$WorkDirectory /var/lib/rsyslog # Where spool files will live             NAGIOSLOGSERVER
$ActionQueueFileName nlsFwdRule0 # Unique name prefix for spool files     NAGIOSLOGSERVER
$ActionQueueMaxDiskSpace 1g   # 1GB space limit (use as much as possible) NAGIOSLOGSERVER
$ActionQueueSaveOnShutdown on # Save messages to disk on shutdown         NAGIOSLOGSERVER
$ActionQueueType LinkedList   # Use asynchronous processing               NAGIOSLOGSERVER
$ActionResumeRetryCount -1    # Infinite retries if host is down          NAGIOSLOGSERVER
# Remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional       NAGIOSLOGSERVER
*.* @@nagios-log1.HosTNameHere:5544                                               # NAGIOSLOGSERVER
### End of Nagios Log Server forwarding rule                              NAGIOSLOGSERVER

Also restarted rsyslog.

[jolson]

Everything looks great.

I take it elasticsearch is fine?

Code: Select all

service elasticsearch status
tail /var/log/elasticsearch/*.log

Can you search for one of your hosts that is exporting across the last few days? Maybe they just aren't sending many logs.

2015-05-06 13_58_07-Dashboard • Nagios Log Server.png

[ucemike]

Okay, so since the 5 minute test you had be run I have started seeing some entries. It seems that it's running about 1-2(?) hours after I added the client/log source. It also seems the logs are running delayed like that as well. I now see all three. They slowly started showing entries in the order I added them as a log source.

Is the delay in logs normal? I double checked the VM wasn't starved for resources but initial review doesn't seem to indicate cpu/disk/memory issues.

[jolson]

Can you please check the date of your Nagios Log Server? It's possible that the date is causing these delays.

You can change the date using the following script:

Code: Select all

/usr/local/nagioslogserver/scripts/change_timezone.sh -z America/Chicago

Replace 'America/Chicago' with your appropriate timezone.

[ucemike]

So I double checked that and it was not wildly off.

I'm using the "logger" tool to add entries to the system and waiting for it to show up. I'll update when comes through.

[jolson]

Sounds good - let us know what you find out. Thanks!

[ucemike]

So I double checked that and it was not wildly off.

I'm using the "logger" tool to add entries to the system and waiting for it to show up. I'll update when comes through.

So it looks like the log entries on the NLS are running 4 hours behind the actual alert on the log source. I ran the simulated log add at the same time as I was running tcpdump so I'm fairly certain it was sending it immediately.

Is there a way to look at incoming entries to the NLS? I've not been able to locate anything like that yet.

[Box293]

Run this command:

Code: Select all

ls -al /etc/localtime

Which will output something like:

Code: Select all

lrwxrwxrwx 1 root root 39 Jun 26  2014 /etc/localtime -> /usr/share/zoneinfo/Australia/Melbourne

Make sure this is your correct timezone.

Run these two commands

Code: Select all

date

hwclock

Make sure they have the same date/time.

Run this command:

Code: Select all

cat /etc/php.ini | grep date.time

Make sure it has the correct timezone, in my example above it should be:

Code: Select all

date.timezone = Australia/Melbourne

If you do change it restart the httpd service

Code: Select all

service httpd restart

Run this command:

Code: Select all

cat /etc/sysconfig/clock

Make sure it has the correct timezone, in my example it should be

Code: Select all

ZONE="Australia/Melbourne"

If you change it, restart logstash

Code: Select all

service logstash restart

[ucemike]

Code: Select all

[root@nagios-log1 log]#  ls -al /etc/localtime
lrwxrwxrwx 1 root root 35 May  6 19:21 /etc/localtime -> /usr/share/zoneinfo/America/Chicago
[root@nagios-log1 log]# date
Thu May  7 04:02:02 CDT 2015
[root@nagios-log1 log]# hwclock
Thu 07 May 2015 05:02:05 AM CDT  -0.575713 seconds

I ran that at 12AM central (may 7)

Looks like the OVF has some weird time sync issue.

That said, why would NLS care? Shouldn't the data show up in the log list when it was received regardless of what time the OS thinks it is?

Thank you for the debug help so far. You've been great help. Normally I'd create my own VM and manually install and I might of had less issues (it's pretty standardized) but I wanted to try the "easy" way

[ucemike]

So, I set my timezone to GMT as that's what all the log sources are and my initial tests look good.

I'll experiment and see how it works out. Thanks again for your assistance.