Filter build help..

[GhostRider2110]

Nagios Log Server • 1.4.4

I have some logs from an application. I setup the .conf file for rsyslog to sending the log to the NLS via port 5514, which I checked my other inputs and was not being used.

Code: Select all

$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/lib/rsyslog

# Input for API_ACCESS
$InputFileName /opt/in1_public_api_internal/logs/access.log
$InputFileTag API_ACCESS:
$InputFileStateFile nls-state-opt_in1_public_api_internal_logs_access.log # Must be unique for each file being polled
# Uncomment the folowing line to override the default severity for messages
# from this file.
#$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor

# Forward to Nagios Log Server and then discard, otherwise these messages
# will end up in the syslog file (/var/log/messages) unless there are other
# overriding rules.
if $programname == 'api_access' then @@iganagioslog.iga.local:5514
if $programname == 'api_access' then ~

I have checked the firewall and there is nothing in the way there.

I setup under inputs to set the type to api_access:

Code: Select all

tcp {
    type => 'api_access'
    port => 5514
}

Then I created a filter for that.

Code: Select all

if [type] == 'api_access' {
  grok {
       match => [ "message" , "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{HOSTNAME:loghost} %{DATA:syslog_program}: %{TIMESTAMP_ISO8601},%{USERNAME},%{DATA:rawrequest},%{BASE10NUM:response},%{NUMBER:bytes}" ]
   }
}

I used one of the grok builder sites to help me this that.

I'm still getting the logs but they are as type syslog.

api_access-01.png

What am I missing?

My goal is the break out the messages in them as with the filter above to give me better reports. Example message looks like:

Code: Select all

<133>Feb 22 10:21:12 igarh7pubapi API_ACCESS: 2017-02-22T10:21:05.817983,nagios1,/2016/bills,200,368449

Thanks

[mcapra]

Hmm, the filter looks solid so I don't think that's the problem.

Can you share the outputs of the following commands executed from a Nagios Log Server node:

Code: Select all

grep '' /usr/local/nagioslogserver/logstash/etc/conf.d/*
netstat -an | grep 5514

As well as the outputs of the following commands executed from the remote machine containing the API logs:

Code: Select all

cat /etc/rsyslog.conf
grep '' /etc/rsyslog.d/*
netstat -an | grep 5514
netstat -an | grep 5544

[GhostRider2110]

Thanks, after looking at some network captures, I can't seem to see anything from that host coming into the NLS port 5514.
It does seem that the NLS is listening on that port:

Code: Select all

[root@IGAnagioslog ~]# netstat -an | grep 5514
tcp        0      0 :::5514                     :::*                        LISTEN  

Code: Select all

[root@IGAnagioslog ~]# cd /usr/local/nagioslogserver/logstash/etc/conf.d/
[root@IGAnagioslog conf.d]# grep '' *
000_inputs.conf:# 
000_inputs.conf:# Logstash Configuration File
000_inputs.conf:# Dynamically created by Nagios Log Server
000_inputs.conf:#
000_inputs.conf:# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
000_inputs.conf:#
000_inputs.conf:# Created Wed, 22 Feb 2017 09:27:18 -0500
000_inputs.conf:#
000_inputs.conf:
000_inputs.conf:#
000_inputs.conf:# Global inputs
000_inputs.conf:#
000_inputs.conf:
000_inputs.conf:input {
000_inputs.conf:    tcp {
000_inputs.conf:        type => 'api_access'
000_inputs.conf:        port => 5514
000_inputs.conf:    }
000_inputs.conf:    tcp {
000_inputs.conf:        type => 'eventlog'
000_inputs.conf:        port => 3515
000_inputs.conf:        codec => json {
000_inputs.conf:            charset => 'CP1252'
000_inputs.conf:        }
000_inputs.conf:    }
000_inputs.conf:    tcp {
000_inputs.conf:        type => 'import_raw'
000_inputs.conf:        tags => 'import_raw'
000_inputs.conf:        port => 2056
000_inputs.conf:    }
000_inputs.conf:    tcp {
000_inputs.conf:        type => 'import_json'
000_inputs.conf:        tags => 'import_json'
000_inputs.conf:        port => 2057
000_inputs.conf:        codec => json
000_inputs.conf:    }
000_inputs.conf:    syslog {
000_inputs.conf:        type => 'syslog'
000_inputs.conf:        port => 514
000_inputs.conf:    }
000_inputs.conf:    syslog {
000_inputs.conf:        type => 'asa'
000_inputs.conf:        port => 6544
000_inputs.conf:    }
000_inputs.conf:    tcp {
000_inputs.conf:        port => 5544
000_inputs.conf:        type => 'syslog'
000_inputs.conf:    }
000_inputs.conf:    udp {
000_inputs.conf:        port => 5544
000_inputs.conf:        type => 'syslog'
000_inputs.conf:    
000_inputs.conf:    }
000_inputs.conf:    tcp {
000_inputs.conf:        type => 'apache_access'
000_inputs.conf:        port => 5581
000_inputs.conf:    }
000_inputs.conf:    udp {
000_inputs.conf:        type => 'apache_access'
000_inputs.conf:        port => 5581
000_inputs.conf:    }
000_inputs.conf:    tcp {
000_inputs.conf:        port => 5540
000_inputs.conf:        type => 'Wowza'
000_inputs.conf:    }
000_inputs.conf:    tcp {
000_inputs.conf:        type => 'infoblox'
000_inputs.conf:        port => 6514
000_inputs.conf:    }
000_inputs.conf:    udp {
000_inputs.conf:        type => 'infoblox'
000_inputs.conf:        port => 6514
000_inputs.conf:    }
000_inputs.conf:}
000_inputs.conf:
000_inputs.conf:#
000_inputs.conf:# Local inputs
000_inputs.conf:#
000_inputs.conf:
000_inputs.conf:
500_filters.conf:# 
500_filters.conf:# Logstash Configuration File
500_filters.conf:# Dynamically created by Nagios Log Server
500_filters.conf:#
500_filters.conf:# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
500_filters.conf:#
500_filters.conf:# Created Wed, 22 Feb 2017 09:27:18 -0500
500_filters.conf:#
500_filters.conf:
500_filters.conf:#
500_filters.conf:# Global filters
500_filters.conf:#
500_filters.conf:
500_filters.conf:filter {
500_filters.conf:    if [type] == 'apache_access' {
500_filters.conf:        grok {
500_filters.conf:            match => [ 'message', '%{SYSLOGBASE}']
500_filters.conf:        }
500_filters.conf:        grok {
500_filters.conf:            match => [ 'message', '%{COMBINEDAPACHELOG}']
500_filters.conf:        }
500_filters.conf:        date {
500_filters.conf:             match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z' ]
500_filters.conf:        }
500_filters.conf:        mutate {
500_filters.conf:            replace => [ 'type', 'apache_access' ]
500_filters.conf:             convert => [ 'bytes', 'integer' ]
500_filters.conf:             convert => [ 'response', 'integer' ]
500_filters.conf:        }
500_filters.conf:    }
500_filters.conf:     
500_filters.conf:    if [program] == 'apache_error' {
500_filters.conf:        grok {
500_filters.conf:            match => [ 'message', '\[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\] \[%{WORD:class}\] \[%{WORD:originator} %{IP:clientip}\] %{GREEDYDATA:errmsg}']
500_filters.conf:        }
500_filters.conf:        mutate {
500_filters.conf:            replace => [ 'type', 'apache_error' ]
500_filters.conf:        }
500_filters.conf:    }
500_filters.conf:    if [type] == 'api_access' {
500_filters.conf:      grok {
500_filters.conf:           match => [ "message" , "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{HOSTNAME:loghost} %{DATA:syslog_program}: %{TIMESTAMP_ISO8601},%{USERNAME},%{DATA:rawrequest},%{BASE10NUM:response},%{NUMBER:bytes}" ]
500_filters.conf:       }
500_filters.conf:    }
500_filters.conf:    if [type] == "syslog" {
500_filters.conf:       grok {
500_filters.conf:           match => [ "message" , "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" ]
500_filters.conf:       }
500_filters.conf:    }
500_filters.conf:    
500_filters.conf:    if [type] == "Wowza" {
500_filters.conf:       grok {
500_filters.conf:           match => [ message,"%{DATE:date}\t%{TIME:time}\t%{DATA:tz}\t%{DATA:x-event}\t%{DATA:x-category}\t%{DATA:x-severity}\t%{NUMBER:x-status:integer}\t%{DATA:x-ctx}\t%{DATA:x-comment}\t%{DATA:x-vhost}\t%{DATA:x-app}\t%{DATA:x-appinst}\t%{DATA:x-duration:float}\t%{DATA:s-ip}\t%{DATA:s-port:integer}\t%{DATA:s-uri}\t%{DATA:c-ip}\t%{DATA:c-proto}\t%{DATA:c-referrer}\t%{DATA:c-user-agent}\t%{DATA:client-id}\t%{DATA:cs-bytes:float}\t%{DATA:sc-bytes:float}\t%{DATA:x-stream-id}\t%{DATA:x-spos:float}\t%{DATA:cs-stream-bytes:float}\t%{DATA:sc-stream-bytes:float}\t%{DATA:x-sname}\t%{DATA:x-sname-query}\t%{DATA:x-file-name}\t%{DATA:x-file-ext}\t%{DATA:x-file-size:float}\t%{DATA:x-file-length:float}\t%{DATA:x-suri}\t%{DATA:x-suri-stem}\t%{DATA:x-suri-query}\t%{DATA:cs-uri-stem}\t%{GREEDYDATA:cs-uri-query}" ]
500_filters.conf:    
500_filters.conf:           add_field => ["logtimestamp", "%{date} %{time}" ]
500_filters.conf:           remove_field => [ "time","date" ]
500_filters.conf:         }
500_filters.conf:    }
500_filters.conf:    
500_filters.conf:    if [type] == 'asa' {
500_filters.conf:    grok{
500_filters.conf:    match => ['message', '%{SYSLOG5424PRI}%%{WORD:LogType}-%{INT:LogSeverity}-%{INT:LogMessageNumber}: Group = %{IPORHOST:Group}, Username = %{IPORHOST:username}, IP = %{IP:IPAddress}, Session disconnected. Session Type: %{WORD:SessionType}, Duration: %{CUSTOM1:DurationDays=[0-9]?}%{CUSTOM2=d? ?}%{INT:DurationHours:int}h:%{INT:DurationMinutes:int}m:%{INT:DurationSeconds:int}s, Bytes xmt: %{INT:BytesTransmitted:int}, Bytes rcv: %{INT:BytesReceived:int}, Reason: %{GREEDYDATA:Reason}']
500_filters.conf:    }
500_filters.conf:    geoip {
500_filters.conf:      source => "IPAddress"
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:    if [syslog_program] == 'TrexSyncPubRep' {
500_filters.conf:    mutate {
500_filters.conf:    replace => [ 'type', 'TrexSyncPubRep' ]
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:    if [type] == 'apache_access' {
500_filters.conf:        geoip {
500_filters.conf:            source => 'clientip'
500_filters.conf:        }
500_filters.conf:    }
500_filters.conf:    if [program] == 'TrexSyncRep' {
500_filters.conf:    mutate {
500_filters.conf:    replace => [ 'type', 'TrexSyncRep' ]
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:    if [program] == 'Jupiter_log' {
500_filters.conf:    mutate {
500_filters.conf:    replace => [ 'type', 'Jupiter' ]
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:    if [program] == 'diablo_in1_video_management' {
500_filters.conf:    mutate {
500_filters.conf:    replace => [ 'type', 'diablo' ]
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:    if [program] == 'PUB_API_ACCESS' {
500_filters.conf:    mutate {
500_filters.conf:    replace => [ 'type', 'APIaccess' ]
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:    if [syslog_program] == 'sudo' {
500_filters.conf:    mutate {
500_filters.conf:    replace => [ 'type', 'sudo' ]
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:    if [program] == 'opt_lrms_logs_cmgopher' {
500_filters.conf:    mutate {
500_filters.conf:    replace => [ 'type', 'CMGopher_LRMS' ]
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:    if [program] == 'lrms_user_mgmt' {
500_filters.conf:    mutate {
500_filters.conf:    replace => [ 'type', 'User_Mgmt_LRMS' ]
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:    if [program] == 'opt_lrms_logs_uam' {
500_filters.conf:    mutate {
500_filters.conf:    replace => [ 'type', 'UAMGopher_LRMS' ]
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:    if [syslog_program] == 'cm_log' {
500_filters.conf:    mutate {
500_filters.conf:    replace => [ 'type', 'CM_LOG' ]
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:    if [program] == 'Epsy_log' {
500_filters.conf:    mutate {
500_filters.conf:    replace => [ 'type', 'Epsy_log' ]
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:    if [program] == 'Wowzastream_access' {
500_filters.conf:    mutate {
500_filters.conf:    replace => [ 'type', 'wowzastream' ]
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:    if [program] == 'Wowzastream_error' {
500_filters.conf:    mutate {
500_filters.conf:    replace => [ 'type', 'wowzastream' ]
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:    if [program] == 'lrms_logs' {
500_filters.conf:    mutate {
500_filters.conf:    replace => [ 'type', 'LRMS_LOG' ]
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:    if [program] == 'ca_logs' {
500_filters.conf:    mutate {
500_filters.conf:    replace => [ 'type', 'CA_LOG' ]
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:    if [program] == 'datacleanup_logs' {
500_filters.conf:    mutate {
500_filters.conf:    replace => [ 'type', 'DATACLEANUP_LOG' ]
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:    if [program] == 'draftingrequest_logs' {
500_filters.conf:    mutate {
500_filters.conf:    replace => [ 'type', 'DRAFTINGREQUEST_LOG' ]
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:    if [syslog_program] == 'cmgopher_log' {
500_filters.conf:    mutate {
500_filters.conf:    replace => [ 'type', 'CMGHPHER_LOG' ]
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:    if [program] == 'lm_log' {
500_filters.conf:    mutate {
500_filters.conf:    replace => [ 'type', 'LM_LOG' ]
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:    if [syslog_program] == 'sshd' {
500_filters.conf:    mutate {
500_filters.conf:    replace => [ 'type', 'sshd' ]
500_filters.conf:    }
500_filters.conf:    }
500_filters.conf:}
500_filters.conf:
500_filters.conf:#
500_filters.conf:# Local filters
500_filters.conf:#
500_filters.conf:
500_filters.conf:
999_outputs.conf:# 
999_outputs.conf:# Logstash Configuration File
999_outputs.conf:# Dynamically created by Nagios Log Server
999_outputs.conf:#
999_outputs.conf:# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
999_outputs.conf:#
999_outputs.conf:# Created Wed, 22 Feb 2017 09:27:18 -0500
999_outputs.conf:#
999_outputs.conf:
999_outputs.conf:#
999_outputs.conf:# Required output for Nagios Log Server
999_outputs.conf:#
999_outputs.conf:
999_outputs.conf:output {
999_outputs.conf:    elasticsearch {
999_outputs.conf:        cluster => '25e0abdc-5b56-4815-adcb-4239555d0899'
999_outputs.conf:        host => 'localhost'
999_outputs.conf:        document_type => '%{type}'
999_outputs.conf:        node_name => 'bb8f313e-98b6-4e1d-8ac4-19e6421ac511'
999_outputs.conf:        protocol => 'transport'
999_outputs.conf:        workers => 4
999_outputs.conf:    }
999_outputs.conf:}
999_outputs.conf:
999_outputs.conf:#
999_outputs.conf:# Global outputs
999_outputs.conf:#
999_outputs.conf:
999_outputs.conf:
999_outputs.conf:
999_outputs.conf:#
999_outputs.conf:# Local outputs
999_outputs.conf:#
999_outputs.conf:
999_outputs.conf:

From the client:

Code: Select all

[root@igarh7pubapi ~]# cat /etc/rsyslog.conf 
# rsyslog configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

# The imjournal module bellow is now used as a message source instead of imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
$imjournalRatelimitInterval 300
$imjournalRatelimitBurst 20000

#$ModLoad imklog # reads kernel messages (the same are read from journald)
#$ModLoad immark  # provides --MARK-- message capability

# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514

# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514


#### GLOBAL DIRECTIVES ####

# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

# Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
$OmitLocalLogging on

# File to store the position in the journal
$IMJournalStateFile imjournal.state


#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog


# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 :omusrmsg:*

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log


# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList   # run asynchronously
#$ActionResumeRetryCount -1    # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###

Code: Select all

[root@igarh7pubapi rsyslog.d]# grep '' *
90-nagioslogserver_opt_in1_public_api_internal_logs_access.log.conf:$ModLoad imfile
90-nagioslogserver_opt_in1_public_api_internal_logs_access.log.conf:$InputFilePollInterval 10
90-nagioslogserver_opt_in1_public_api_internal_logs_access.log.conf:$PrivDropToGroup adm
90-nagioslogserver_opt_in1_public_api_internal_logs_access.log.conf:$WorkDirectory /var/lib/rsyslog
90-nagioslogserver_opt_in1_public_api_internal_logs_access.log.conf:
90-nagioslogserver_opt_in1_public_api_internal_logs_access.log.conf:# Input for API_ACCESS
90-nagioslogserver_opt_in1_public_api_internal_logs_access.log.conf:$InputFileName /opt/in1_public_api_internal/logs/access.log
90-nagioslogserver_opt_in1_public_api_internal_logs_access.log.conf:$InputFileTag API_ACCESS:
90-nagioslogserver_opt_in1_public_api_internal_logs_access.log.conf:$InputFileStateFile nls-state-opt_in1_public_api_internal_logs_access.log # Must be unique for each file being polled
90-nagioslogserver_opt_in1_public_api_internal_logs_access.log.conf:# Uncomment the folowing line to override the default severity for messages
90-nagioslogserver_opt_in1_public_api_internal_logs_access.log.conf:# from this file.
90-nagioslogserver_opt_in1_public_api_internal_logs_access.log.conf:#$InputFileSeverity info
90-nagioslogserver_opt_in1_public_api_internal_logs_access.log.conf:$InputFilePersistStateInterval 20000
90-nagioslogserver_opt_in1_public_api_internal_logs_access.log.conf:$InputRunFileMonitor
90-nagioslogserver_opt_in1_public_api_internal_logs_access.log.conf:
90-nagioslogserver_opt_in1_public_api_internal_logs_access.log.conf:# Forward to Nagios Log Server and then discard, otherwise these messages
90-nagioslogserver_opt_in1_public_api_internal_logs_access.log.conf:# will end up in the syslog file (/var/log/messages) unless there are other
90-nagioslogserver_opt_in1_public_api_internal_logs_access.log.conf:# overriding rules.
90-nagioslogserver_opt_in1_public_api_internal_logs_access.log.conf:if $programname == 'api_access' then @@iganagioslog.iga.local:5514
90-nagioslogserver_opt_in1_public_api_internal_logs_access.log.conf:if $programname == 'api_access' then ~
90-nagioslogserver_opt_in1_public_api_internal_logs_debug.log.conf:$ModLoad imfile
90-nagioslogserver_opt_in1_public_api_internal_logs_debug.log.conf:$InputFilePollInterval 10
90-nagioslogserver_opt_in1_public_api_internal_logs_debug.log.conf:$PrivDropToGroup adm
90-nagioslogserver_opt_in1_public_api_internal_logs_debug.log.conf:$WorkDirectory /var/lib/rsyslog
90-nagioslogserver_opt_in1_public_api_internal_logs_debug.log.conf:
90-nagioslogserver_opt_in1_public_api_internal_logs_debug.log.conf:# Input for API_DEBUG
90-nagioslogserver_opt_in1_public_api_internal_logs_debug.log.conf:$InputFileName /opt/in1_public_api_internal/logs/debug.log
90-nagioslogserver_opt_in1_public_api_internal_logs_debug.log.conf:$InputFileTag API_DEBUG:
90-nagioslogserver_opt_in1_public_api_internal_logs_debug.log.conf:$InputFileStateFile nls-state-opt_in1_public_api_internal_logs_debug.log # Must be unique for each file being polled
90-nagioslogserver_opt_in1_public_api_internal_logs_debug.log.conf:# Uncomment the folowing line to override the default severity for messages
90-nagioslogserver_opt_in1_public_api_internal_logs_debug.log.conf:# from this file.
90-nagioslogserver_opt_in1_public_api_internal_logs_debug.log.conf:#$InputFileSeverity info
90-nagioslogserver_opt_in1_public_api_internal_logs_debug.log.conf:$InputFilePersistStateInterval 20000
90-nagioslogserver_opt_in1_public_api_internal_logs_debug.log.conf:$InputRunFileMonitor
90-nagioslogserver_opt_in1_public_api_internal_logs_debug.log.conf:
90-nagioslogserver_opt_in1_public_api_internal_logs_debug.log.conf:# Forward to Nagios Log Server and then discard, otherwise these messages
90-nagioslogserver_opt_in1_public_api_internal_logs_debug.log.conf:# will end up in the syslog file (/var/log/messages) unless there are other
90-nagioslogserver_opt_in1_public_api_internal_logs_debug.log.conf:# overriding rules.
90-nagioslogserver_opt_in1_public_api_internal_logs_debug.log.conf:if $programname == 'API_DEBUG' then @@iganagioslog.iga.local:5544
90-nagioslogserver_opt_in1_public_api_internal_logs_debug.log.conf:if $programname == 'API_DEBUG' then ~
90-nagioslogserver_opt_in1_public_api_internal_logs_info.log.conf:$ModLoad imfile
90-nagioslogserver_opt_in1_public_api_internal_logs_info.log.conf:$InputFilePollInterval 10
90-nagioslogserver_opt_in1_public_api_internal_logs_info.log.conf:$PrivDropToGroup adm
90-nagioslogserver_opt_in1_public_api_internal_logs_info.log.conf:$WorkDirectory /var/lib/rsyslog
90-nagioslogserver_opt_in1_public_api_internal_logs_info.log.conf:
90-nagioslogserver_opt_in1_public_api_internal_logs_info.log.conf:# Input for API_INFO
90-nagioslogserver_opt_in1_public_api_internal_logs_info.log.conf:$InputFileName /opt/in1_public_api_internal/logs/info.log
90-nagioslogserver_opt_in1_public_api_internal_logs_info.log.conf:$InputFileTag API_INFO:
90-nagioslogserver_opt_in1_public_api_internal_logs_info.log.conf:$InputFileStateFile nls-state-opt_in1_public_api_internal_logs_info.log # Must be unique for each file being polled
90-nagioslogserver_opt_in1_public_api_internal_logs_info.log.conf:# Uncomment the folowing line to override the default severity for messages
90-nagioslogserver_opt_in1_public_api_internal_logs_info.log.conf:# from this file.
90-nagioslogserver_opt_in1_public_api_internal_logs_info.log.conf:#$InputFileSeverity info
90-nagioslogserver_opt_in1_public_api_internal_logs_info.log.conf:$InputFilePersistStateInterval 20000
90-nagioslogserver_opt_in1_public_api_internal_logs_info.log.conf:$InputRunFileMonitor
90-nagioslogserver_opt_in1_public_api_internal_logs_info.log.conf:
90-nagioslogserver_opt_in1_public_api_internal_logs_info.log.conf:# Forward to Nagios Log Server and then discard, otherwise these messages
90-nagioslogserver_opt_in1_public_api_internal_logs_info.log.conf:# will end up in the syslog file (/var/log/messages) unless there are other
90-nagioslogserver_opt_in1_public_api_internal_logs_info.log.conf:# overriding rules.
90-nagioslogserver_opt_in1_public_api_internal_logs_info.log.conf:if $programname == 'API_INFO' then @@iganagioslog.iga.local:5544
90-nagioslogserver_opt_in1_public_api_internal_logs_info.log.conf:if $programname == 'API_INFO' then ~
90-nagioslogserver_var_log_httpd_access_log.conf:$ModLoad imfile
90-nagioslogserver_var_log_httpd_access_log.conf:$InputFilePollInterval 10
90-nagioslogserver_var_log_httpd_access_log.conf:$PrivDropToGroup adm
90-nagioslogserver_var_log_httpd_access_log.conf:$WorkDirectory /var/lib/rsyslog
90-nagioslogserver_var_log_httpd_access_log.conf:
90-nagioslogserver_var_log_httpd_access_log.conf:# Input for apache_access
90-nagioslogserver_var_log_httpd_access_log.conf:$InputFileName /var/log/httpd/access_log
90-nagioslogserver_var_log_httpd_access_log.conf:$InputFileTag apache_access:
90-nagioslogserver_var_log_httpd_access_log.conf:$InputFileStateFile nls-state-var_log_httpd_access_log # Must be unique for each file being polled
90-nagioslogserver_var_log_httpd_access_log.conf:# Uncomment the folowing line to override the default severity for messages
90-nagioslogserver_var_log_httpd_access_log.conf:# from this file.
90-nagioslogserver_var_log_httpd_access_log.conf:#$InputFileSeverity info
90-nagioslogserver_var_log_httpd_access_log.conf:$InputFilePersistStateInterval 20000
90-nagioslogserver_var_log_httpd_access_log.conf:$InputRunFileMonitor
90-nagioslogserver_var_log_httpd_access_log.conf:
90-nagioslogserver_var_log_httpd_access_log.conf:# Forward to Nagios Log Server and then discard, otherwise these messages
90-nagioslogserver_var_log_httpd_access_log.conf:# will end up in the syslog file (/var/log/messages) unless there are other
90-nagioslogserver_var_log_httpd_access_log.conf:# overriding rules.
90-nagioslogserver_var_log_httpd_access_log.conf:if $programname == 'apache_access' then @@iganagioslog.iga.local:5581
90-nagioslogserver_var_log_httpd_access_log.conf:if $programname == 'apache_access' then ~
90-nagioslogserver_var_log_httpd_error_log.conf:$ModLoad imfile
90-nagioslogserver_var_log_httpd_error_log.conf:$InputFilePollInterval 10
90-nagioslogserver_var_log_httpd_error_log.conf:$PrivDropToGroup adm
90-nagioslogserver_var_log_httpd_error_log.conf:$WorkDirectory /var/lib/rsyslog
90-nagioslogserver_var_log_httpd_error_log.conf:
90-nagioslogserver_var_log_httpd_error_log.conf:# Input for apache_error
90-nagioslogserver_var_log_httpd_error_log.conf:$InputFileName /var/log/httpd/error_log
90-nagioslogserver_var_log_httpd_error_log.conf:$InputFileTag apache_error:
90-nagioslogserver_var_log_httpd_error_log.conf:$InputFileStateFile nls-state-var_log_httpd_error_log # Must be unique for each file being polled
90-nagioslogserver_var_log_httpd_error_log.conf:# Uncomment the folowing line to override the default severity for messages
90-nagioslogserver_var_log_httpd_error_log.conf:# from this file.
90-nagioslogserver_var_log_httpd_error_log.conf:#$InputFileSeverity info
90-nagioslogserver_var_log_httpd_error_log.conf:$InputFilePersistStateInterval 20000
90-nagioslogserver_var_log_httpd_error_log.conf:$InputRunFileMonitor
90-nagioslogserver_var_log_httpd_error_log.conf:
90-nagioslogserver_var_log_httpd_error_log.conf:# Forward to Nagios Log Server and then discard, otherwise these messages
90-nagioslogserver_var_log_httpd_error_log.conf:# will end up in the syslog file (/var/log/messages) unless there are other
90-nagioslogserver_var_log_httpd_error_log.conf:# overriding rules.
90-nagioslogserver_var_log_httpd_error_log.conf:if $programname == 'apache_error' then @@iganagioslog.iga.local:5544
90-nagioslogserver_var_log_httpd_error_log.conf:if $programname == 'apache_error' then ~
99-nagioslogserver.conf:### Begin forwarding rule for Nagios Log Server                           NAGIOSLOGSERVER
99-nagioslogserver.conf:$WorkDirectory /var/lib/rsyslog # Where spool files will live             NAGIOSLOGSERVER
99-nagioslogserver.conf:$ActionQueueFileName nlsFwdRule0 # Unique name prefix for spool files     NAGIOSLOGSERVER
99-nagioslogserver.conf:$ActionQueueMaxDiskSpace 1g   # 1GB space limit (use as much as possible) NAGIOSLOGSERVER
99-nagioslogserver.conf:$ActionQueueSaveOnShutdown on # Save messages to disk on shutdown         NAGIOSLOGSERVER
99-nagioslogserver.conf:$ActionQueueType LinkedList   # Use asynchronous processing               NAGIOSLOGSERVER
99-nagioslogserver.conf:$ActionResumeRetryCount -1    # Infinite retries if host is down          NAGIOSLOGSERVER
99-nagioslogserver.conf:# Remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional       NAGIOSLOGSERVER
99-nagioslogserver.conf:*.* @@iganagioslog.iga.local:5544                                               # NAGIOSLOGSERVER
99-nagioslogserver.conf:### End of Nagios Log Server forwarding rule                              NAGIOSLOGSERVER
listen.conf:$SystemLogSocketName /run/systemd/journal/syslog

And no connections to the NLS port 5514

Code: Select all

[root@igarh7pubapi rsyslog.d]# netstat -an | grep 5514
[root@igarh7pubapi rsyslog.d]# netstat -an | grep 5544
tcp        0      0 192.168.40.158:58448    10.100.30.9:5544        ESTABLISHED
tcp        0      0 192.168.40.158:56234    10.100.30.9:5544        ESTABLISHED
tcp        0      0 192.168.40.158:58846    10.100.30.9:5544        ESTABLISHED
tcp        0      0 192.168.40.158:58492    10.100.30.9:5544        ESTABLISHED

[GhostRider2110]

Just noticed something else strange. Even with the "stop" statement in the .conf file for those logs, they are also getting sent to /var/log/messages. That would explain why they are still getting there as type syslog.

Guess I should have put more background info:
Client running RHEL 7.3
NSL is CentOS release 6.7 (Final)

[mcapra]

So that explains the syslog typed messages, but are you getting api_access typed messages into NLS at all?

Even if rsyslog is tossing them into /var/log/messages for some reason, the rule should still be shipping the file's contents to the destination on 5514.

You could try adding a udp rule to your Logstash input, though rsyslog typically uses tcp:

Code: Select all

udp {
    type => 'api_access'
    port => 5514
}

You might also try changing the severity of the message to be debug. On line 12 of 90-nagioslogserver_opt_in1_public_api_internal_logs_access.log.conf:

Code: Select all

$InputFileSeverity debug

Or, as a slightly hacky debug fix, you could try changing the filter to run on the host field:

Code: Select all

if [host] == '192.168.40.158' {
  grok {
       match => [ "message" , "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{HOSTNAME:loghost} %{DATA:syslog_program}: %{TIMESTAMP_ISO8601},%{USERNAME},%{DATA:rawrequest},%{BASE10NUM:response},%{NUMBER:bytes}" ]
   }
}

Is this log being constantly written?

[GhostRider2110]

So that explains the syslog typed messages, but are you getting api_access typed messages into NLS at all?

No, still nothing is getting typed as api_access

Even if rsyslog is tossing them into /var/log/messages for some reason, the rule should still be shipping the file's contents to the destination on 5514.

That is what I thought, and I do have the apache_access log going to port 5581 and that one works fine.

You could try adding a udp rule to your Logstash input, though rsyslog typically uses tcp:

Code: Select all

udp {
    type => 'api_access'
    port => 5514
}

You might also try changing the severity of the message to be debug. On line 12 of 90-nagioslogserver_opt_in1_public_api_internal_logs_access.log.conf:

Code: Select all

$InputFileSeverity debug

Did both of those and no change.

Or, as a slightly hacky debug fix, you could try changing the filter to run on the host field:

Code: Select all

if [host] == '192.168.40.158' {
  grok {
       match => [ "message" , "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{HOSTNAME:loghost} %{DATA:syslog_program}: %{TIMESTAMP_ISO8601},%{USERNAME},%{DATA:rawrequest},%{BASE10NUM:response},%{NUMBER:bytes}" ]
   }
}

Ok, that did break up the message mostly like I want it. syslog_program and syslog_pri now have double entries. I guess since they are getting parsed as both syslog type and with the host

api-002.png

Is this log being constantly written?

For the most part yes. Might I have to restart the process to allow everything to take affect?

[GhostRider2110]

Was able to fix the user, repeated syslog_pri and syslog_program in the filter.
Moved it back to use port 5544, but it is still also being sent to /var/log/messages which is strange.

Close, but I am still baffled why I can get it to use the 5514 port and retype this program?

[GhostRider2110]

OK, even stranger.

That app also produces two other logs, an info log and a debug log along with the access log. So I moved the debug log to port 5545 and info to 5546 and added 4 more inputs to type those based on port. That is working, BUT the access log is still coming in via 5544 and being typed as syslog.

Code: Select all

[root@igarh7pubapi rsyslog.d]# netstat -an | grep 5545
tcp        0      0 192.168.40.158:51232    10.100.30.9:5545        ESTABLISHED
[root@igarh7pubapi rsyslog.d]# netstat -an | grep 5546
tcp        0      0 192.168.40.158:48988    10.100.30.9:5546        ESTABLISHED

I have since moved the api_access log to 5547 and still no go... Weird???

[mcapra]

We can at least say it's on the rsyslog end of things for sure now. You might try altering the 90-nagioslogserver_opt_in1_public_api_internal_logs_access.log.conf config file to use capitals for the $programname field like so:

Code: Select all

if $programname == 'API_ACCESS' then @@iganagioslog.iga.local:5514
if $programname == 'API_ACCESS' then ~

And restarting rsyslog to apply the changes.

[GhostRider2110]

THANKS!...

I had already change the case a couple ways to see if that did anything. BUT that made me look closer at the rest of the conf file... and there was an : in the $InputFileTag api_access.

Removed that, restarted rsyslog and all is working.

Call this fixed!!! THANKS