rsyslog not forwarding all messages NLS

[dworthcsl]

Hi,

I was going through and verify information in /var/log/messages and noticed that not all of the information in messages is making it into the NLS. The source server is RHEL 6.7 and it is running rsyslog7-7.4.10-3.el6_6.x86_64. NLS is running on RHEL 7.2 and NLS is ver 1.4.0.

I am still debugging, but I am not sure if we are missing data from other files as well. Here is the rsyslog config:

Code: Select all

# rsyslog configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imklog   # provides kernel logging support (previously done by rklogd)
#$ModLoad immark  # provides --MARK-- message capability

# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514

# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514


#### GLOBAL DIRECTIVES ####

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog


# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 :omusrmsg:*

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log


# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/lib/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList   # run asynchronously
#$ActionResumeRetryCount -1    # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###

Here is the config to forward to the NLS:

Code: Select all

### Begin forwarding rule for Nagios Log Server                           NAGIOSLOGSERVER
$WorkDirectory /var/lib/rsyslog # Where spool files will live             NAGIOSLOGSERVER
$ActionQueueFileName nlsFwdRule0 # Unique name prefix for spool files     NAGIOSLOGSERVER
$ActionQueueMaxDiskSpace 1g   # 1GB space limit (use as much as possible) NAGIOSLOGSERVER
$ActionQueueSaveOnShutdown on # Save messages to disk on shutdown         NAGIOSLOGSERVER
$ActionQueueType LinkedList   # Use asynchronous processing               NAGIOSLOGSERVER
$ActionResumeRetryCount -1    # Infinite retries if host is down          NAGIOSLOGSERVER
# Remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional       NAGIOSLOGSERVER
*.* @@nlsserver1:5544                                               # NAGIOSLOGSERVER
### End of Nagios Log Server forwarding rule                              NAGIOSLOGSERVER

We are also sending some Oracle logs. So I am not sure that is effecting things. Here is one of the configs:

Code: Select all

$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/lib/rsyslog

# Input for Oracle_Audit
$InputFileName /oracle/audit_trail/*.csv
$InputFileTag Oracle_Audit:
$InputFileStateFile nls-state-oracle_audit_trail_*.csv # Must be unique for each file being 
polled
# Uncomment the folowing line to override the default severity for messages
# from this file.
#$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor

# Forward to Nagios Log Server and then discard, otherwise these messages
# will end up in the syslog file (/var/log/messages) unless there are other
# overriding rules.
if $programname == 'Oracle_Audit' then @@nlsserver1:5544
if $programname == 'Oracle_Audit' then ~

Please let me know if you need any additional information.

Thanks,
David

[jolson]

What kind of information is missing specifically? Oracle Logs or system information (the type that would show up in /var/log/messages)?

[dworthcsl]

From what I am seeing, it is getting some data from messages as well as the cron log. There was some data in messages that is not coming through. Also, secure log is not being forward to NLS.

I also removed all the conf files in rsyslog.d except for the default syslog config. Here is an example of what is making it in.

Code: Select all

14:51  INFO    OGG-01407  Oracle GoldenGate Delivery for Oracle, r1epr.prm:  Setting current schema for DDL operation to [SYS].
Jan  5 10:14:51 epdbl-p001s0 Oracle GoldenGate Delivery for Oracle[25972]: 2016-01-05 10:14:51  INFO    OGG-01408  Oracle GoldenGate Delivery for Oracle, r1epr.prm:  Restoring current schema for DDL operation to [ggreplicat].
Jan  5 10:15:01 epdbl-p001s0 Oracle GoldenGate Command Interpreter for Oracle[110039]: 2016-01-05 10:15:01  INFO    OGG-00987  Oracle GoldenGate Command Interpreter for Oracle:  GGSCI command (oracle): info r1epr.

Here is an example of what is not making it into NLS from messages.

Code: Select all

Jan  3 03:31:19 epdbl-p001s0 smhMonitor[106317]: An unspecified action was performed on /opt/hp/hpsmh/data/htdocs/hmanics
Jan  3 03:31:19 epdbl-p001s0 smhMonitor[106317]: An unspecified action was performed on /opt/hp/hpsmh/data/htdocs/hmanics/#001

I also did a global search on ssh and I am getting no hits. This looks to be a global issue and we have 61 hosts sending data. The examples above were from a single server.

[dworthcsl]

Some further debug and it looks like data started to drop after 12/31/2015. I ran a query with ssh and the smh message posted above and they both showed up. After 12/31, nothing.

I also noticed that under fields, logsource is not available. When I clicked all fields, I got the following message.

Code: Select all

Note These fields have been
extracted from your mapping.
Not all fields may be available
in your source document. 

[jolson]

I'm interested in performing a remote session for this issue, because throughout the session we can decide what information is important to you and what is not. It'd be easier to do this live than go back and forth here, I think. Fire me an email at [email protected] and I'd be happy to perform a remote session with you. Thanks!

Best,


Jesse