Modification of current config to include NLS

[GhostRider2110]

So if I leave off the :JupiterFormat from the line that sends to the NLS then the format should be the raw log info?

--Mitch

[ssax]

Yes, if you remove ;JupiterFormat from the end it will send the raw logs.

[GhostRider2110]

Sorry, took me so long to get back on this.

I tried the conf example you provided and I get everything duplicated in /var/log/messages.

Other various tries started giving me:

2017-03-22T11:33:41.082484-04:00 2017-03-22T11:33:41.046828-04:00 2017-03-22T11:33:31.002047-04:00 2017-03-22T11:33:20.959401-04:00 2017-03-22T11:33:10.920558-04:00 2017-03-22T11:33:00.884004-04:00 2017-03-22T11:32:50.851367-04:00 2017-03-22T11:32:40.825301-04:00 2017-03-22T11:29:46.031027-04:00 2017-03-22T11:29:46.018131-04:00 2017-03-22T11:29:36.001018-04:00 2017-03-22T11:29:25.980880-04:00 2017-03-22T11:29:15.954798-04:00 2017-03-22T11:29:05.935377-04:00 2017-03-22T11:28:55.925467-04:00 2017-03-22T11:28:45.907382-04:00 2017-03-22T11:28:35.897488-04:00 2017-03-22T11:28:25.876323-04:00 2017-03-22T11:28:15.855158-04:00 2017-03-22T11:28:05.839367-04:00 2017-03-22T11:27:55.824125-04:00 2017-03-22T11:27:45.809280-04:00 2017-03-22T11:27:35.794695-04:00 2017-03-22T11:27:25.783573-04:00 2017-03-22T11:27:15.767790-04:00 2017-03-22T11:27:05.752419-04:00 2017-03-22T11:26:55.737355-04:00 2017-03-22T11:26:45.727166-04:00 2017-03-22T11:26:44.409603-04:00 2017-03-22T11:26:34.397356-04:00 2017-03-22T11:26:24.384377-04:00 2017-03-22T11:26:23.202772-04:00 2017-03-22T11:26:13.190824-04:00 2017-03-22T11:26:08.716811-04:00 - [ jupiter ] - 0.0006489754 - 6b1d79b9-dcb7-4c58-a4ae-3781bdef9b89 - site:dev - INFO -- User: admin - jupiter.lib.middleware:65
2017-03-22T11:33:41.082504-04:00 2017-03-22T11:33:41.046844-04:00 2017-03-22T11:33:31.002066-04:00 2017-03-22T11:33:20.959419-04:00 2017-03-22T11:33:10.920575-04:00 2017-03-22T11:33:00.884019-04:00 2017-03-22T11:32:50.851381-04:00 2017-03-22T11:32:40.825317-04:00 2017-03-22T11:29:46.031040-04:00 2017-03-22T11:29:46.018151-04:00 2017-03-22T11:29:36.001031-04:00 2017-03-22T11:29:25.980917-04:00 2017-03-22T11:29:15.954822-04:00 2017-03-22T11:29:05.935391-04:00 2017-03-22T11:28:55.925477-04:00 2017-03-22T11:28:45.907392-04:00 2017-03-22T11:28:35.897498-04:00 2017-03-22T11:28:25.876343-04:00 2017-03-22T11:28:15.855177-04:00 2017-03-22T11:28:05.839376-04:00 2017-03-22T11:27:55.824133-04:00 2017-03-22T11:27:45.809288-04:00 2017-03-22T11:27:35.794703-04:00 2017-03-22T11:27:25.783580-04:00 2017-03-22T11:27:15.767804-04:00 2017-03-22T11:27:05.752432-04:00 2017-03-22T11:26:55.737367-04:00 2017-03-22T11:26:45.727172-04:00 2017-03-22T11:26:44.409607-04:00 2017-03-22T11:26:34.397360-04:00 2017-03-22T11:26:24.384380-04:00 2017-03-22T11:26:23.202776-04:00 2017-03-22T11:26:13.190827-04:00 2017-03-22T11:26:08.717043-04:00 - [ jupiter ] - 0.0007648468 - 6b1d79b9-dcb7-4c58-a4ae-3781bdef9b89 - site:dev - INFO -- IP: 10.100.36.84 - jupiter.lib.middleware:67

Adding lots of timestamps into the log...

So I currently have the original file 100-jupiter.conf

Code: Select all

$template JupiterFormat,"%TIMESTAMP:::date-rfc3339% %msg:::sp-if-no-1st-sp% %msg:::drop-last-lf%\n"
:msg,contains,"jupiter" /var/opt/lrms/log/jupiter.log;JupiterFormat
& stop

I have also tried moving 100-jupiter.conf to 89-jupiter.conf when using part of your conf file as 90-nagioslogserver_var_opt_lrms_log_jupiter.log.conf.

When I do that I again get lots of lines like above with multiple time stamps and from the looks of it, repeating old entries into /var/opt/lrms/log/jupiter.log.

90-nagioslogserver_var_opt_lrms_log_jupiter.log.conf

Code: Select all

$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/lib/rsyslog

# Input for jupiter
$InputFileName /var/opt/lrms/log/jupiter.log
$InputFileTag jupiter:
$InputFileStateFile nls-state-var_opt_lrms_log_jupiter.log # Must be unique for each file being polled
# Uncomment the folowing line to override the default severity for messages
# from this file.
#$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor

# Forward to Nagios Log Server and then discard, otherwise these messages
# will end up in the syslog file (/var/log/messages) unless there are other
# overriding rules.
if $programname == 'jupiter' then @@iganagioslog.iga.local:5544
if $programname == 'jupiter' then ~

[mcapra]

Can you share all of the rsyslog configs:

Code: Select all

grep '' /etc/rsyslog.d/*.conf

Also the contents of this machine's main apache config file? Typically On Cent/RHEL systems, /etc/httpd/conf/httpd.conf.

[GhostRider2110]

Can you share all of the rsyslog configs:

Code: Select all

grep '' /etc/rsyslog.d/*.conf

Code: Select all

/etc/rsyslog.d/100-jupiter.conf:$template JupiterFormat,"%TIMESTAMP:::date-rfc3339% %msg:::sp-if-no-1st-sp% %msg:::drop-last-lf%\n"
/etc/rsyslog.d/100-jupiter.conf::msg,contains,"jupiter" /var/opt/lrms/log/jupiter.log;JupiterFormat
/etc/rsyslog.d/100-jupiter.conf:& ~
/etc/rsyslog.d/90-nagioslogserver_var_log_httpd_access_log.conf:$ModLoad imfile
/etc/rsyslog.d/90-nagioslogserver_var_log_httpd_access_log.conf:$InputFilePollInterval 10
/etc/rsyslog.d/90-nagioslogserver_var_log_httpd_access_log.conf:$PrivDropToGroup adm
/etc/rsyslog.d/90-nagioslogserver_var_log_httpd_access_log.conf:$WorkDirectory /var/lib/rsyslog
/etc/rsyslog.d/90-nagioslogserver_var_log_httpd_access_log.conf:
/etc/rsyslog.d/90-nagioslogserver_var_log_httpd_access_log.conf:# Input for apache_access
/etc/rsyslog.d/90-nagioslogserver_var_log_httpd_access_log.conf:$InputFileName /var/log/httpd/access_log
/etc/rsyslog.d/90-nagioslogserver_var_log_httpd_access_log.conf:$InputFileTag apache_access:
/etc/rsyslog.d/90-nagioslogserver_var_log_httpd_access_log.conf:$InputFileStateFile nls-state-var_log_httpd_access_log # Must be unique for each file being polled
/etc/rsyslog.d/90-nagioslogserver_var_log_httpd_access_log.conf:# Uncomment the folowing line to override the default severity for messages
/etc/rsyslog.d/90-nagioslogserver_var_log_httpd_access_log.conf:# from this file.
/etc/rsyslog.d/90-nagioslogserver_var_log_httpd_access_log.conf:#$InputFileSeverity info
/etc/rsyslog.d/90-nagioslogserver_var_log_httpd_access_log.conf:$InputFilePersistStateInterval 20000
/etc/rsyslog.d/90-nagioslogserver_var_log_httpd_access_log.conf:$InputRunFileMonitor
/etc/rsyslog.d/90-nagioslogserver_var_log_httpd_access_log.conf:
/etc/rsyslog.d/90-nagioslogserver_var_log_httpd_access_log.conf:# Forward to Nagios Log Server and then discard, otherwise these messages
/etc/rsyslog.d/90-nagioslogserver_var_log_httpd_access_log.conf:# will end up in the syslog file (/var/log/messages) unless there are other
/etc/rsyslog.d/90-nagioslogserver_var_log_httpd_access_log.conf:# overriding rules.
/etc/rsyslog.d/90-nagioslogserver_var_log_httpd_access_log.conf:if $programname == 'apache_access' then @@iganagioslog.iga.local:5581
/etc/rsyslog.d/90-nagioslogserver_var_log_httpd_access_log.conf:if $programname == 'apache_access' then stop
/etc/rsyslog.d/90-nagioslogserver_var_log_httpd_error_log.conf:$ModLoad imfile
/etc/rsyslog.d/90-nagioslogserver_var_log_httpd_error_log.conf:$InputFilePollInterval 10
/etc/rsyslog.d/90-nagioslogserver_var_log_httpd_error_log.conf:$PrivDropToGroup adm
/etc/rsyslog.d/90-nagioslogserver_var_log_httpd_error_log.conf:$WorkDirectory /var/lib/rsyslog
/etc/rsyslog.d/90-nagioslogserver_var_log_httpd_error_log.conf:
/etc/rsyslog.d/90-nagioslogserver_var_log_httpd_error_log.conf:# Input for apache_error
/etc/rsyslog.d/90-nagioslogserver_var_log_httpd_error_log.conf:$InputFileName /var/log/httpd/error_log
/etc/rsyslog.d/90-nagioslogserver_var_log_httpd_error_log.conf:$InputFileTag apache_error:
/etc/rsyslog.d/90-nagioslogserver_var_log_httpd_error_log.conf:$InputFileStateFile nls-state-var_log_httpd_error_log # Must be unique for each file being polled
/etc/rsyslog.d/90-nagioslogserver_var_log_httpd_error_log.conf:# Uncomment the folowing line to override the default severity for messages
/etc/rsyslog.d/90-nagioslogserver_var_log_httpd_error_log.conf:# from this file.
/etc/rsyslog.d/90-nagioslogserver_var_log_httpd_error_log.conf:#$InputFileSeverity info
/etc/rsyslog.d/90-nagioslogserver_var_log_httpd_error_log.conf:$InputFilePersistStateInterval 20000
/etc/rsyslog.d/90-nagioslogserver_var_log_httpd_error_log.conf:$InputRunFileMonitor
/etc/rsyslog.d/90-nagioslogserver_var_log_httpd_error_log.conf:
/etc/rsyslog.d/90-nagioslogserver_var_log_httpd_error_log.conf:# Forward to Nagios Log Server and then discard, otherwise these messages
/etc/rsyslog.d/90-nagioslogserver_var_log_httpd_error_log.conf:# will end up in the syslog file (/var/log/messages) unless there are other
/etc/rsyslog.d/90-nagioslogserver_var_log_httpd_error_log.conf:# overriding rules.
/etc/rsyslog.d/90-nagioslogserver_var_log_httpd_error_log.conf:if $programname == 'apache_error' then @@iganagioslog.iga.local:5582
/etc/rsyslog.d/90-nagioslogserver_var_log_httpd_error_log.conf:if $programname == 'apache_error' then stop
/etc/rsyslog.d/99-nagioslogserver.conf:### Begin forwarding rule for Nagios Log Server                           NAGIOSLOGSERVER
/etc/rsyslog.d/99-nagioslogserver.conf:$WorkDirectory /var/lib/rsyslog # Where spool files will live             NAGIOSLOGSERVER
/etc/rsyslog.d/99-nagioslogserver.conf:$ActionQueueFileName nlsFwdRule0 # Unique name prefix for spool files     NAGIOSLOGSERVER
/etc/rsyslog.d/99-nagioslogserver.conf:$ActionQueueMaxDiskSpace 1g   # 1GB space limit (use as much as possible) NAGIOSLOGSERVER
/etc/rsyslog.d/99-nagioslogserver.conf:$ActionQueueSaveOnShutdown on # Save messages to disk on shutdown         NAGIOSLOGSERVER
/etc/rsyslog.d/99-nagioslogserver.conf:$ActionQueueType LinkedList   # Use asynchronous processing               NAGIOSLOGSERVER
/etc/rsyslog.d/99-nagioslogserver.conf:$ActionResumeRetryCount -1    # Infinite retries if host is down          NAGIOSLOGSERVER
/etc/rsyslog.d/99-nagioslogserver.conf:# Remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional       NAGIOSLOGSERVER
/etc/rsyslog.d/99-nagioslogserver.conf:*.* @@iganagioslog.iga.local:5544                                               # NAGIOSLOGSERVER
/etc/rsyslog.d/99-nagioslogserver.conf:### End of Nagios Log Server forwarding rule                              NAGIOSLOGSERVER
/etc/rsyslog.d/listen.conf:$SystemLogSocketName /run/systemd/journal/syslog

Also the contents of this machine's main apache config file? Typically On Cent/RHEL systems, /etc/httpd/conf/httpd.conf.

The httpd.conf file is pretty much stock. Here is the application .conf file:

Code: Select all

[root@igaqarep conf.d]# cat lrms-datastore-dev.conf 
WSGIScriptAlias /dev /opt/lrms/datastore/dev/lrms-datastore-dev.wsgi
WSGISocketPrefix /var/run/wsgi
WSGIDaemonProcess /dev user=apache group=apache processes=15 threads=1 display-name=lrms-dev python-path=/opt/lrms/lib/python2.7/site-packages/:/opt/lrms/lib64/python2.7/site-packages/:/opt/lrms/lib/python2.6/site-packages/:/opt/lrms/lib64/python2.6/site-packages/
WSGIApplicationGroup %{GLOBAL}

<Location /dev>
  WSGIProcessGroup /dev
</Location>

Alias /dev/static/ /var/www/lrms-static/

<Directory /opt/lrms/datastore/dev>
    <Files lrms-datastore-dev.wsgi>
        Require all granted
    </Files>
</Directory>

<Directory /var/www/lrms-static/>
    Require all granted
</Directory>[/icode]

[icode]#
# This is the main Apache HTTP server configuration file.  It contains the
# configuration directives that give the server its instructions.
# See <URL:http://httpd.apache.org/docs/2.4/> for detailed information.
# In particular, see 
# <URL:http://httpd.apache.org/docs/2.4/mod/directives.html>
# for a discussion of each configuration directive.
#
# Do NOT simply read the instructions in here without understanding
# what they do.  They're here only as hints or reminders.  If you are unsure
# consult the online docs. You have been warned.  
#
# Configuration and logfile names: If the filenames you specify for many
# of the server's control files begin with "/" (or "drive:/" for Win32), the
# server will use that explicit path.  If the filenames do *not* begin
# with "/", the value of ServerRoot is prepended -- so 'log/access_log'
# with ServerRoot set to '/www' will be interpreted by the
# server as '/www/log/access_log', where as '/log/access_log' will be
# interpreted as '/log/access_log'.

#
# ServerRoot: The top of the directory tree under which the server's
# configuration, error, and log files are kept.
#
# Do not add a slash at the end of the directory path.  If you point
# ServerRoot at a non-local disk, be sure to specify a local disk on the
# Mutex directive, if file-based mutexes are used.  If you wish to share the
# same ServerRoot for multiple httpd daemons, you will need to change at
# least PidFile.
#
ServerRoot "/etc/httpd"

#
# Listen: Allows you to bind Apache to specific IP addresses and/or
# ports, instead of the default. See also the <VirtualHost>
# directive.
#
# Change this to Listen on specific IP addresses as shown below to 
# prevent Apache from glomming onto all bound IP addresses.
#
#Listen 12.34.56.78:80
Listen 80

#
# Dynamic Shared Object (DSO) Support
#
# To be able to use the functionality of a module which was built as a DSO you
# have to place corresponding `LoadModule' lines at this location so the
# directives contained in it are actually available _before_ they are used.
# Statically compiled modules (those listed by `httpd -l') do not need
# to be loaded here.
#
# Example:
# LoadModule foo_module modules/mod_foo.so
#
Include conf.modules.d/*.conf

#
# If you wish httpd to run as a different user or group, you must run
# httpd as root initially and it will switch.  
#
# User/Group: The name (or #number) of the user/group to run httpd as.
# It is usually good practice to create a dedicated user and group for
# running httpd, as with most system services.
#
User apache
Group apache

# 'Main' server configuration
#
# The directives in this section set up the values used by the 'main'
# server, which responds to any requests that aren't handled by a
# <VirtualHost> definition.  These values also provide defaults for
# any <VirtualHost> containers you may define later in the file.
#
# All of these directives may appear inside <VirtualHost> containers,
# in which case these default settings will be overridden for the
# virtual host being defined.
#

#
# ServerAdmin: Your address, where problems with the server should be
# e-mailed.  This address appears on some server-generated pages, such
# as error documents.  e.g. [email protected]
#
ServerAdmin root@localhost

#
# ServerName gives the name and port that the server uses to identify itself.
# This can often be determined automatically, but we recommend you specify
# it explicitly to prevent problems during startup.
#
# If your host doesn't have a registered DNS name, enter its IP address here.
#
#ServerName http://www.example.com:80

#
# Deny access to the entirety of your server's filesystem. You must
# explicitly permit access to web content directories in other 
# <Directory> blocks below.
#
<Directory />
    AllowOverride none
    Require all denied
</Directory>

#
# Note that from this point forward you must specifically allow
# particular features to be enabled - so if something's not working as
# you might expect, make sure that you have specifically enabled it
# below.
#

#
# DocumentRoot: The directory out of which you will serve your
# documents. By default, all requests are taken from this directory, but
# symbolic links and aliases may be used to point to other locations.
#
DocumentRoot "/var/www/html"

#
# Relax access to content within /var/www.
#
<Directory "/var/www">
    AllowOverride None
    # Allow open access:
    Require all granted
</Directory>

# Further relax access to the default document root:
<Directory "/var/www/html">
    #
    # Possible values for the Options directive are "None", "All",
    # or any combination of:
    #   Indexes Includes FollowSymLinks SymLinksifOwnerMatch ExecCGI MultiViews
    #
    # Note that "MultiViews" must be named *explicitly* --- "Options All"
    # doesn't give it to you.
    #
    # The Options directive is both complicated and important.  Please see
    # http://httpd.apache.org/docs/2.4/mod/core.html#options
    # for more information.
    #
    Options Indexes FollowSymLinks

    #
    # AllowOverride controls what directives may be placed in .htaccess files.
    # It can be "All", "None", or any combination of the keywords:
    #   Options FileInfo AuthConfig Limit
    #
    AllowOverride None

    #
    # Controls who can get stuff from this server.
    #
    Require all granted
</Directory>

#
# DirectoryIndex: sets the file that Apache will serve if a directory
# is requested.
#
<IfModule dir_module>
    DirectoryIndex index.html
</IfModule>

#
# The following lines prevent .htaccess and .htpasswd files from being 
# viewed by Web clients. 
#
<Files ".ht*">
    Require all denied
</Files>

#
# ErrorLog: The location of the error log file.
# If you do not specify an ErrorLog directive within a <VirtualHost>
# container, error messages relating to that virtual host will be
# logged here.  If you *do* define an error logfile for a <VirtualHost>
# container, that host's errors will be logged there and not here.
#
ErrorLog "logs/error_log"

#
# LogLevel: Control the number of messages logged to the error_log.
# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
#
LogLevel warn

<IfModule log_config_module>
    #
    # The following directives define some format nicknames for use with
    # a CustomLog directive (see below).
    #
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
    LogFormat "%h %l %u %t \"%r\" %>s %b" common

    <IfModule logio_module>
      # You need to enable mod_logio.c to use %I and %O
      LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
    </IfModule>

    #
    # The location and format of the access logfile (Common Logfile Format).
    # If you do not define any access logfiles within a <VirtualHost>
    # container, they will be logged here.  Contrariwise, if you *do*
    # define per-<VirtualHost> access logfiles, transactions will be
    # logged therein and *not* in this file.
    #
    #CustomLog "logs/access_log" common

    #
    # If you prefer a logfile with access, agent, and referer information
    # (Combined Logfile Format) you can use the following directive.
    #
    CustomLog "logs/access_log" combined
</IfModule>

<IfModule alias_module>
    #
    # Redirect: Allows you to tell clients about documents that used to 
    # exist in your server's namespace, but do not anymore. The client 
    # will make a new request for the document at its new location.
    # Example:
    # Redirect permanent /foo http://www.example.com/bar

    #
    # Alias: Maps web paths into filesystem paths and is used to
    # access content that does not live under the DocumentRoot.
    # Example:
    # Alias /webpath /full/filesystem/path
    #
    # If you include a trailing / on /webpath then the server will
    # require it to be present in the URL.  You will also likely
    # need to provide a <Directory> section to allow access to
    # the filesystem path.

    #
    # ScriptAlias: This controls which directories contain server scripts. 
    # ScriptAliases are essentially the same as Aliases, except that
    # documents in the target directory are treated as applications and
    # run by the server when requested rather than as documents sent to the
    # client.  The same rules about trailing "/" apply to ScriptAlias
    # directives as to Alias.
    #
    ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"

</IfModule>

#
# "/var/www/cgi-bin" should be changed to whatever your ScriptAliased
# CGI directory exists, if you have that configured.
#
<Directory "/var/www/cgi-bin">
    AllowOverride None
    Options None
    Require all granted
</Directory>

<IfModule mime_module>
    #
    # TypesConfig points to the file containing the list of mappings from
    # filename extension to MIME-type.
    #
    TypesConfig /etc/mime.types

    #
    # AddType allows you to add to or override the MIME configuration
    # file specified in TypesConfig for specific file types.
    #
    #AddType application/x-gzip .tgz
    #
    # AddEncoding allows you to have certain browsers uncompress
    # information on the fly. Note: Not all browsers support this.
    #
    #AddEncoding x-compress .Z
    #AddEncoding x-gzip .gz .tgz
    #
    # If the AddEncoding directives above are commented-out, then you
    # probably should define those extensions to indicate media types:
    #
    AddType application/x-compress .Z
    AddType application/x-gzip .gz .tgz

    #
    # AddHandler allows you to map certain file extensions to "handlers":
    # actions unrelated to filetype. These can be either built into the server
    # or added with the Action directive (see below)
    #
    # To use CGI scripts outside of ScriptAliased directories:
    # (You will also need to add "ExecCGI" to the "Options" directive.)
    #
    #AddHandler cgi-script .cgi

    # For type maps (negotiated resources):
    #AddHandler type-map var

    #
    # Filters allow you to process content before it is sent to the client.
    #
    # To parse .shtml files for server-side includes (SSI):
    # (You will also need to add "Includes" to the "Options" directive.)
    #
    AddType text/html .shtml
    AddOutputFilter INCLUDES .shtml
</IfModule>

#
# Specify a default charset for all content served; this enables
# interpretation of all content as UTF-8 by default.  To use the 
# default browser choice (ISO-8859-1), or to allow the META tags
# in HTML content to override this choice, comment out this
# directive:
#
AddDefaultCharset UTF-8

<IfModule mime_magic_module>
    #
    # The mod_mime_magic module allows the server to use various hints from the
    # contents of the file itself to determine its type.  The MIMEMagicFile
    # directive tells the module where the hint definitions are located.
    #
    MIMEMagicFile conf/magic
</IfModule>

#
# Customizable error responses come in three flavors:
# 1) plain text 2) local redirects 3) external redirects
#
# Some examples:
#ErrorDocument 500 "The server made a boo boo."
#ErrorDocument 404 /missing.html
#ErrorDocument 404 "/cgi-bin/missing_handler.pl"
#ErrorDocument 402 http://www.example.com/subscription_info.html
#

#
# EnableMMAP and EnableSendfile: On systems that support it, 
# memory-mapping or the sendfile syscall may be used to deliver
# files.  This usually improves server performance, but must
# be turned off when serving from networked-mounted 
# filesystems or if support for these functions is otherwise
# broken on your system.
# Defaults if commented: EnableMMAP On, EnableSendfile Off
#
#EnableMMAP off
EnableSendfile on

# Supplemental configuration
#
# Load config files in the "/etc/httpd/conf.d" directory, if any.
IncludeOptional conf.d/*.conf

[GhostRider2110]

EDIT*** I just thought even though this is related to this issue, I probably should have opened a new thread on this. Let me know and I will do that.

Something else I noticed. When I did get the logs sent to the NLS, mutate to replace the type is not working. In fact I had to add Inputs and move logs to individual ports to set the "type" to what I wanted for other filters to work. Everything was marked as type syslog.

Code: Select all

# 
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Fri, 24 Mar 2017 08:00:20 -0400
#

#
# Global inputs
#

input {
    tcp {
        type => 'eventlog'
        port => 3515
        codec => json {
            charset => 'CP1252'
        }
    }
    tcp {
        type => 'import_raw'
        tags => 'import_raw'
        port => 2056
    }
    tcp {
        type => 'import_json'
        tags => 'import_json'
        port => 2057
        codec => json
    }
    syslog {
        type => 'syslog'
        port => 514
    }
    syslog {
        type => 'asa'
        port => 6544
    }
    tcp {
        port => 5544
        type => 'syslog'
    }
    udp {
        port => 5544
        type => 'syslog'
    
    }
    tcp {
        type => 'apache_access'
        port => 5581
    }
    udp {
        type => 'apache_access'
        port => 5581
    }
    tcp {
        port => 5540
        type => 'Wowza'
    }
    tcp {
        type => 'infoblox'
        port => 6514
    }
    udp {
        type => 'infoblox'
        port => 6514
    }
    tcp {
        type => 'api_access'
        port => 5547
    }
    udp {
        type => 'api_access'
        port => 5547
    }
    tcp {
        type => 'api_debug'
        port => 5545
    }
    udp {
        type => 'api_debug'
        port => 5545
    }
    tcp {
        type => 'api_info'
        port => 5546
    }
    udp {
        type => 'api_info'
        port => 5546
    }
    tcp {
        type => 'apache_error'
        port => 5582
    }
    udp {
        type => 'apache_error'
        port => 5582
    }
}

#
# Local inputs
#



#
# Global filters
#

filter {
    if [program] == 'VARNISHNCSA_LOG' {
    mutate {
    replace => [ 'type', 'VARNISHNCSA_LOG' ]
    }
    }
    if [type] == 'apache_access' {
        grok {
            match => [ 'message', '%{SYSLOGBASE}']
        }
        grok {
            match => [ 'message', '%{COMBINEDAPACHELOG}']
        }
        date {
             locale => "en"
             match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z' ]
        }
        mutate {
             convert => [ 'bytes', 'integer' ]
             convert => [ 'response', 'integer' ]
             remove => [ 'timestamp' ]
        }
    }
     
    if [syslog_program] == 'apache_error' {
        grok {
            match => [ 'message', '\[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\] \[%{WORD:class}\] \[%{WORD:originator} %{IP:clientip}\] %{GREEDYDATA:errmsg}']
        }
        mutate {
            replace => [ 'type', 'apache_error' ]
        }
    }
    if [type] == 'api_access' {
      grok {
           match => [ "message" , "<%{POSINT}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{HOSTNAME:loghost} %{DATA} %{TIMESTAMP_ISO8601},%{USERNAME:user},%{DATA:rawrequest},%{BASE10NUM:response},%{NUMBER:bytes}" ]
       }
       mutate {
            convert => [ 'response', 'integer']
            convert => [ 'bytes', 'integer']
        }
    }
    if [type] == "syslog" {
       grok {
           match => [ "message" , "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" ]
       }
    }
    
    if [type] == "Wowza" {
       grok {
           match => [ message,"%{DATE:date}\t%{TIME:time}\t%{DATA:tz}\t%{DATA:x-event}\t%{DATA:x-category}\t%{DATA:x-severity}\t%{NUMBER:x-status:integer}\t%{DATA:x-ctx}\t%{DATA:x-comment}\t%{DATA:x-vhost}\t%{DATA:x-app}\t%{DATA:x-appinst}\t%{DATA:x-duration:float}\t%{DATA:s-ip}\t%{DATA:s-port:integer}\t%{DATA:s-uri}\t%{DATA:c-ip}\t%{DATA:c-proto}\t%{DATA:c-referrer}\t%{DATA:c-user-agent}\t%{DATA:client-id}\t%{DATA:cs-bytes:float}\t%{DATA:sc-bytes:float}\t%{DATA:x-stream-id}\t%{DATA:x-spos:float}\t%{DATA:cs-stream-bytes:float}\t%{DATA:sc-stream-bytes:float}\t%{DATA:x-sname}\t%{DATA:x-sname-query}\t%{DATA:x-file-name}\t%{DATA:x-file-ext}\t%{DATA:x-file-size:float}\t%{DATA:x-file-length:float}\t%{DATA:x-suri}\t%{DATA:x-suri-stem}\t%{DATA:x-suri-query}\t%{DATA:cs-uri-stem}\t%{GREEDYDATA:cs-uri-query}" ]
    
           add_field => ["logtimestamp", "%{date} %{time}" ]
           remove_field => [ "time","date" ]
         }
    }
    
    if [type] == 'asa' {
    grok{
    match => ['message', '%{SYSLOG5424PRI}%%{WORD:LogType}-%{INT:LogSeverity}-%{INT:LogMessageNumber}: Group = %{IPORHOST:Group}, Username = %{IPORHOST:username}, IP = %{IP:IPAddress}, Session disconnected. Session Type: %{WORD:SessionType}, Duration: %{CUSTOM1:DurationDays=[0-9]?}%{CUSTOM2=d? ?}%{INT:DurationHours:int}h:%{INT:DurationMinutes:int}m:%{INT:DurationSeconds:int}s, Bytes xmt: %{INT:BytesTransmitted:int}, Bytes rcv: %{INT:BytesReceived:int}, Reason: %{GREEDYDATA:Reason}']
    }
    geoip {
      source => "IPAddress"
    }
    }
    if [syslog_program] == 'TrexSyncPubRep' {
    mutate {
    replace => [ 'type', 'TrexSyncPubRep' ]
    }
    }
    if [type] == 'apache_access' {
        geoip {
            source => 'clientip'
        }
    }
    if [program] == 'TrexSyncRep' {
    mutate {
    replace => [ 'type', 'TrexSyncRep' ]
    }
    }
    if [program] == 'Jupiter_log' {
    mutate {
    replace => [ 'type', 'Jupiter' ]
    }
    }
    if [program] == 'diablo_in1_video_management' {
    mutate {
    replace => [ 'type', 'diablo' ]
    }
    }
    if [program] == 'PUB_API_ACCESS' {
    mutate {
    replace => [ 'type', 'APIaccess' ]
    }
    }
    if [syslog_program] == 'sudo' {
    mutate {
    replace => [ 'type', 'sudo' ]
    }
    }
    if [program] == 'opt_lrms_logs_cmgopher' {
    mutate {
    replace => [ 'type', 'CMGopher_LRMS' ]
    }
    }
    if [program] == 'lrms_user_mgmt' {
    mutate {
    replace => [ 'type', 'User_Mgmt_LRMS' ]
    }
    }
    if [program] == 'opt_lrms_logs_uam' {
    mutate {
    replace => [ 'type', 'UAMGopher_LRMS' ]
    }
    }
    if [syslog_program] == 'cm_log' {
    mutate {
    replace => [ 'type', 'CM_LOG' ]
    }
    }
    if [program] == 'Epsy_log' {
    mutate {
    replace => [ 'type', 'Epsy_log' ]
    }
    }
    if [program] == 'Wowzastream_access' {
    mutate {
    replace => [ 'type', 'wowzastream' ]
    }
    }
    if [program] == 'Wowzastream_error' {
    mutate {
    replace => [ 'type', 'wowzastream' ]
    }
    }
    if [program] == 'lrms_logs' {
    mutate {
    replace => [ 'type', 'LRMS_LOG' ]
    }
    }
    if [program] == 'ca_logs' {
    mutate {
    replace => [ 'type', 'CA_LOG' ]
    }
    }
    if [program] == 'datacleanup_logs' {
    mutate {
    replace => [ 'type', 'DATACLEANUP_LOG' ]
    }
    }
    if [program] == 'draftingrequest_logs' {
    mutate {
    replace => [ 'type', 'DRAFTINGREQUEST_LOG' ]
    }
    }
    if [syslog_program] == 'cmgopher_log' {
    mutate {
    replace => [ 'type', 'CMGHPHER_LOG' ]
    }
    }
    if [program] == 'lm_log' {
    mutate {
    replace => [ 'type', 'LM_LOG' ]
    }
    }
    if [syslog_program] == 'sshd' {
    mutate {
    replace => [ 'type', 'sshd' ]
    }
    }
    if [type] == 'api_debug' {
      grok {
           match => [ "message" , "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{HOSTNAME:loghost} %{DATA:syslog_program}: %{DATA},%{BASE10NUM:response} %{GREEDYDATA:Info}" ]
       }
    }
    if [type] == 'api_info' {
      grok {
           match => [ "message" , "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{HOSTNAME:loghost} %{DATA:syslog_program}: %{DATA},%{BASE10NUM:response} %{GREEDYDATA:Info}" ]
       }
    }
}

#
# Local filters
#



#
# Global outputs
#



#
# Local outputs
#

[mcapra]

Just to clarify since this thread is a bit old, you're attempting to create a special rsyslog configuration for your Jupiter logs correct?

Would it be possible for me to get a fresh sample of these logs so I can test out some options? Can you also tell me the rsyslog version running on this machine?

When I did get the logs sent to the NLS, mutate to replace the type is not working.

The syntax on your mutate looks a bit off. See if using { } instead of [ ] makes a difference. Like so:

Code: Select all

mutate {
    replace => { "message" => "%{source_host}: My new message" }
  }

[GhostRider2110]

Sure.. Not a problem...
Server RHEL 7
[root@igaqarep rsyslog.d]# rsyslogd -v
rsyslogd 7.4.7, compiled with:
FEATURE_REGEXP: Yes
FEATURE_LARGEFILE: No
GSSAPI Kerberos 5 support: Yes
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
Runtime Instrumentation (slow code): No
uuid support: Yes

See http://www.rsyslog.com for more information.

Here is the default setup from the app. It uses the 100-jupiter.conf file in /etc/rsyslog.d/ directory which logs to /var/opt/lrms/log/jupiter.log

100-jupiter.conf

I want to keep the log sending to /var/opt/lrms/log/jupiter.log, but also send to our NLS. The app is python, and for some reason is the only one logging through rsyslog using the keyword in the message of "jupiter". The rest of the app's log direct to disk via python so sending those have been simple.

jupiter.log

The log format is Timestamp, Identifier, execution time, process identifier, site ID, Log level, Message

Thanks
Mitch

[GhostRider2110]

The syntax on your mutate looks a bit off. See if using { } instead of [ ] makes a difference. Like so:

Code: Select all

mutate {
    replace => { "message" => "%{source_host}: My new message" }
  }

I tried this and the verify failed:

Code: Select all

Error: Expected one of #, => at line 289, column 24 (byte 7211) after filter
 [syslog_program] == 'sshd' {
    mutate {
    replace => { 'type'

[mcapra]

Hmm, I had this working with the following rsyslog rule:

Code: Select all

$template JupiterFormat,"%TIMESTAMP:::date-rfc3339% %msg:::sp-if-no-1st-sp% %msg:::drop-last-lf%\n"
if ($msg contains "jupiter") then /var/opt/lrms/log/jupiter.log;JupiterFormat
if ($msg contains "jupiter") then ~

$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/lib/rsyslog

# Input for import_json
$InputFileName /var/opt/lrms/log/jupiter.log
$InputFileTag jupiter:
$InputFileStateFile nls-state-var_opt_lrms_log_jupiter_log # Must be unique for each file being polled
# Uncomment the folowing line to override the default severity for messages
# from this file.
#$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor

# Forward to Nagios Log Server and then discard, otherwise these messages
# will end up in the syslog file (/var/log/messages) unless there are other
# overriding rules.
if $programname == "jupiter" then @@192.168.67.4:2090
if $programname == "jupiter" then ~

But then it mysteriously stopped working for no reason on my lab machine. Can you give it a try and see if it produces the desired results?