Modification of current config to include NLS

GhostRider2110 · Post by **GhostRider2110** » Thu May 21, 2015 9:05 am

Forgive me if this is really a basic question. I'm still going though documentation on NLS and rsyslog.

I have a current config file in rsyslog.d that contains:

Code: Select all

$template JupiterFormat,"%TIMESTAMP:::date-rfc3339% %msg:::sp-if-no-1st-sp% %msg:::drop-last-lf%\n"
:msg,contains,"jupiter" /var/opt/lrms/log/jupiter.log;JupiterFormat
& ~

I want to just add what would be needed to also send the log to NLS. Can I do this within one file? I know I can create another to send the logs to NLS and order them properly but if can be accomplished by combining them, I think it would be better. Any help/advice greatly appreciated.

See-ya
Mitch

jolson · Post by **jolson** » Thu May 21, 2015 9:28 am

Can I do this within one file? I know I can create another to send the logs to NLS and order them properly but if can be accomplished by combining them, I think it would be better.

Is that the full config contained in rsyslog.d? If you have a more full version, I'd like to see it.

I don't see why you wouldn't be able to do this in one file. You can use the following as a template for adding a file to be sent to NLS:

Code: Select all

[root@localhost ~]# cat /etc/rsyslog.d/90-nagioslogserver_root_.mysql_history.conf
$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/lib/rsyslog

# Input for sometag
$InputFileName /root/.mysql_history
$InputFileTag sometag:
$InputFileStateFile nls-state-root_.mysql_history # Must be unique for each file being polled
# Uncomment the folowing line to override the default severity for messages
# from this file.
#$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor

# Forward to Nagios Log Server and then discard, otherwise these messages
# will end up in the syslog file (/var/log/messages) unless there are other
# overriding rules.
if $programname == 'sometag' then @@192.168.x.x:5544
if $programname == 'sometag' then ~

GhostRider2110 · Post by **GhostRider2110** » Thu May 21, 2015 9:52 am

That is the complete file called 85-jupiter.conf. Do you want to see the rsyslog.conf file as well?

--Mitch

GhostRider2110 · Post by **GhostRider2110** » Thu May 21, 2015 9:58 am

I have setup similar to what you posted as a template for me (Thanks) on the system to send the entries from the jupiter.log file to NLS. So I have two files, the 85-jupiter.conf then a 90-jupiter.conf. So the first on put the log entries in

/var/opt/lrms/log/jupiter.log

and I then would have the 90-jupiter.conf look at /var/opt/lrms/log/jupiter.log and send them to the NLS. The merge of the two is the desired outcome.

See-ya
Mitch

jolson · Post by **jolson** » Thu May 21, 2015 10:52 am

Could you provide a few example logs? I would like to get this working on my end - once I find a suitable configuration I can send it through an additional forum post. But first, I'd like some log lines to work with.

Thanks Mitch,

Jesse

GhostRider2110 · Post by **GhostRider2110** » Thu May 21, 2015 11:21 am

Here you go.

Code: Select all

2015-05-21T12:19:28.228420-04:00   [ jupiter ] - 0.0005371571 - 5be45952-8f20-4484-b7e4-e330be9d4c1e - site:dev -     INFO -- IP: 10.8.54.206 - jupiter.lib.middleware:67
2015-05-21T12:19:28.230524-04:00   [ jupiter ] - 0.0006361008 - 5be45952-8f20-4484-b7e4-e330be9d4c1e - site:dev -     INFO -- User Agent:  - jupiter.lib.middleware:69
2015-05-21T12:19:28.230534-04:00   [ jupiter ] - 0.0010690689 - 5be45952-8f20-4484-b7e4-e330be9d4c1e - site:dev -     INFO -- Parameters: <QueryDict: {}> - jupiter.lib.middleware:71
2015-05-21T12:19:28.230537-04:00   [ jupiter ] - 0.0011410713 - 5be45952-8f20-4484-b7e4-e330be9d4c1e - site:dev -     INFO -- ------------------------------------------------------------ - jupiter.lib.middleware:72
2015-05-21T12:19:28.230539-04:00   [ jupiter ] - 0.0025401115 - 5be45952-8f20-4484-b7e4-e330be9d4c1e - site:dev -     INFO -- Finished processing request - jupiter.lib.middleware:75
2015-05-21T12:19:33.233389-04:00   [ jupiter ] - 0.0002012253 - db1940d1-8a5f-45fb-bf40-ef36273c1ea5 - site:dev -     INFO -- ============================================================ - jupiter.lib.middleware:59
2015-05-21T12:19:33.233411-04:00   [ jupiter ] - 0.0003690720 - db1940d1-8a5f-45fb-bf40-ef36273c1ea5 - site:dev -     INFO -- Received GET request. - jupiter.lib.middleware:61
2015-05-21T12:19:33.233414-04:00   [ jupiter ] - 0.0004620552 - db1940d1-8a5f-45fb-bf40-ef36273c1ea5 - site:dev -     INFO -- Path: /dev/ - jupiter.lib.middleware:63
2015-05-21T12:19:33.233416-04:00   [ jupiter ] - 0.0005691051 - db1940d1-8a5f-45fb-bf40-ef36273c1ea5 - site:dev -     INFO -- User:  - jupiter.lib.middleware:65
2015-05-21T12:19:33.233419-04:00   [ jupiter ] - 0.0006451607 - db1940d1-8a5f-45fb-bf40-ef36273c1ea5 - site:dev -     INFO -- IP: 10.8.54.206 - jupiter.lib.middleware:67
2015-05-21T12:19:33.233421-04:00   [ jupiter ] - 0.0007171631 - db1940d1-8a5f-45fb-bf40-ef36273c1ea5 - site:dev -     INFO -- User Agent:  - jupiter.lib.middleware:69
2015-05-21T12:19:33.233424-04:00   [ jupiter ] - 0.0008020401 - db1940d1-8a5f-45fb-bf40-ef36273c1ea5 - site:dev -     INFO -- Parameters: <QueryDict: {}> - jupiter.lib.middleware:71
2015-05-21T12:19:33.233426-04:00   [ jupiter ] - 0.0008730888 - db1940d1-8a5f-45fb-bf40-ef36273c1ea5 - site:dev -     INFO -- ------------------------------------------------------------ - jupiter.lib.middleware:72
2015-05-21T12:19:33.235633-04:00   [ jupiter ] - 0.0030941963 - db1940d1-8a5f-45fb-bf40-ef36273c1ea5 - site:dev -     INFO -- Finished processing request - jupiter.lib.middleware:75
2015-05-21T12:19:38.238380-04:00   [ jupiter ] - 0.0001580715 - 58a6a91f-0873-4893-adb4-b770abbd1396 - site:dev -     INFO -- ============================================================ - jupiter.lib.middleware:59
2015-05-21T12:19:38.238404-04:00   [ jupiter ] - 0.0003039837 - 58a6a91f-0873-4893-adb4-b770abbd1396 - site:dev -     INFO -- Received GET request. - jupiter.lib.middleware:61
2015-05-21T12:19:38.238407-04:00   [ jupiter ] - 0.0003991127 - 58a6a91f-0873-4893-adb4-b770abbd1396 - site:dev -     INFO -- Path: /dev/ - jupiter.lib.middleware:63
2015-05-21T12:19:38.238410-04:00   [ jupiter ] - 0.0005221367 - 58a6a91f-0873-4893-adb4-b770abbd1396 - site:dev -     INFO -- User:  - jupiter.lib.middleware:65
2015-05-21T12:19:38.238412-04:00   [ jupiter ] - 0.0006000996 - 58a6a91f-0873-4893-adb4-b770abbd1396 - site:dev -     INFO -- IP: 10.8.54.206 - jupiter.lib.middleware:67
2015-05-21T12:19:38.238415-04:00   [ jupiter ] - 0.0006749630 - 58a6a91f-0873-4893-adb4-b770abbd1396 - site:dev -     INFO -- User Agent:  - jupiter.lib.middleware:69
2015-05-21T12:19:38.240367-04:00   [ jupiter ] - 0.0007550716 - 58a6a91f-0873-4893-adb4-b770abbd1396 - site:dev -     INFO -- Parameters: <QueryDict: {}> - jupiter.lib.middleware:71
2015-05-21T12:19:38.240377-04:00   [ jupiter ] - 0.0011179447 - 58a6a91f-0873-4893-adb4-b770abbd1396 - site:dev -     INFO -- ------------------------------------------------------------ - jupiter.lib.middleware:72
2015-05-21T12:19:38.240379-04:00   [ jupiter ] - 0.0025560856 - 58a6a91f-0873-4893-adb4-b770abbd1396 - site:dev -     INFO -- Finished processing request - jupiter.lib.middleware:75
2015-05-21T12:19:43.242383-04:00   [ jupiter ] - 0.0001530647 - 5118c930-a71c-4ca7-8104-62d3058a1ebc - site:dev -     INFO -- ============================================================ - jupiter.lib.middleware:59
2015-05-21T12:19:43.244856-04:00   [ jupiter ] - 0.0005240440 - 5118c930-a71c-4ca7-8104-62d3058a1ebc - site:dev -     INFO -- Received GET request. - jupiter.lib.middleware:61
2015-05-21T12:19:43.244866-04:00   [ jupiter ] - 0.0006389618 - 5118c930-a71c-4ca7-8104-62d3058a1ebc - site:dev -     INFO -- Path: /dev/ - jupiter.lib.middleware:63
2015-05-21T12:19:43.244869-04:00   [ jupiter ] - 0.0007600784 - 5118c930-a71c-4ca7-8104-62d3058a1ebc - site:dev -     INFO -- User:  - jupiter.lib.middleware:65
2015-05-21T12:19:43.244872-04:00   [ jupiter ] - 0.0008361340 - 5118c930-a71c-4ca7-8104-62d3058a1ebc - site:dev -     INFO -- IP: 10.8.54.206 - jupiter.lib.middleware:67
2015-05-21T12:19:43.244874-04:00   [ jupiter ] - 0.0009100437 - 5118c930-a71c-4ca7-8104-62d3058a1ebc - site:dev -     INFO -- User Agent:  - jupiter.lib.middleware:69
2015-05-21T12:19:43.244877-04:00   [ jupiter ] - 0.0009901524 - 5118c930-a71c-4ca7-8104-62d3058a1ebc - site:dev -     INFO -- Parameters: <QueryDict: {}> - jupiter.lib.middleware:71
2015-05-21T12:19:43.244879-04:00   [ jupiter ] - 0.0010609627 - 5118c930-a71c-4ca7-8104-62d3058a1ebc - site:dev -     INFO -- ------------------------------------------------------------ - jupiter.lib.middleware:72
2015-05-21T12:19:43.244881-04:00   [ jupiter ] - 0.0026180744 - 5118c930-a71c-4ca7-8104-62d3058a1ebc - site:dev -     INFO -- Finished processing request - jupiter.lib.middleware:75
2015-05-21T12:19:48.247383-04:00   [ jupiter ] - 0.0001759529 - ddbfcead-ec8d-4c06-bb45-690fa9d9e451 - site:dev -     INFO -- ============================================================ - jupiter.lib.middleware:59
2015-05-21T12:19:48.247407-04:00   [ jupiter ] - 0.0002980232 - ddbfcead-ec8d-4c06-bb45-690fa9d9e451 - site:dev -     INFO -- Received GET request. - jupiter.lib.middleware:61
2015-05-21T12:19:48.247410-04:00   [ jupiter ] - 0.0004000664 - ddbfcead-ec8d-4c06-bb45-690fa9d9e451 - site:dev -     INFO -- Path: /dev/ - jupiter.lib.middleware:63
2015-05-21T12:19:48.247413-04:00   [ jupiter ] - 0.0005300045 - ddbfcead-ec8d-4c06-bb45-690fa9d9e451 - site:dev -     INFO -- User:  - jupiter.lib.middleware:65
2015-05-21T12:19:48.247415-04:00   [ jupiter ] - 0.0006060600 - ddbfcead-ec8d-4c06-bb45-690fa9d9e451 - site:dev -     INFO -- IP: 10.8.54.206 - jupiter.lib.middleware:67
2015-05-21T12:19:48.247417-04:00   [ jupiter ] - 0.0006840229 - ddbfcead-ec8d-4c06-bb45-690fa9d9e451 - site:dev -     INFO -- User Agent:  - jupiter.lib.middleware:69
2015-05-21T12:19:48.247420-04:00   [ jupiter ] - 0.0007650852 - ddbfcead-ec8d-4c06-bb45-690fa9d9e451 - site:dev -     INFO -- Parameters: <QueryDict: {}> - jupiter.lib.middleware:71
2015-05-21T12:19:48.247423-04:00   [ jupiter ] - 0.0008339882 - ddbfcead-ec8d-4c06-bb45-690fa9d9e451 - site:dev -     INFO -- ------------------------------------------------------------ - jupiter.lib.middleware:72
2015-05-21T12:19:48.249567-04:00   [ jupiter ] - 0.0029950142 - ddbfcead-ec8d-4c06-bb45-690fa9d9e451 - site:dev -     INFO -- Finished processing request - jupiter.lib.middleware:75
2015-05-21T12:19:53.280389-04:00   [ jupiter ] - 0.0001399517 - 5338152f-804f-4ca5-9e67-83fad0bdc439 - site:dev -     INFO -- ============================================================ - jupiter.lib.middleware:59
2015-05-21T12:19:53.280421-04:00   [ jupiter ] - 0.0003280640 - 5338152f-804f-4ca5-9e67-83fad0bdc439 - site:dev -     INFO -- Received GET request. - jupiter.lib.middleware:61
2015-05-21T12:19:53.280427-04:00   [ jupiter ] - 0.0004589558 - 5338152f-804f-4ca5-9e67-83fad0bdc439 - site:dev -     INFO -- Path: /dev/ - jupiter.lib.middleware:63
2015-05-21T12:19:53.280431-04:00   [ jupiter ] - 0.0005719662 - 5338152f-804f-4ca5-9e67-83fad0bdc439 - site:dev -     INFO -- User:  - jupiter.lib.middleware:65
2015-05-21T12:19:53.280436-04:00   [ jupiter ] - 0.0006499290 - 5338152f-804f-4ca5-9e67-83fad0bdc439 - site:dev -     INFO -- IP: 10.8.54.206 - jupiter.lib.middleware:67
2015-05-21T12:19:53.280440-04:00   [ jupiter ] - 0.0007240772 - 5338152f-804f-4ca5-9e67-83fad0bdc439 - site:dev -     INFO -- User Agent:  - jupiter.lib.middleware:69
2015-05-21T12:19:53.280444-04:00   [ jupiter ] - 0.0008120537 - 5338152f-804f-4ca5-9e67-83fad0bdc439 - site:dev -     INFO -- Parameters: <QueryDict: {}> - jupiter.lib.middleware:71
2015-05-21T12:19:53.280449-04:00   [ jupiter ] - 0.0008840561 - 5338152f-804f-4ca5-9e67-83fad0bdc439 - site:dev -     INFO -- ------------------------------------------------------------ - jupiter.lib.middleware:72
2015-05-21T12:19:53.282478-04:00   [ jupiter ] - 0.0028331280 - 5338152f-804f-4ca5-9e67-83fad0bdc439 - site:dev -     INFO -- Finished processing request - jupiter.lib.middleware:75

I have also setup a filter just FYI:

Code: Select all

if [program] == 'Jupiter_log' {
mutate {
replace => [ 'type', 'Jupiter' ]
}
}

This matched the input I had from using the "wizard" from NLS

jolson · Post by **jolson** » Thu May 21, 2015 12:36 pm

I've been testing this for awhile, and I have a question for you.

Is there a particular reason that you're applying the following template: $template JupiterFormat,"%TIMESTAMP:::date-rfc3339% %msg:::sp-if-no-1st-sp% %msg:::drop-last-lf%\n" to your logs before shipping them via rsyslog? I ask because it would likely be easier to ship logs from /var/opt/lrms/log/jupiter.log matching the line 'jupiter' and modify them on the logstash-side of things. The syntax for Logstash is a little less complicated, and if you needed to make changes the configuration would be more central.

I almost have this working, but I'd like to know the answer to the above before I post my solution. Thanks!

GhostRider2110 · Post by **GhostRider2110** » Thu May 28, 2015 7:17 am

Sorry for the delay, I will have to ask around, it's legacy. Will get back to you soon.

See-ya
Mitch

GhostRider2110 · Post by **GhostRider2110** » Thu May 28, 2015 2:07 pm

That is an install by the vendor for the core development team. They want the detailed timestamp. Is there a way in which the msg would be sent to NLS then set the template for the file to be written to the local log?

ssax · Post by **ssax** » Fri May 29, 2015 12:59 pm

Here is what is working for me with one file:

Code: Select all

$ModLoad imfile
$InputFilePollInterval 1
$PrivDropToGroup adm
$WorkDirectory /var/lib/rsyslog

# Input for sometag
$InputFileName /root/jupiter.log
$InputFileTag jupiter:
$InputFileStateFile nls-state-root.jupiter # Must be unique for each file being polled
# Uncomment the folowing line to override the default severity for messages
# from this file.
#$InputFileSeverity info
$InputFilePersistStateInterval 1
$InputRunFileMonitor

# Add timestamp
$template JupiterFormat,"%TIMESTAMP:::date-rfc3339% %msg:::sp-if-no-1st-sp% %msg:::drop-last-lf%\n"

# Forward to Nagios Log Server and then discard, otherwise these messages
# will end up in the syslog file (/var/log/messages) unless there are other
# overriding rules.
if $programname == 'jupiter' then @@192.168.4.194:5544;JupiterFormat
if $programname == 'jupiter' then /var/log/jupiter.log;JupiterFormat
if $programname == 'jupiter' then ~

Nagios Support Forum

Modification of current config to include NLS

Modification of current config to include NLS

Re: Modification of current config to include NLS

Re: Modification of current config to include NLS

Re: Modification of current config to include NLS

Re: Modification of current config to include NLS

Re: Modification of current config to include NLS

Re: Modification of current config to include NLS

Re: Modification of current config to include NLS

Re: Modification of current config to include NLS

Re: Modification of current config to include NLS