Modification of current config to include NLS

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
Locked
GhostRider2110
Posts: 193
Joined: Thu Oct 30, 2014 8:04 am
Location: Indiana
Contact:
Contact GhostRider2110

Re: Modification of current config to include NLS

Post by GhostRider2110 »

But it seems that a restart of rsyslog, will then start the process over and add another time stamp to the file. The Second restart I did, was without any other change to the conf file.

The steps I did:
1) Removed all old log files from /var/opt/lrms/log directory
2) cp /dev/null /var/opt/lrms/log/jupiter.log
3) systemctl restart rsyslog

wait a while, log file looks correct, and log being sent to NLS properly as well.

4) systemctl restart rsyslog

After the second restart, it looks like the file /var/opt/lrms/log/jupiter is now being resent. And a second time stamp is being placed on each entry. If I trigger the app to do something that will log, then that entry comes in, and looks OK, but then the log keeps being sent entries from what I can only think is the re-read of the file. If I restart rsyslog again, then another time stamp entry is added and it again seems to be reading the file from the start.
Top
avandemore
Posts: 1597
Joined: Tue Sep 27, 2016 4:57 pm

Re: Modification of current config to include NLS

Post by avandemore »

Is $WorkDirectorydefined and pointed to a writable location which doesn't overlap with another dynamically generated file?

You can try statically assigning one: $InputFileStateFile /your/state/file
Previous Nagios employee
Top
GhostRider2110
Posts: 193
Joined: Thu Oct 30, 2014 8:04 am
Location: Indiana
Contact:
Contact GhostRider2110

Re: Modification of current config to include NLS

Post by GhostRider2110 »

avandemore wrote:Is $WorkDirectorydefined and pointed to a writable location which doesn't overlap with another dynamically generated file?

You can try statically assigning one: $InputFileStateFile /your/state/file
Yes $WorkDirectory is defined and writable and the filename does not overlap any other file.

I think I'm just getting a loop going.

Any msg coming in with the defined string "[ jupiter ]" is being sent to the log file /var/opt/lrms/log/jupiter.log.
Then I'm taking entries form /var/opt/lrms/log/jupiter.log and sending them to the NLS, but rsyslog sees the string agian as rsyslog is sending the entry to
the NLS and then sending it back to the log file /var/opt/lrms/log/jupiter.log?
Top
avandemore
Posts: 1597
Joined: Tue Sep 27, 2016 4:57 pm

Re: Modification of current config to include NLS

Post by avandemore »

What is the current configuration? You can use @mcapra's method earlier: https://support.nagios.com/forum/viewto ... 10#p215950
Can you also show:

Code: Select all

# service rsyslog restart
# tail -n 25 /var/log/messages
Previous Nagios employee
Top
GhostRider2110
Posts: 193
Joined: Thu Oct 30, 2014 8:04 am
Location: Indiana
Contact:
Contact GhostRider2110

Re: Modification of current config to include NLS

Post by GhostRider2110 »

I've tried all the combinations that have been posted and others I came up with and nothing seems to work 100%.

I believe we are getting into a rsyslog logic loop. Since rsyslog is looking for a msg with the "[ jupiter ]" string in it,

Code: Select all

$template JupiterFormat,"%TIMESTAMP:::date-rfc3339% %msg:::sp-if-no-1st-sp% %msg:::drop-last-lf%\n"
if ($msg contains "[ jupiter ]") then /var/opt/lrms/log/jupiter.log;JupiterFormat
#if ($msg contains "[ jupiter ]") then ~
we get into a loop when we tell rsyslog to process the "programname" with
"jupiter".

Code: Select all

$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/lib/rsyslog

# Input for import_json
$InputFileName /var/opt/lrms/log/jupiter.log
$InputFileTag jupiter:
$InputFileStateFile nls-state-var_opt_lrms_log_jupiter_log # Must be unique for each file being polled
# Uncomment the folowing line to override the default severity for messages
# from this file.
#$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor

# Forward to Nagios Log Server and then discard, otherwise these messages
# will end up in the syslog file (/var/log/messages) unless there are other
# overriding rules.
if $programname == "jupiter" then @@iganagioslog:5583
if $programname == "jupiter" then ~
When that parts gets processed, then rsyslog sees the string "[ jupiter ]" and thinks.. Oh I need to send that to the log file so it does, which in turn is then processed, needing to be sent to NSL, then Oh, a message processed with the string "[ jupiter ]".. I need to log that to the file... which then gets sent to the NLS... rinse and repeat..... forever.....

I've put some questions out on some of the rsyslog forums to see about some help there as well...

I am not by any stretch of the imagination a rsyslog expert.... I'm digging into docs as I go.

From /var/log/messages after a restart of rsyslog.

Code: Select all

Mar 28 10:04:21 igaqarep rsyslogd: [origin software="rsyslogd" swVersion="7.4.7" x-pid="30707" x-info="http://www.rsyslog.com"] exiting on signal 15.
Mar 28 10:04:21 igaqarep rsyslogd-2040: fatal error on disk queue 'action 5 queue[DA]', emergency switch to direct mode [try http://www.rsyslog.com/e/2040 ]
Mar 28 10:04:21 igaqarep rsyslogd: [origin software="rsyslogd" swVersion="7.4.7" x-pid="30888" x-info="http://www.rsyslog.com"] start
Mar 28 10:04:21 igaqarep rsyslogd-2307: warning: ~ action is deprecated, consider using the 'stop' statement instead [try http://www.rsyslog.com/e/2307 ]
Mar 28 10:04:21 igaqarep rsyslogd-2221: module 'imfile' already in this config, cannot be added
 [try http://www.rsyslog.com/e/2221 ]
Mar 28 10:04:21 igaqarep rsyslogd: rsyslogd's groupid changed to 4
Mar 28 10:04:21 igaqarep systemd[1]: Stopping System Logging Service...
Mar 28 10:04:21 igaqarep systemd[1]: Starting System Logging Service...
Mar 28 10:04:21 igaqarep systemd[1]: Started System Logging Service.

Code: Select all

$template JupiterFormat,"%TIMESTAMP:::date-rfc3339% %msg:::sp-if-no-1st-sp% %msg:::drop-last-lf%\n"
if ($msg contains "[ jupiter ]") then /var/opt/lrms/log/jupiter.log;JupiterFormat
if ($msg contains "[ jupiter ]") then ~

#$ModLoad imfile
#$InputFilePollInterval 10
#$PrivDropToGroup adm
#$WorkDirectory /var/lib/rsyslog
#
## Input for import_json
#$InputFileName /var/opt/lrms/log/jupiter.log
#$InputFileTag jupiter:
#$InputFileStateFile nls-state-var_opt_lrms_log_jupiter_log # Must be unique for each file being polled
## Uncomment the folowing line to override the default severity for messages
## from this file.
##$InputFileSeverity info
#$InputFilePersistStateInterval 20000
#$InputRunFileMonitor
#
## Forward to Nagios Log Server and then discard, otherwise these messages
## will end up in the syslog file (/var/log/messages) unless there are other
## overriding rules.
#if $programname == "jupiter" then @@iganagioslog:5583
#if $programname == "jupiter" then ~
The fatal error on disk message I get with or without the part to sent to the NLS added. I can't leave that going since the logic loop will fill up the log.. I can watch the progression now that I know how to recreate it in a way I can watch the logs.

If I clear out /var/opt/lrms/log/jupiter.log, change the conf file to this:

Code: Select all

$template JupiterFormat,"%TIMESTAMP:::date-rfc3339% %msg:::sp-if-no-1st-sp% %msg:::drop-last-lf%\n"
if ($msg contains "[ jupiter ]") then /var/opt/lrms/log/jupiter.log;JupiterFormat
#if ($msg contains "[ jupiter ]") then ~

$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/lib/rsyslog

# Input for import_json
$InputFileName /var/opt/lrms/log/jupiter.log
$InputFileTag jupiter:
$InputFileStateFile nls-state-var_opt_lrms_log_jupiter_log # Must be unique for each file being polled
# Uncomment the folowing line to override the default severity for messages
# from this file.
#$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor

# Forward to Nagios Log Server and then discard, otherwise these messages
# will end up in the syslog file (/var/log/messages) unless there are other
# overriding rules.
if $programname == "jupiter" then @@iganagioslog:5583
if $programname == "jupiter" then ~
Then thump the app to send something to the log, I will get what I expect, but to both /var/opt/lrml/log/jupiter and /var/log/messages,

/var/log/messages
Mar 28 10:03:29 igaqarep journal: - [ jupiter ] - 0.0002648830 - c3a8c281-5d27-4f91-8195-e76edff0219b - site:dev - INFO -- ============================================================ - jupiter.lib.middleware:59
Mar 28 10:03:29 igaqarep journal: - [ jupiter ] - 0.0004968643 - c3a8c281-5d27-4f91-8195-e76edff0219b - site:dev - INFO -- Received GET request. - jupiter.lib.middleware:61
Mar 28 10:03:29 igaqarep journal: - [ jupiter ] - 0.0006029606 - c3a8c281-5d27-4f91-8195-e76edff0219b - site:dev - INFO -- Path: /dev/admin/jupiter/lrms_revision/1948166/ - jupiter.lib.middleware:63
Mar 28 10:03:29 igaqarep journal: - [ jupiter ] - 0.0007328987 - c3a8c281-5d27-4f91-8195-e76edff0219b - site:dev - INFO -- User: admin - jupiter.lib.middleware:65
Mar 28 10:03:29 igaqarep journal: - [ jupiter ] - 0.0008518696 - c3a8c281-5d27-4f91-8195-e76edff0219b - site:dev - INFO -- IP: 10.100.52.117 - jupiter.lib.middleware:67
Mar 28 10:03:29 igaqarep journal: - [ jupiter ] - 0.0009419918 - c3a8c281-5d27-4f91-8195-e76edff0219b - site:dev - INFO -- User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.110 Safari/537.36 - jupiter.lib.middleware:69
Mar 28 10:03:29 igaqarep journal: - [ jupiter ] - 0.0010638237 - c3a8c281-5d27-4f91-8195-e76edff0219b - site:dev - INFO -- Parameters: <QueryDict: {}> - jupiter.lib.middleware:71
Mar 28 10:03:29 igaqarep journal: - [ jupiter ] - 0.0011548996 - c3a8c281-5d27-4f91-8195-e76edff0219b - site:dev - INFO -- ------------------------------------------------------------ - jupiter.lib.middleware:72
Mar 28 10:03:38 igaqarep journal: - [ jupiter ] - 9.0586388111 - c3a8c281-5d27-4f91-8195-e76edff0219b - site:dev - INFO -- Finished processing request - jupiter.lib.middleware:75
Mar 28 10:03:40 igaqarep journal: - [ jupiter ] - 0.0001912117 - 4172deb3-e5cd-418f-a3ac-410eecdece73 - site:dev - INFO -- ============================================================ - jupiter.lib.middleware:59
Mar 28 10:03:40 igaqarep journal: - [ jupiter ] - 0.0003471375 - 4172deb3-e5cd-418f-a3ac-410eecdece73 - site:dev - INFO -- Received GET request. - jupiter.lib.middleware:61
Mar 28 10:03:40 igaqarep journal: - [ jupiter ] - 0.0004510880 - 4172deb3-e5cd-418f-a3ac-410eecdece73 - site:dev - INFO -- Path: /dev/admin/jsi18n/ - jupiter.lib.middleware:63
Mar 28 10:03:40 igaqarep journal: - [ jupiter ] - 0.0005800724 - 4172deb3-e5cd-418f-a3ac-410eecdece73 - site:dev - INFO -- User: admin - jupiter.lib.middleware:65
Mar 28 10:03:40 igaqarep journal: - [ jupiter ] - 0.0007011890 - 4172deb3-e5cd-418f-a3ac-410eecdece73 - site:dev - INFO -- IP: 10.100.52.117 - jupiter.lib.middleware:67
Mar 28 10:03:40 igaqarep journal: - [ jupiter ] - 0.0007910728 - 4172deb3-e5cd-418f-a3ac-410eecdece73 - site:dev - INFO -- User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.110 Safari/537.36 - jupiter.lib.middleware:69
Mar 28 10:03:40 igaqarep journal: - [ jupiter ] - 0.0009860992 - 4172deb3-e5cd-418f-a3ac-410eecdece73 - site:dev - INFO -- Parameters: <QueryDict: {}> - jupiter.lib.middleware:71
Mar 28 10:03:40 igaqarep journal: - [ jupiter ] - 0.0010812283 - 4172deb3-e5cd-418f-a3ac-410eecdece73 - site:dev - INFO -- ------------------------------------------------------------ - jupiter.lib.middleware:72
Mar 28 10:03:40 igaqarep journal: - [ jupiter ] - 0.0450851917 - 4172deb3-e5cd-418f-a3ac-410eecdece73 - site:dev - INFO -- Finished processing request - jupiter.lib.middleware:75

/var/opt/lrms/log/jupiter.log
2017-03-28T10:03:29.818650-04:00 - [ jupiter ] - 0.0002648830 - c3a8c281-5d27-4f91-8195-e76edff0219b - site:dev - INFO -- ============================================================ - jupiter.lib.middleware:59
2017-03-28T10:03:29.819027-04:00 - [ jupiter ] - 0.0004968643 - c3a8c281-5d27-4f91-8195-e76edff0219b - site:dev - INFO -- Received GET request. - jupiter.lib.middleware:61
2017-03-28T10:03:29.819223-04:00 - [ jupiter ] - 0.0006029606 - c3a8c281-5d27-4f91-8195-e76edff0219b - site:dev - INFO -- Path: /dev/admin/jupiter/lrms_revision/1948166/ - jupiter.lib.middleware:63
2017-03-28T10:03:29.819485-04:00 - [ jupiter ] - 0.0007328987 - c3a8c281-5d27-4f91-8195-e76edff0219b - site:dev - INFO -- User: admin - jupiter.lib.middleware:65
2017-03-28T10:03:29.819697-04:00 - [ jupiter ] - 0.0008518696 - c3a8c281-5d27-4f91-8195-e76edff0219b - site:dev - INFO -- IP: 10.100.52.117 - jupiter.lib.middleware:67
2017-03-28T10:03:29.819948-04:00 - [ jupiter ] - 0.0009419918 - c3a8c281-5d27-4f91-8195-e76edff0219b - site:dev - INFO -- User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.110 Safari/537.36 - jupiter.lib.middleware:69
2017-03-28T10:03:29.820158-04:00 - [ jupiter ] - 0.0010638237 - c3a8c281-5d27-4f91-8195-e76edff0219b - site:dev - INFO -- Parameters: <QueryDict: {}> - jupiter.lib.middleware:71
2017-03-28T10:03:29.820362-04:00 - [ jupiter ] - 0.0011548996 - c3a8c281-5d27-4f91-8195-e76edff0219b - site:dev - INFO -- ------------------------------------------------------------ - jupiter.lib.middleware:72
2017-03-28T10:03:34.050627-04:00 2017-03-28T10:03:29.818650-04:00 - [ jupiter ] - 0.0002648830 - c3a8c281-5d27-4f91-8195-e76edff0219b - site:dev - INFO -- ============================================================ - jupiter.lib.middleware:59
2017-03-28T10:03:34.050646-04:00 2017-03-28T10:03:29.819027-04:00 - [ jupiter ] - 0.0004968643 - c3a8c281-5d27-4f91-8195-e76edff0219b - site:dev - INFO -- Received GET request. - jupiter.lib.middleware:61
2017-03-28T10:03:34.050651-04:00 2017-03-28T10:03:29.819223-04:00 - [ jupiter ] - 0.0006029606 - c3a8c281-5d27-4f91-8195-e76edff0219b - site:dev - INFO -- Path: /dev/admin/jupiter/lrms_revision/1948166/ - jupiter.lib.middleware:63
2017-03-28T10:03:34.050655-04:00 2017-03-28T10:03:29.819485-04:00 - [ jupiter ] - 0.0007328987 - c3a8c281-5d27-4f91-8195-e76edff0219b - site:dev - INFO -- User: admin - jupiter.lib.middleware:65
2017-03-28T10:03:34.050659-04:00 2017-03-28T10:03:29.819697-04:00 - [ jupiter ] - 0.0008518696 - c3a8c281-5d27-4f91-8195-e76edff0219b - site:dev - INFO -- IP: 10.100.52.117 - jupiter.lib.middleware:67
2017-03-28T10:03:34.050664-04:00 2017-03-28T10:03:29.819948-04:00 - [ jupiter ] - 0.0009419918 - c3a8c281-5d27-4f91-8195-e76edff0219b - site:dev - INFO -- User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.110 Safari/537.36 - jupiter.lib.middleware:69
2017-03-28T10:03:34.050668-04:00 2017-03-28T10:03:29.820158-04:00 - [ jupiter ] - 0.0010638237 - c3a8c281-5d27-4f91-8195-e76edff0219b - site:dev - INFO -- Parameters: <QueryDict: {}> - jupiter.lib.middleware:71
2017-03-28T10:03:34.050687-04:00 2017-03-28T10:03:29.820362-04:00 - [ jupiter ] - 0.0011548996 - c3a8c281-5d27-4f91-8195-e76edff0219b - site:dev - INFO -- ------------------------------------------------------------ - jupiter.lib.middleware:72

And it pretty much kept going like that, adding a timestamp to the each entry as they went through... Until I commented out all the info to send to NLS.. Hope this helps in explanation of what I am seeing.
Top
avandemore
Posts: 1597
Joined: Tue Sep 27, 2016 4:57 pm

Re: Modification of current config to include NLS

Post by avandemore »

Did you see my earlier question?
What is the current configuration? You can use @mcapra's method earlier: viewtopic.php?f=38&t=32999&start=10#p215950
Also please note @scottwilkerson's comments:
https://support.nagios.com/forum/viewto ... 20#p216348
https://support.nagios.com/forum/viewto ... 20#p216407
Previous Nagios employee
Top
GhostRider2110
Posts: 193
Joined: Thu Oct 30, 2014 8:04 am
Location: Indiana
Contact:
Contact GhostRider2110

Re: Modification of current config to include NLS

Post by GhostRider2110 »

avandemore wrote:Did you see my earlier question?
What is the current configuration? You can use @mcapra's method earlier: viewtopic.php?f=38&t=32999&start=10#p215950
Also please note @scottwilkerson's comments:
https://support.nagios.com/forum/viewto ... 20#p216348
https://support.nagios.com/forum/viewto ... 20#p216407
Yes https://support.nagios.com/forum/viewto ... 20#p216145

I have tried every config in the thread... Will go through them again and see what I can find out.
Top
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: Modification of current config to include NLS

Post by mcapra »

Do you know if whatever code is passing off the jupiter messages (I think you mentioned it was Python based) allows you to define a specific syslog facility to use? That might allow us to better isolate those messages.
Former Nagios employee
https://www.mcapra.com/
Top
GhostRider2110
Posts: 193
Joined: Thu Oct 30, 2014 8:04 am
Location: Indiana
Contact:
Contact GhostRider2110

Re: Modification of current config to include NLS

Post by GhostRider2110 »

I have tried these, both do the same thing, send the log to /var/log/messages and nothing is sent to NLS

Code: Select all

$template JupiterFormat,"%TIMESTAMP:::date-rfc3339% %msg:::sp-if-no-1st-sp% %msg:::drop-last-lf%\n"

#if ($msg contains "[ jupiter ]") then /var/opt/lrms/log/jupiter.log;JupiterFormat
#if ($msg contains "[ jupiter ]") then ~

$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/lib/rsyslog

# Input for import_json
$InputFileName /var/opt/lrms/log/jupiter.log
$InputFileTag jupiter:
$InputFileStateFile nls-state-var_opt_lrms_log_jupiter # Must be unique for each file being polled
# Uncomment the folowing line to override the default severity for messages
# from this file.
#$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor

# Forward to Nagios Log Server and then discard, otherwise these messages
# will end up in the syslog file (/var/log/messages) unless there are other
# overriding rules.
if $programname == "jupiter" then /var/opt/lrms/log/jupiter.log;JupiterFormat
if $programname == "jupiter" then @@iganagioslog:5583
if $programname == "jupiter" then ~

Code: Select all

$template JupiterFormat,"%TIMESTAMP:::date-rfc3339% %msg:::sp-if-no-1st-sp% %msg:::drop-last-lf%\n"
#if ($msg contains "[ jupiter ]") then /var/opt/lrms/log/jupiter.log;JupiterFormat
#if ($msg contains "[ jupiter ]") then ~

$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/lib/rsyslog

# Input for import_json
$InputFileName /var/opt/lrms/log/jupiter.log
$InputFileTag jupiter:
$InputFileStateFile nls-state-var_opt_lrms_log_jupiter # Must be unique for each file being polled
# Uncomment the folowing line to override the default severity for messages
# from this file.
#$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor

# Forward to Nagios Log Server and then discard, otherwise these messages
# will end up in the syslog file (/var/log/messages) unless there are other
# overriding rules.
if $programname == "jupiter" then @@iganagioslog:5583
if $programname == "jupiter" then /var/opt/lrms/log/jupiter.log;JupiterFormat
if $programname == "jupiter" then ~
This version will send to NLS and to /var/opt/lrms/log/jupiter.log, but will also start an endless loop.

Code: Select all

$template JupiterFormat,"%TIMESTAMP:::date-rfc3339% %msg:::sp-if-no-1st-sp% %msg:::drop-last-lf%\n"
if ($msg contains "[ jupiter ]") then /var/opt/lrms/log/jupiter.log;JupiterFormat
#if ($msg contains "[ jupiter ]") then ~

$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/lib/rsyslog

# Input for import_json
$InputFileName /var/opt/lrms/log/jupiter.log
$InputFileTag jupiter:
$InputFileStateFile nls-state-var_opt_lrms_log_jupiter # Must be unique for each file being polled
# Uncomment the folowing line to override the default severity for messages
# from this file.
#$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor

# Forward to Nagios Log Server and then discard, otherwise these messages
# will end up in the syslog file (/var/log/messages) unless there are other
# overriding rules.
if $programname == "jupiter" then /var/opt/lrms/log/jupiter.log;JupiterFormat
if $programname == "jupiter" then @@iganagioslog:5583
if $programname == "jupiter" then ~
I have changed the

Code: Select all

$InputFileStateFile
to different file, no change.
Top
GhostRider2110
Posts: 193
Joined: Thu Oct 30, 2014 8:04 am
Location: Indiana
Contact:
Contact GhostRider2110

Re: Modification of current config to include NLS

Post by GhostRider2110 »

mcapra wrote:Do you know if whatever code is passing off the jupiter messages (I think you mentioned it was Python based) allows you to define a specific syslog facility to use? That might allow us to better isolate those messages.
I'm checking on that code with the developers.
Top
Locked

Return to “Nagios Log Server”