Nagios Log Server - rsyslog and JSON Formatted Log Files

Problem Description

When you are using rsyslog to send JSON formatted data to Nagios Log Server, the data is not being correctly processed. Here is a screenshot that demonstrates the problem:

As you can see, the message field shows the data is prepended with a timestamp, which is causing the import_json input to fail as it's not valid JSON data.

The solution to the problem is to update the rsyslog configuration on the sending side to ensure that only the RAW JSON string is sent.

Update rsyslog Configuration

Establish a terminal session (as the root user) to your server that is using rsyslog to send the data to your Nagios Log Server.

The default configuration file is located in /etc/rsyslog.d/ and begins with 90-nagioslogserver_ . In this example the file being updated is /etc/rsyslog.d/90-nagioslogserver_var_tmp_logs_test.log.conf.

Execute the following command to open the file in vi:

vi /etc/rsyslog.d/90-nagioslogserver_var_tmp_logs_test.log.conf

When using the vi editor, to make changes press i on the keyboard first to enter insert mode. Press Esc to exit insert mode.

The following line needs to be added, you can put it after the $InputRunFileMonitor line:

$template cleanJson,"%rawmsg%"

You need to find the line that sends the data to Nagios Log Server, it looks something like:

if $programname == 'import_json' then @@10.25.5.86:2057

It needs to be appended with ";cleanJson" :

if $programname == 'import_json' then @@10.25.5.86:2057;cleanJson

Here is the entire config file with those changes:

$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/lib/rsyslog

# Input for import_json
$InputFileName /var/tmp/logs/test.log
$InputFileTag import_json:
$InputFileStateFile nls-state-var_tmp_logs_test.log # Must be unique for each file being polled
# Uncomment the folowing line to override the default severity for messages
# from this file.
#$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor

$template cleanJson,"%rawmsg%"

# Forward to Nagios Log Server and then discard, otherwise these messages
# will end up in the syslog file (/var/log/messages) unless there are other
# overriding rules.
if $programname == 'import_json' then @@10.25.5.86:2057;cleanJson
if $programname == 'import_json' then ~

When you have finished, save the changes in vi by typing:

:wq

and press Enter.

After making those changes you need to restart the rsyslog service using one of the commands below:

RHEL 7 + | CentOS 7 +

systemctl restart rsyslog.service

Wait for Nagios Log Server to receive new log data and it should be correctly processed. Here is a screenshot showing this:

You can see that the JSON data has been correctly processed into fields. You'll also notice that the message field is no longer present.

Final Thoughts

For any support related questions please visit the Nagios Support Forums at:

http://support.nagios.com/forum/