Pull logs from a Node

[lukedevon]

Hi

Is it possible to pull logs from a remote node to Nagios log server periodically? I really don't wanna store logs in the Nagios log server before processing, which means, my requirement is, directly inject logs to Logstash/Nagios log server.

I have seen some articles that they have done using rsync and SSH. But in that case, they are storing the logs before processing.

Thank you
Luke.

[scottwilkerson]

In Log Server if you go to
Add Log Source -> Import From File

You can see a shipper.py script and some examples of how to use it

This you can run on demand whenever you like.

[cdienger]

Most of the inputs available(https://www.elastic.co/guide/en/logstas ... ugins.html) listen for incoming data and will parse it as soon as it comes in and NLS doesn't really store the data before processing. That said, a few of the plugins can pull data - see rabbitmqand http_poller examples in the link provided.

[lukedevon]

Hi

Thank you for all the valuable responses.

Just another question;

I do not have rights to access some remote nodes, but particular system admin able to send me the relevant logs to my Nagios log server. They keep sending those application logs to NLS /mnt/path, periodically. (via a crone job)

How can I input these logs to NLS directly as when receiving the logs to NLS? can I use the same rabbitmq or http_poller?

your advice is highly appreciated.

Thank you
Luke.

[cdienger]

Given the new info, rabbitmq and http_poller probably are not what you'd want for this. If possible, I would see if the remote admin would be able to set up syslog to send logs to NLS. This is covered in NLS if you point your browser to http://your_nls_ip/nagioslogserver/configure/source/linux-files.

Otherwise, I would probably go with the shipper.py option that @scottwilkersonpointed out.

[lukedevon]

Hi

Remote servers having some restrictions as the systems are live and mission critical platforms. Hence admin unable to set up syslog to send logs to NLS. However, they are scp the logs to NLS directly every 10mins. Files are in txt format. (something.txt)

host --> NLS
log receiving path --> /mnt/remote-logs/
file names --> app_critical.txt , dev_north.txt

How can I input these logs to NLS? can I use shipper.py for the same? or may I know what is the baes way?

Thank you
Luke

[mcapra]

However, they are scp the logs to NLS directly every 10mins.

That's ... inefficient to say the least. Most native syslog agents/daemons have been able to ship to remote syslog servers for over a decade now and they'll do a much better job than scp ever could.

shipper.py on a cron job could work. Otherwise, the cleaner option would be to configure the Nagios Log Server's own rsyslog daemon to reap those files for you and stuff them into the Nagios Log Server syslog input.

[scottwilkerson]

I agree with @mcapra that the shipper.py would be way more efficient, but there is a way for have Nagios Log Server Look at files in the path on the server like you suggested and that is with the file input plugin
https://www.elastic.co/guide/en/logstas ... -file.html

At the most basic you would add the following input, but it would likely need to be customized so it knows when the files get overwritten

Code: Select all

file {
    path => "/mnt/remote-logs/*.txt"
}