Page 1 of 1
Writing directly to elasticsearch db?
Posted: Thu Dec 10, 2015 9:45 am
by polarbear1
Greetings!
My devs are complaining feeding logs via nxlog is way too slow and they want to pipe logs directly into the elasticsearch db. I know this is probably way out of scope, but I want to get your opinion on a few questions on doing this.
1 - Any ideas on how to approach writing directly to the db?
2 - Would this break the db or any other NLS related functionality?
3 - If they are able to successfully write to the db, could I still use the NLS web UI for all the usual dashboards and alerts using that data?
Cheers.
Re: Writing directly to elasticsearch db?
Posted: Thu Dec 10, 2015 1:28 pm
by jolson
While not necessarily common practice, inserting data directly into ES is supported. It's a question of whether or not you want to give the devs that kind of power. It can also be rather complex.
There are many Logstash inputs that could serve a similar purpose - you can send any raw data to a tcp/udp port, for example - it wouldn't be much slower than using the Elasticsearch API to insert data directly into the database. A full list of inputs can be found here:
https://www.elastic.co/guide/en/logstas ... ugins.html
If your developers would prefer writing events directly into the Elasticsearch database, you'll need to have them read up on the proper way of doing so:
https://www.elastic.co/guide/en/elastic ... arted.html
Check out the "Data In, Data Out" section in particular - I imagine it will be very useful.
Now, you can either choose to expose port 9200 to the world - which is a very dangerous thing - or you can use our built-in API system (which I highly recommend). You can get your API key by clicking your username in Nagios Log Server.
Example API submission (using a bogus key):
Code: Select all
curl -XPUT 'http://192.168.x.x/nagioslogserver/index.php/api/backend/logstash-2015.12.10/syslog/133?token=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' -d '{
"title": "My first blog entry",
"text": "Just trying this out...",
"date": "2014/01/01"
}'
Re: Writing directly to elasticsearch db?
Posted: Tue Dec 15, 2015 10:59 am
by krobertson71
Just backing up what Olson is saying here, Dev's should not have that level of authority is a production environment. I have never worked anywhere where the Dev's could interact directly with anything in a live environment.
The API option is something, as Developers, should be able to easily handle. If not, then they are most likely just wanting to do things "Their way". I have had these types of battles myself.
Tell them API is their gateway.
Re: Writing directly to elasticsearch db?
Posted: Tue Dec 15, 2015 11:54 am
by jolson
I agree with everything krobertson71 is stating - API access is really the way to go here. Let us know if you have any further questions!