Page 1 of 2

question about moving data

Posted: Wed Feb 22, 2017 3:30 pm
by benhank
I have a project. YAAYY!
I have to set up the elastics elk stack [complete]
set up grafana [complete]
Have grafana aaccess the elasticsearch data stored locally and make pretty graphs[incomplete]
somehow get the data already collected in NLS to the elk stack [incomplete]

The purpose of the project is to highlight the differences between grafana and kibana, as well as showcase the features that may be included in NLS when you guys roll out the upgrades mentioned in the NLS road map
=======================================================
So far I have created a filter as detailed here:

Code: Select all

https://support.nagios.com/forum/viewtopic.php?f=38&t=42225
Which worked a little too well lol.
It pushed my NLS data to the new server, but i must have configured something wrong because I never found the data, and now I have a 24 hr gap in my NLS.
SO, what I am asking is how do I safely either configure NLS to send a COPY of its incoming logs to the ELK stack, or how do I move a copy of my NLS data to the elk server? Thanks guys!
BTW, working with elastics elk stack has reminded my why I was crying (with joy and relief) like chewbacca when I found out you guys had created NLS. NLS is SOOOO much easier to use.

Re: question about moving data

Posted: Wed Feb 22, 2017 4:42 pm
by mcapra
The lazy way to integrate grafana would be to configure the Elasticsearch settings on your Grafana machine to broadcast the HTTP API on it's LAN address. You can do this (on the Grafana machine) in /etc/elasticsearch/elasticsearch.yml by changing the http.host setting to this:

Code: Select all

http.host: localhost,GRAFANA_MACHINE_ADDRESS_HERE
And restarting the Elasticsearch service. That *should* allow you to point Nagios Log Server at the Grafana box with the following output configured on your Nagios Log Server environment:

Code: Select all

elasticsearch {
        host => 'GRAFANA_MACHINE_ADDRESS_HERE'
        protocol => 'http'
    }
2017_02_22_15_41_05_Instance_Configuration_Nagios_Log_Server.png
2017_02_22_15_41_30_Instance_Configuration_Nagios_Log_Server.png
That all worked just fine when I tried it.

Re: question about moving data

Posted: Wed Feb 22, 2017 4:54 pm
by benhank
MAN!That was FAST! whooOOOO!
I have grafana and the elk stack running on the same machine. NLS is the only thing on a separate server.
so this will allow me to send incoming data from NLS AND keep a copy of the data on my NLS server?
also

Code: Select all

2017_02_22_15_41_30_Instance_Configuration_Nagios_Log_Server.png (14.03 KiB) Not viewed yet
I think you were trying to show me another image. Just making sure I don't miss anything =D

Re: question about moving data

Posted: Wed Feb 22, 2017 4:57 pm
by mcapra
Should only be 2 images, probably just the forums being silly.
benhank wrote:so this will allow me to send incoming data from NLS AND keep a copy of the data on my NLS server?
That's the idea :) It worked that way on my lab machine. NLS has a default output rule that you can't see in the GUI which handles transferring events to the local Elasticsearch database.

Re: question about moving data

Posted: Wed Feb 22, 2017 5:03 pm
by benhank
THANKS MAN Ill let you know how it went!

Re: question about moving data

Posted: Wed Feb 22, 2017 5:58 pm
by mcapra
Sure thing! Feel free to update the thread if something breaks or doesn't work right.

Re: question about moving data

Posted: Thu Feb 23, 2017 9:42 am
by benhank
well can you show me how to get the data that NLS sent to the elk stack back?

Re: question about moving data

Posted: Thu Feb 23, 2017 2:53 pm
by mcapra
That is much trickier and will depend heavily on the Elasticsearch version the remote machine is using among other things.

You might be able to target the export application I made at your Grafana setup, but I have no idea what that machine is using for it's indices/mapping/etc. That would (hopefully) at least get the data exported as raw JSON which you could probably feed into Nagios Log Server using a file input with the JSON codec.

Re: question about moving data

Posted: Fri Feb 24, 2017 10:58 am
by benhank
thanks man, but I accidentally deleted the data. doh!

Re: question about moving data

Posted: Fri Feb 24, 2017 11:43 am
by dwhitfield
@benhank, I'm sorry to hear that!

I'm guessing that means we can lock this up?