Nagios Support Forum

I didn't use syslog because of the multiline format Oracle uses. Also, we wanted to preserve the entire message and format instead of breaking each line into a separate message. I didn't know how to do that without a custom script.

Even if I didn't choose the most efficient method, it still doesn't explain the difference in the number loaded for the same data on the two different machines. I know the ELK stack can handle a tremendous amount of data per second, but I'm wondering if the Logstash HTTP plugin somehow got overloaded and lost messages. Anyone have experience with that plugin?

I've had luck using logstash forwarder/lumberjack for multiline. That's an agent you'd have to install on your oracle server, though. Logstash can take a beating, I'd be very surprised if it was actually a bottleneck. Could you try rate limiting the plugin and see if that's the actual problem?

hsmith wrote:Could you try rate limiting the plugin and see if that's the actual problem?

I was thinking of that too. I'll try it and report back.

How did the testing go? We just got back in for the weekend.

99.97% success!
I put a 50ms delay between each HTTP POST, and I got all but 5 entries out of 22K+. The remaining 5 could be parse errors; it would be very hard to find them to see what happened.

I was thinking about my earlier example where the two loads of the same data resulted in different numbers received. I sent one batch while working at home, so the link was a little slower. That one didn't lose as much data as the one I sent over the LAN. So that supports the idea of load rate being limited.

I guess I'm satisfied at this point. The delay seems a little kludgey, but if it works I can live with that.

Is it alright if we lock this thread and mark the issue as resolved?

We can certainly leave this issue open if you would like to do additional testing.

Yes, consider it resolved. Thanks!