Some log entries missing after loading very old logs

[burkm]

I didn't use syslog because of the multiline format Oracle uses. Also, we wanted to preserve the entire message and format instead of breaking each line into a separate message. I didn't know how to do that without a custom script.

Even if I didn't choose the most efficient method, it still doesn't explain the difference in the number loaded for the same data on the two different machines. I know the ELK stack can handle a tremendous amount of data per second, but I'm wondering if the Logstash HTTP plugin somehow got overloaded and lost messages. Anyone have experience with that plugin?

[hsmith]

I've had luck using logstash forwarder/lumberjack for multiline. That's an agent you'd have to install on your oracle server, though. Logstash can take a beating, I'd be very surprised if it was actually a bottleneck. Could you try rate limiting the plugin and see if that's the actual problem?

[burkm]

Could you try rate limiting the plugin and see if that's the actual problem?

I was thinking of that too. I'll try it and report back.

[hsmith]

How did the testing go? We just got back in for the weekend.

[burkm]

99.97% success!
I put a 50ms delay between each HTTP POST, and I got all but 5 entries out of 22K+. The remaining 5 could be parse errors; it would be very hard to find them to see what happened.

I was thinking about my earlier example where the two loads of the same data resulted in different numbers received. I sent one batch while working at home, so the link was a little slower. That one didn't lose as much data as the one I sent over the LAN. So that supports the idea of load rate being limited.

I guess I'm satisfied at this point. The delay seems a little kludgey, but if it works I can live with that.

[mcapra]

Is it alright if we lock this thread and mark the issue as resolved?

We can certainly leave this issue open if you would like to do additional testing.

[burkm]

Yes, consider it resolved. Thanks!