Benjamin Krein wrote:
> I posted the following in the Users mailing list, but got to thinking
> that maybe Developer's list was a better place for it.
>
> Am I the only one who is experiencing very slow performance importing
> archived log files into MySQL with log2ndo? I'm averaging ~1.5 log
> entries per second being inserted. This seems slow to me.
>
> I've tried dividing the parsing & MySQL service across 2 servers as
> well as move all the parsing/inserting locally on a remote (to Nagios)
> server which is much beefier than our actual Nagios server. My
> insertion speed remains fairly consistent.
>
> I'm currently using the Unix socket option. Is this the wrong one?
> Would TCP be faster (doesn't seem so)? Should I process the logs to
> flat text files first then push those to ndo2db?
>
> At this rate it's going to take FOREVER! I have ~10 million archived
> lines to parse and insert. Maybe I'm expecting too much?
>
First thing to try is to disable all indexes on the sql table
while running the import, and then re-enabling them afterwards.
If you're feeling hardcore, you could try first using the
importer from op5 reports-module and then convert the imported
table to ndoutils format.
The reports-module importer has done live imports of 800M lines
of log-data in < 6 minutes. You'll find it at http://www.op5.org/
under the 'projects' section. We have no snapshots available,
but the gitweb can generate one for you on-the-fly if you want.
Note that it'll drop quite a lot though, as it only bothers with
host states, service states, downtime start/stop and program start/
stop.
YMMV.
--
Andreas Ericsson [email protected]
OP5 AB www.op5.se
Tel: +46 8-230225 Fax: +46 8-230231
This post was automatically imported from historical nagios-devel mailing list archives
Original poster: [email protected]