Re: [Nagios-devel] High latencies problem.
Posted: Thu Mar 26, 2009 2:13 pm
On 3/26/2009 3:05 AM, Hendrik Baecker wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Alessandro Ren schrieb:
> =20
>> Just to let you know that my latency gets under 0.2s if a disabl=
ed
>> performance data processing.
>> =20
> 1) Nice to hear about the nice performance
> 2) Bad to hear that performance data processing was your bottleneck
>
> =20
>> I=B4ve changed the way we process them and everything is very ni=
ce
>> with 11k services.
>>
>> =20
> How did you process them before your action?
> That could be interesting for other users who ran into that problem on =
a
> bigger installation, and maybe we are able to find a better way overall=
?
> =20
I have nagios writing all perfomance data to a file and rotating it=20
each 60s and I=B4ve a importer written in C, perl would to catch up with=20
the files being rotated, to import them each 3 minutes.
Yesterday we went to 19k services, more then 50% are checs with=20
NRPE, a lot of perls beeng run and latency is te 0.4s using the schema=20
above.
I am running nagios on a two node cluster, 64bits RedHat=20
enterprise, each node is a Two dual core Xeon 3GHz processors with 8GB=20
of RAM.
Mysql runs on this cluster in a multi-master replication setup and=20
I=B4ve nagios running on one side with the importer and the CGIs and our=20
own custom interface running on the other, with rsync sending the status=20
file each 2 seconds two the other side. All watched by heartbeat.
I=B4ve noticed that with this many sevices, the interface get very=20
slow and starves the server, so I=B4ve moved all comments to the=20
database, as nagios would go throught the comments linked list for each=20
service, changed the nagios code to write it direclty there and the CGis=20
to read from there. We also serialize all status data to a memcahe to=20
speed up the interface, creating and 70s cache, which worked very nice=20
brought the load that was at 15/17 to 2/4.
[]s.
>
> Regards
> Hendrik
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.8 (Darwin)
>
> iEYEARECAAYFAknLGysACgkQlI0PwfxLQjmw9ACeNLUkL7dvtSimIkA2kMSIWEY0
> e+YAniQ6bWoh8JKX7Oez83Gx+ZR1/p1W
> =3D0eYu
> -----END PGP SIGNATURE-----
>
> -----------------------------------------------------------------------=
-------
> _______________________________________________
> Nagios-devel mailing list
> [email protected]
> https://lists.sourceforge.net/lists/lis ... gios-devel
> =20
This post was automatically imported from historical nagios-devel mailing list archives
Original poster: [email protected]
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Alessandro Ren schrieb:
> =20
>> Just to let you know that my latency gets under 0.2s if a disabl=
ed
>> performance data processing.
>> =20
> 1) Nice to hear about the nice performance
> 2) Bad to hear that performance data processing was your bottleneck
>
> =20
>> I=B4ve changed the way we process them and everything is very ni=
ce
>> with 11k services.
>>
>> =20
> How did you process them before your action?
> That could be interesting for other users who ran into that problem on =
a
> bigger installation, and maybe we are able to find a better way overall=
?
> =20
I have nagios writing all perfomance data to a file and rotating it=20
each 60s and I=B4ve a importer written in C, perl would to catch up with=20
the files being rotated, to import them each 3 minutes.
Yesterday we went to 19k services, more then 50% are checs with=20
NRPE, a lot of perls beeng run and latency is te 0.4s using the schema=20
above.
I am running nagios on a two node cluster, 64bits RedHat=20
enterprise, each node is a Two dual core Xeon 3GHz processors with 8GB=20
of RAM.
Mysql runs on this cluster in a multi-master replication setup and=20
I=B4ve nagios running on one side with the importer and the CGIs and our=20
own custom interface running on the other, with rsync sending the status=20
file each 2 seconds two the other side. All watched by heartbeat.
I=B4ve noticed that with this many sevices, the interface get very=20
slow and starves the server, so I=B4ve moved all comments to the=20
database, as nagios would go throught the comments linked list for each=20
service, changed the nagios code to write it direclty there and the CGis=20
to read from there. We also serialize all status data to a memcahe to=20
speed up the interface, creating and 70s cache, which worked very nice=20
brought the load that was at 15/17 to 2/4.
[]s.
>
> Regards
> Hendrik
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.8 (Darwin)
>
> iEYEARECAAYFAknLGysACgkQlI0PwfxLQjmw9ACeNLUkL7dvtSimIkA2kMSIWEY0
> e+YAniQ6bWoh8JKX7Oez83Gx+ZR1/p1W
> =3D0eYu
> -----END PGP SIGNATURE-----
>
> -----------------------------------------------------------------------=
-------
> _______________________________________________
> Nagios-devel mailing list
> [email protected]
> https://lists.sourceforge.net/lists/lis ... gios-devel
> =20
This post was automatically imported from historical nagios-devel mailing list archives
Original poster: [email protected]