Upgrade 4.1.1 -> 4.2.0, nagios -v is very slow now

[davidnewhall]

Hello,

My organization has a large Nagios installation. I am testing 4.2.0 on our staging server and finding that the startup/restart time has quadrupled. It used to be sub 30 seconds, but it's now close to two minutes. 4.0.8 and 4.1.1 do not exhibit the same issue with the same configs. The extended time to verify configs is breaking my "Service check" that verifies the nagios config. The timeout was 30 seconds and now that it takes longer than that it's broken. (One nagios host checks others for valid config syntax since it's dynamically generated.)

I'm not sure if this is a new feature, a bug, or some oddity in my configs that is suddenly a problem for this new version. Any thoughts or troubleshooting advice is appreciated! Below you will find some output that shows the problem. I have the 4.0.8 and the 4.2.0 binaries on the same host and ran a -v with both on the same configs. Sorry, I don't have a 4.1.1 binary immediately available, but I do not have this problem with that version (I did use it briefly before 4.2 came out).

Code: Select all

# time nagios -v /etc/nagios/nagios.cfg

Nagios Core 4.0.8
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 08-12-2014
License: GPL

Website: http://www.nagios.org
Reading configuration data...
   Read main config file okay...
   Read object config files okay...

Running pre-flight check on configuration data...

Checking objects...
        Checked 53180 services.
        Checked 3301 hosts.
        Checked 239 host groups.
        Checked 2398 service groups.
        Checked 6 contacts.
        Checked 7 contact groups.
        Checked 47 commands.
        Checked 3 time periods.
        Checked 0 host escalations.
        Checked 0 service escalations.
Checking for circular paths...
        Checked 3301 hosts
        Checked 29100 service dependencies
        Checked 0 host dependencies
        Checked 3 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 0
Total Errors:   0

Things look okay - No serious problems were detected during the pre-flight check

real    0m1.003s
user    0m0.777s
sys     0m0.229s

Code: Select all

# time /usr/bin/nagios -v /etc/nagios/nagios.cfg

Nagios Core 4.2.0
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 08-01-2016
License: GPL

Website: https://www.nagios.org
Reading configuration data...
   Read main config file okay...
   Read object config files okay...

Running pre-flight check on configuration data...

Checking objects...
        Checked 53180 services.
        Checked 3301 hosts.
        Checked 239 host groups.
        Checked 2398 service groups.
        Checked 6 contacts.
        Checked 7 contact groups.
        Checked 47 commands.
        Checked 3 time periods.
        Checked 0 host escalations.
        Checked 0 service escalations.
Checking for circular paths...
        Checked 3301 hosts
        Checked 29100 service dependencies
        Checked 0 host dependencies
        Checked 3 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 0
Total Errors:   0

Things look okay - No serious problems were detected during the pre-flight check

real    1m49.622s
user    1m49.167s
sys     0m0.268s

[davidnewhall]

A strace during the "pause" reveals lots and lots of lines like this:

Code: Select all

brk(0x2ba3000)                          = 0x2ba3000
brk(0x2bc4000)                          = 0x2bc4000
brk(0x2be5000)                          = 0x2be5000
brk(0x2c06000)                          = 0x2c06000

[davidnewhall]

I reverted this code change in the 4.2.0 source and recompiled. The issue is not present afterward. I'm not sure what this code does, but it's slow. https://github.com/NagiosEnterprises/na ... 99bf4d3c45

Bug report opened: https://github.com/NagiosEnterprises/na ... issues/154

[tmcdonald]

Just wanted to let you know we've received the GitHub issue (thanks for filing that, by the way) so we'll be moving any further conversation over there so the devs can see it.

For your information, our in-house Core dev is out for the week, but when he gets back I'm quite certain this will be on his radar.