5.5.1 Httpd high load

[Envera IT]

@Ehamby, Please send in your Nagios XI System Profile.
To send us your system profile. Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and upload it to a cloud storage of your choice. You can share a link with me in a personal message.
After you upload the profile please post something in this thread to bring it up in the support queue.

This has been sent, thanks for looking into this.

[npolovenko]

@Ehamby, Thanks for sending in your profile. I examined the log files and I didn't see any particular errors related to your configuration that could be causing the high apache load.
The only thing I can think of is that XI 5.5 has a more complicated user permission structure that could contribute to a higher load since all the data is going through more filters then it used to. You could make this better by giving the administrator rights to more users. That way the system won't have to work on figuring out whether X user has an access to Y and Z servers and so on. This should decrease the httpd load.

[BanditBBS]

@Ehamby, how many checks does your system perform and how many users do you think could be browsing your system at one time? What you are describing is what I have always seen in my heavily used system and if your numbers are high enough to my questions I'll send yo some tweaks myself and others have dont to make it perform better(this is even before 5.5 as I haven't upgraded yet).

[lmiltchev]

@Ehamby, any updates?

[Envera IT]

@Ehamby, how many checks does your system perform and how many users do you think could be browsing your system at one time? What you are describing is what I have always seen in my heavily used system and if your numbers are high enough to my questions I'll send yo some tweaks myself and others have dont to make it perform better(this is even before 5.5 as I haven't upgraded yet).

I'd really appreciate that! Total per minute checks are around 1900-2000. We have 3237 hosts and 10218 services, we add between 10-50 hosts a week so growth is a thing. We probably have around 10-15 people in the server maximum, I'd estimate an average of 6-8 people logged in at a time.

Server is a VM, with 4 cores, 8GB RAM, and a SAN for storage backend. We're looking at offloading the database but haven't moved on that yet. I'm aware we're starting to get to a point where we need to look at performance tuning, I was just surprised the update increased load by as much as it did.

[Envera IT]

@Ehamby, any updates?

I mean, honestly I don't really think making more people admins is a good solution. I can set people to have access to view all objects, would that be similar to making them admins in terms of load reduction?

[lmiltchev]

You have quite large instance of Nagios XI, and not enough resources - 4 cores, 8GB RAM with 3237 hosts and 10218 services... Please review our documentation on the hardware requirements needed to run Nagios XI here:

https://assets.nagios.com/downloads/nag ... ements.pdf

In addition to adding more cores and RAM, you would need to move XI to a physical server.

For installations monitoring more than 1000 hosts or 5000 services, we generally recommend installing Nagios XI on a physical server as opposed to a VM. Because of shared hardware resources and the way VM's handle process forking, virtual machines may not be able to meet the hardware needs of large environments even with performance enhancements.

Make sure you use very fast HDDs. I don't think using SAN for storage backend is a very good idea.

You can do some performance tweaks - review the "Boosting Performance" section in the Administrator's Guide here:

https://assets.nagios.com/downloads/nag ... p#boosting

Lastly, if you are adding more and more hosts to this system, maybe it is time to consider adding a new Nagios XI instance.

[Envera IT]

You have quite large instance of Nagios XI, and not enough resources - 4 cores, 8GB RAM with 3237 hosts and 10218 services... Please review our documentation on the hardware requirements needed to run Nagios XI here:

https://assets.nagios.com/downloads/nag ... ements.pdf

In addition to adding more cores and RAM, you would need to move XI to a physical server.

For installations monitoring more than 1000 hosts or 5000 services, we generally recommend installing Nagios XI on a physical server as opposed to a VM. Because of shared hardware resources and the way VM's handle process forking, virtual machines may not be able to meet the hardware needs of large environments even with performance enhancements.

Make sure you use very fast HDDs. I don't think using SAN for storage backend is a very good idea.

You can do some performance tweaks - review the "Boosting Performance" section in the Administrator's Guide here:

https://assets.nagios.com/downloads/nag ... p#boosting

Lastly, if you are adding more and more hosts to this system, maybe it is time to consider adding a new Nagios XI instance.

That sizing doc is pretty sparse on real advice. We sit at 60% ram utilization and the doc says 8GB without clear advice on larger amounts. The SAN is a hybrid flash array, it's pretty fast. CPU wasnt an issue until the upgrade. I can look at increasing things but I'm just calling it like I see it on the recent performance issue. We upgraded, load went from 1.2 to 4+ over night and only when people are logged in actively using the system. I would think that should warrant an investigation on your end rather than the customer having to change their environment or make everyone an admin.

[Envera IT]

We've doubled the resources on the VM to 8 vCPU's and 16GB of ram. We do have a spare physical server that we could set it up on if necessary but we're going to see how it performs with these resources for now and I'll report back tomorrow on how its going.

[lmiltchev]

Sounds good. Let us know if the load wend down. We will keep the thread open for the time being.