Is there a number of Client or services Nagios can connect?

[jlamar]

Is there a number of Client or services Nagios can connect to. Server overload...

I added a bunch of vm services and everything turned red, seamed like it needed to catch up, but after a day it didn't. So I started deleting.
I deleted about 1000 services and it came back up monitoring as usual.

we now have 4028 items its monitoring. Seams like it started to choke as well.

And many flapping services. Socket Time out

Server Redhat 7.8 64 bit 8gb memory 4 cpu's

[jlamar]

***** Nagios XI Alert *****

Nagios has detected a problem with this service.

Notification Type: PROBLEM

Service: Current Load
Host: localhost
Address: 127.0.0.1
State: CRITICAL
Info:
CRITICAL - load average: 11.55, 14.06, 13.35
Date/Time: 2020-04-06 10:07:01

[jlamar]

I did finally get the flapping services to stop..

[benjaminsmith]

Hi @jlamar,

That's a pretty typical system, Is this a VM or physical server. Let's take a look at your system profile to make sure you don't have any other issues here and review the current check load (hosts + services).

To send us your system profile.
Login to the Nagios XI GUI using a web browser
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and share this in a private message and then reply to this post to bring it up in the queue.

Also, here's a useful guide for improving XI performance:
Maximizing Performance In Nagios XI

[jlamar]

Virtual Server, I sent the profile.zip

[benjaminsmith]

Hi,

Thanks for sending that over. The load on the server is definitely higher than normal, and that's likely due to the presence of corrupted database tables.

Log in to the terminal, run the following script as root:

Code: Select all

/usr/local/nagiosxi/scripts/repair_databases.sh

Let the server run for a while and let me know if the issue is resolved.

Benjamin

[jlamar]

ok, I ran the script, So there is no limit to the services I can run? It was just a database issue? I want to re add the 1000 more vm services and dont want it to crash again (the email, the helpdesk tickets created, kill me..)

[benjaminsmith]

Hello,

Let's tail the database log to make sure there are no errors.

Code: Select all

/var/log/mariadb/mariadb.log

Is this a VM environment, and are you able to add more CPU cores? If you add another 1000 services at a check interval of 5 minutes, that's 3 new checks every second, so if you can increase the interval or add more CPU's that would improve performance and reduce the load on the server.

Also, the Nagios XI license allows for 3 installations, production, test, and backup. I highly recommend setting up a test server to make your changes on before rolling them out to production.

https://support.nagios.com/kb/article.php?id=145

[jlamar]

says command not found! for mariadb

[benjaminsmith]

Hello,

Apologies, try this command:

Code: Select all

tail /var/log/mysqld.log

Nagios XI Log Locations and Descriptions