Page 1 of 4

Is there a number of Client or services Nagios can connect?

Posted: Mon Apr 06, 2020 10:59 am
by jlamar
Is there a number of Client or services Nagios can connect to. Server overload...

I added a bunch of vm services and everything turned red, seamed like it needed to catch up, but after a day it didn't. So I started deleting.
I deleted about 1000 services and it came back up monitoring as usual.

we now have 4028 items its monitoring. Seams like it started to choke as well.

And many flapping services. Socket Time out

Server Redhat 7.8 64 bit 8gb memory 4 cpu's

Re: Is there a number of Client or services Nagios can conne

Posted: Mon Apr 06, 2020 12:41 pm
by jlamar
***** Nagios XI Alert *****

Nagios has detected a problem with this service.

Notification Type: PROBLEM

Service: Current Load
Host: localhost
Address: 127.0.0.1
State: CRITICAL
Info:
CRITICAL - load average: 11.55, 14.06, 13.35
Date/Time: 2020-04-06 10:07:01

Re: Is there a number of Client or services Nagios can conne

Posted: Mon Apr 06, 2020 1:49 pm
by jlamar
I did finally get the flapping services to stop..

Re: Is there a number of Client or services Nagios can conne

Posted: Mon Apr 06, 2020 2:42 pm
by benjaminsmith
Hi @jlamar,

That's a pretty typical system, Is this a VM or physical server. Let's take a look at your system profile to make sure you don't have any other issues here and review the current check load (hosts + services).

To send us your system profile.
Login to the Nagios XI GUI using a web browser
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and share this in a private message and then reply to this post to bring it up in the queue.

Also, here's a useful guide for improving XI performance:
Maximizing Performance In Nagios XI

Re: Is there a number of Client or services Nagios can conne

Posted: Mon Apr 06, 2020 3:00 pm
by jlamar
Virtual Server, I sent the profile.zip

Re: Is there a number of Client or services Nagios can conne

Posted: Mon Apr 06, 2020 4:00 pm
by benjaminsmith
Hi,

Thanks for sending that over. The load on the server is definitely higher than normal, and that's likely due to the presence of corrupted database tables.

Log in to the terminal, run the following script as root:

Code: Select all

/usr/local/nagiosxi/scripts/repair_databases.sh
Let the server run for a while and let me know if the issue is resolved.

Benjamin

Re: Is there a number of Client or services Nagios can conne

Posted: Mon Apr 06, 2020 4:25 pm
by jlamar
ok, I ran the script, So there is no limit to the services I can run? It was just a database issue? I want to re add the 1000 more vm services and dont want it to crash again (the email, the helpdesk tickets created, kill me..) :cry:

Re: Is there a number of Client or services Nagios can conne

Posted: Mon Apr 06, 2020 4:51 pm
by benjaminsmith
Hello,

Let's tail the database log to make sure there are no errors.

Code: Select all

/var/log/mariadb/mariadb.log
Is this a VM environment, and are you able to add more CPU cores? If you add another 1000 services at a check interval of 5 minutes, that's 3 new checks every second, so if you can increase the interval or add more CPU's that would improve performance and reduce the load on the server.

Also, the Nagios XI license allows for 3 installations, production, test, and backup. I highly recommend setting up a test server to make your changes on before rolling them out to production.

https://support.nagios.com/kb/article.php?id=145

Re: Is there a number of Client or services Nagios can conne

Posted: Mon Apr 06, 2020 5:07 pm
by jlamar
says command not found! for mariadb

Re: Is there a number of Client or services Nagios can conne

Posted: Mon Apr 06, 2020 5:13 pm
by benjaminsmith
Hello,

Apologies, try this command:

Code: Select all

tail /var/log/mysqld.log
Nagios XI Log Locations and Descriptions