Page 2 of 3
Re: High Load on NagiosXI server
Posted: Tue Mar 23, 2021 12:16 pm
by Dusan.Mandic
| xi_auditlog | 2526.08 |d
script completed successfully, seems like the load is going through the roof again. May be reoptimizing database?
Sent PM with profile
Re: High Load on NagiosXI server
Posted: Tue Mar 23, 2021 2:53 pm
by dchurch
Looks like some Java program running on xxx.xxx.xxx.168 (where xxx.xxx.xxx is the same d-class prefix as your Nagios XI box) is calling the Nagios XI API over and over again (~50 times a minute!), and it's causing a ton of traffic and processing to happen all the time.
That's where all the load is coming from.
Re: High Load on NagiosXI server
Posted: Wed Mar 24, 2021 3:19 pm
by Dusan.Mandic
Can you PM me where you see this?
Re: High Load on NagiosXI server
Posted: Thu Mar 25, 2021 11:15 am
by dchurch
PM sent. 178 requests over the course of 3 minutes, 23/Mar/2021:12:12:28 thru 23/Mar/2021:12:12:31.
Re: High Load on NagiosXI server
Posted: Mon Mar 29, 2021 3:16 pm
by Dusan.Mandic
Looks like we are getting numerous DB connection errors throughout the day
Database Error
A database connection error has been detected, please follow the repair prompt below. If the issue persists, please contact Nagios support.
Run the following from the CLI as root to attempt to repair the DB:
/usr/local/nagiosxi/scripts/repair_databases.sh
Re: High Load on NagiosXI server
Posted: Tue Mar 30, 2021 12:45 pm
by dchurch
Going over the code, to me it looks like that error message is only a guess as to a probable fix. The underlying issue is that the database connection timed out. This can be caused by the mysql server being starved of CPU cycles, where the database is too busy to allow a new connection.
I can see this error intermittently in your logs. Perhaps there's some slow queries going on. If you enable the slow query log, we can maybe diagnose what's causing the database to lag so much:
Edit
/etc/my.cnf.d/server.cnf, adding lines
under [server], e.g.:
Code: Select all
[server]
slow_query_log = 1
long_query_time = 5
slow_query_log_file = /var/log/mariadb/slow-queries.log
Then restart mariadb:
Re: High Load on NagiosXI server
Posted: Wed Mar 31, 2021 5:52 pm
by Dusan.Mandic
| xi_auditlog | 4940.58 | looks like this is still bloating
The database crashed again, two times in a 5 minute span.
I PM'd you the profile from 3/31/21
Re: High Load on NagiosXI server
Posted: Thu Apr 01, 2021 10:39 am
by dchurch
Strange, mysql is reporting a "disk full" error but it doesn't appear that it actually is.
What's the output from this command?
Try this too
On long-running systems with mucho checks, the database can get bogged down with excessive "paper trail" type data and the software's database queries aren't properly utilizing indexes. It just needs better thresholds to get performance back where it should be:
Open Admin => Performance Settings, then click on the Databases tab. Change the following settings:
- Max Log Entries Age: change to 10
- Max Audit Log Age: change to 10
- Max State History Age: change to 30
It might take up to a day for the "cleaner" process to run depending on how your system is configured, but it'll eventually run and clean your database of all these for you.
Re: High Load on NagiosXI server
Posted: Thu Apr 01, 2021 2:15 pm
by Dusan.Mandic
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-vg00_lv_root 242G 43G 189G 19% /
Re: High Load on NagiosXI server
Posted: Thu Apr 01, 2021 4:14 pm
by dchurch
Did you get a chance to change those performance values?