| xi_auditlog | 2526.08 |d
script completed successfully, seems like the load is going through the roof again. May be reoptimizing database?
Sent PM with profile
High Load on NagiosXI server
Re: High Load on NagiosXI server
Looks like some Java program running on xxx.xxx.xxx.168 (where xxx.xxx.xxx is the same d-class prefix as your Nagios XI box) is calling the Nagios XI API over and over again (~50 times a minute!), and it's causing a ton of traffic and processing to happen all the time.
That's where all the load is coming from.
That's where all the load is coming from.
If you didn't get an 8% raise over the course of the pandemic, you took a pay cut.
Discussion of wages is protected speech under the National Labor Relations Act, and no employer can tell you you can't disclose your pay with your fellow employees.
Discussion of wages is protected speech under the National Labor Relations Act, and no employer can tell you you can't disclose your pay with your fellow employees.
-
Dusan.Mandic
- Posts: 60
- Joined: Mon Apr 06, 2020 2:30 pm
Re: High Load on NagiosXI server
Can you PM me where you see this?
Re: High Load on NagiosXI server
PM sent. 178 requests over the course of 3 minutes, 23/Mar/2021:12:12:28 thru 23/Mar/2021:12:12:31.
If you didn't get an 8% raise over the course of the pandemic, you took a pay cut.
Discussion of wages is protected speech under the National Labor Relations Act, and no employer can tell you you can't disclose your pay with your fellow employees.
Discussion of wages is protected speech under the National Labor Relations Act, and no employer can tell you you can't disclose your pay with your fellow employees.
-
Dusan.Mandic
- Posts: 60
- Joined: Mon Apr 06, 2020 2:30 pm
Re: High Load on NagiosXI server
Looks like we are getting numerous DB connection errors throughout the day
Database Error
A database connection error has been detected, please follow the repair prompt below. If the issue persists, please contact Nagios support.
Run the following from the CLI as root to attempt to repair the DB:
/usr/local/nagiosxi/scripts/repair_databases.sh
Database Error
A database connection error has been detected, please follow the repair prompt below. If the issue persists, please contact Nagios support.
Run the following from the CLI as root to attempt to repair the DB:
/usr/local/nagiosxi/scripts/repair_databases.sh
Re: High Load on NagiosXI server
Going over the code, to me it looks like that error message is only a guess as to a probable fix. The underlying issue is that the database connection timed out. This can be caused by the mysql server being starved of CPU cycles, where the database is too busy to allow a new connection.
I can see this error intermittently in your logs. Perhaps there's some slow queries going on. If you enable the slow query log, we can maybe diagnose what's causing the database to lag so much:
Edit /etc/my.cnf.d/server.cnf, adding lines under [server], e.g.:Then restart mariadb:
I can see this error intermittently in your logs. Perhaps there's some slow queries going on. If you enable the slow query log, we can maybe diagnose what's causing the database to lag so much:
Edit /etc/my.cnf.d/server.cnf, adding lines under [server], e.g.:
Code: Select all
[server]
slow_query_log = 1
long_query_time = 5
slow_query_log_file = /var/log/mariadb/slow-queries.log
Code: Select all
service mariadb restartIf you didn't get an 8% raise over the course of the pandemic, you took a pay cut.
Discussion of wages is protected speech under the National Labor Relations Act, and no employer can tell you you can't disclose your pay with your fellow employees.
Discussion of wages is protected speech under the National Labor Relations Act, and no employer can tell you you can't disclose your pay with your fellow employees.
-
Dusan.Mandic
- Posts: 60
- Joined: Mon Apr 06, 2020 2:30 pm
Re: High Load on NagiosXI server
| xi_auditlog | 4940.58 | looks like this is still bloating
The database crashed again, two times in a 5 minute span.
I PM'd you the profile from 3/31/21
The database crashed again, two times in a 5 minute span.
I PM'd you the profile from 3/31/21
You do not have the required permissions to view the files attached to this post.
Re: High Load on NagiosXI server
Strange, mysql is reporting a "disk full" error but it doesn't appear that it actually is.
What's the output from this command?
Try this too
On long-running systems with mucho checks, the database can get bogged down with excessive "paper trail" type data and the software's database queries aren't properly utilizing indexes. It just needs better thresholds to get performance back where it should be:
Open Admin => Performance Settings, then click on the Databases tab. Change the following settings:
- Max Log Entries Age: change to 10
- Max Audit Log Age: change to 10
- Max State History Age: change to 30
It might take up to a day for the "cleaner" process to run depending on how your system is configured, but it'll eventually run and clean your database of all these for you.
What's the output from this command?
Code: Select all
df -h /var/lib/mysql/nagios/On long-running systems with mucho checks, the database can get bogged down with excessive "paper trail" type data and the software's database queries aren't properly utilizing indexes. It just needs better thresholds to get performance back where it should be:
Open Admin => Performance Settings, then click on the Databases tab. Change the following settings:
- Max Log Entries Age: change to 10
- Max Audit Log Age: change to 10
- Max State History Age: change to 30
It might take up to a day for the "cleaner" process to run depending on how your system is configured, but it'll eventually run and clean your database of all these for you.
If you didn't get an 8% raise over the course of the pandemic, you took a pay cut.
Discussion of wages is protected speech under the National Labor Relations Act, and no employer can tell you you can't disclose your pay with your fellow employees.
Discussion of wages is protected speech under the National Labor Relations Act, and no employer can tell you you can't disclose your pay with your fellow employees.
-
Dusan.Mandic
- Posts: 60
- Joined: Mon Apr 06, 2020 2:30 pm
Re: High Load on NagiosXI server
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-vg00_lv_root 242G 43G 189G 19% /
/dev/mapper/VolGroup00-vg00_lv_root 242G 43G 189G 19% /
Last edited by Dusan.Mandic on Thu Apr 01, 2021 4:42 pm, edited 2 times in total.
Re: High Load on NagiosXI server
Did you get a chance to change those performance values?
If you didn't get an 8% raise over the course of the pandemic, you took a pay cut.
Discussion of wages is protected speech under the National Labor Relations Act, and no employer can tell you you can't disclose your pay with your fellow employees.
Discussion of wages is protected speech under the National Labor Relations Act, and no employer can tell you you can't disclose your pay with your fellow employees.