System is slow, CPU usage skyhigh

[mguthrie]

Can you send us your /var/log/messages and /var/log/httpd/error_log files? You can send them to us via email if preferred.

[admin]

Could you run the attached file and give us the output? The attached script will print the value of the postgres sequences for the XI database. If these are off, they could cause problems. We can provide a script to fix the values if they look weird to us.

To install/run the script:

1. Copy it to some directory on the XI server (e.g. /tmp)
2. Run it like so...

Code: Select all

cd /tmp
chmod +x check_postgres_sequences.sh
./check_postgres_sequences.sh

[Symfoni]

Taking the web UI down permanently would not be a viable solution for us.

Certainly not, we were simply hoping to isolate where the bulk of the CPU usage was coming from.

I stopped the httpd service, and according to 'top' it resulted in a couple of postmaster processes going away, but the remaining ones "picked up the slack" and continued to eat up cpu.

We're going to run some tests to try and recreate an issue like this. We'll keep you posted on what you find.

That would be great!

Just out of curiousity, are you using group dashlets quite a bit?

As far as i know, we aren't using any dashlets or dashboards that weren't included in the installation.
A colleague of mine did create a new dashboard at one point, but it's hardly ever been used/viewed since shortly after we installed the system.

[Symfoni]

Can you send us your /var/log/messages and /var/log/httpd/error_log files? You can send them to us via email if preferred.

What address should i send them to?

[Symfoni]

Could you run the attached file and give us the output?
[...]

Here's the output:

Code: Select all

Nagios XI Postgres Database Sequence Information

CURRENT VALUES
--------------
xi_commands_command_id_seq = 614
xi_events_event_id_seq = 66949
xi_meta_meta_id_seq = 67055
xi_options_option_id_seq = 61
xi_sysstat_sysstat_id_seq = 29
xi_usermeta_usermeta_id_seq = 260
xi_users_user_id_seq = 27

Hope it helps with you some

[mguthrie]

You can mail us the log files at [email protected].

I'll pass along the data from that script to our lead developer.


Our tests from this situation revealed some interesting things related to performance that we're probably going to tweak and hopefully have ready for our next release. We did observe yesterday in our tests that as the amount of browsers open up, the CPU usage climbs quite a bit (httpd), so one of the tweaks that we're looking at would be to have Configuration settings for XI's AJAX updates (which sends fresh data to the browser at regular intervals). We're probably going to have it so that the user can turn back the update times in order to allow more users and browsers to be used at once. Our lead developer is also making some improvements related to postgres that are having dramatic performance improvements as well.

[admin]

Thanks for the script output! Could you run the following script? It increases the value of some of the sequences. If they were incorrectly set, it could cause Postgres query problems, which could contribute to high load. Your log files would likely indicate this.

To run this script, upload it to the Ngios XI server and:

Code: Select all

chmod +x fix_postgres_sequences.sh
./fix_postgres_sequences.sh

[Symfoni]

Here is the output from the script:

Code: Select all

OLD VALUES
--------------
xi_commands_command_id_seq = 625
xi_events_event_id_seq = 68632
xi_meta_meta_id_seq = 68737
xi_options_option_id_seq = 61
xi_sysstat_sysstat_id_seq = 29
xi_usermeta_usermeta_id_seq = 327
xi_users_user_id_seq = 29

NEW VALUES
--------------
xi_commands_command_id_seq = 625
xi_events_event_id_seq = 68632
xi_meta_meta_id_seq = 68737
xi_options_option_id_seq = 99
xi_sysstat_sysstat_id_seq = 99
xi_usermeta_usermeta_id_seq = 399
xi_users_user_id_seq = 49

I noticed that most of the old values have gone up a little from when i ran the check script, so it looks like postgre is managing them and adjusting as it needs to. Setting them a little higher still, as the fix script did, does not yet show signs of lowering postmaster's cpu usage though. Will keep an eye on 'top' and let you know if anything changes.

[rdedon]

Please let us know either way whether the postmaster's cpu usage is up/down/same while running top.

Thank you!.

[Symfoni]

It's been running with the postgresql changes for a few days now, and i don't notice change in postmaster's cpu usage.
If there's been any changes, then they are minimal.