Fusion 4.1.6

[hbouma]

I see that the release of Fusion 4.1.6 includes a fix for "Major performance increases with large number of mapped users -BH". Is this the same fix that was provided to us in ticket 100103 (https://support.nagios.com/forum/viewto ... 17&t=50159 and https://support.nagios.com/forum/viewto ... 17&t=50902 and https://support.nagios.com/forum/viewto ... 17&t=49945)

We need to know so we don't break our system by running the upgrade.

[scottwilkerson]

These should include those changes and a few additional enhancements.

The person who did the remote on your system was the developer who added the changes to 4.1.6

[martinnick]

I see that after updating to the new version of fusion we are getting an error when clicking views link and looking at the default views. (The URL provided is invalid. Try editing the view.) Would this happen to be related to something we missed during the upgrade? We also seem to be having an issue with (MariaDB high CPU usage) again after the update. Same as the previous post in which we opened a ticket.

[scottwilkerson]

I see that after updating to the new version of fusion we are getting an error when clicking views link and looking at the default views. (The URL provided is invalid. Try editing the view.) Would this happen to be related to something we missed during the upgrade? We also seem to be having an issue with (MariaDB high CPU usage) again after the update. Same as the previous post in which we opened a ticket.

I don't believe this would have changed with the upgrade.

If you edit the view do you see the URL? Is it valid?

[hbouma]

The URL does work if we manually enter it. For instance, the Tactical Summery does not show in views, but if I go to https://servername/nagiosfusion/includes/components/fusioncore/tactical_summary.php I can get to the page and view it. You can see the view when we attempt to look at Tactical Summary in Fusion on top. The configuration is listed on the bottom. This is happening to all users for all views.

2019-01-08 14_25_07-Document1 - Word.png

As far as the issue with our DB crashing, this has been previously worked on with in posts https://support.nagios.com/forum/viewto ... 17&t=50902 and https://support.nagios.com/forum/viewto ... 17&t=50159 along with ticket 100103. Any help with that would be appreciated. We are seeing our cron jobs for /bin/sh -c /usr/bin/php -q /usr/local/nagiosfusion/cron/dbmaint_subsys.php >>/usr/local/nagiosfusion/var/log/dbmaint_subsys.log 2>&1 not completeting. Since 5:30 AM today, we have 127 of these scripts actively running. After enough time (around 22 to 24 hours), we see the /var/log/mariadb/mariadb.log file show multiple deadlocks. The end result being that the Fusion website becomes unavailable for 20 or 30 minutes while the DB attempts to repair itself. This repair can last for a few hours or less before it happens again.

The band-aid when the system gets into this unhealthy state has been to run the following commands to buy us another 20+ hours before the issue pops up again.
pkill -9 -u nagios
systemctl stop crond
/usr/bin/php /usr/local/nagiosfusion/cron/dbmaint_subsys.php
systemctl start crond
systemctl start ncpa_passive
systemctl start ncpa_listener

[tgriep]

The View URL is missing missing the server name in the link.

Edit the View URl and add the FQDN to the path

Code: Select all

http://<servername>

Like so.

Code: Select all

http://<servername>/nagiosfusion/includes/components/fusioncore/tactical_summary.php

Also, go to the Admin > SystemSettings menu and verify that the Internal URL is valid.
Update it if needed.

Can you look in the following log file and post any errors here so we can view them?

Code: Select all

/usr/local/nagiosfusion/var/log/dbmaint_subsys.log

Lets increate the max connections and the open file limit for the Mariadb database.

Edit the /etc/my.cnf file and under the [mysqld] section ass the following options.

Code: Select all

max_connections 800
open_files_limit    4096

Lets see if that is what is causing the database issues.

[martinnick]

Can you look in the following log file and post any errors here so we can view them?

Code: Select all

/usr/local/nagiosfusion/var/log/dbmaint_subsys.log

A small snippet of those errors within that log.

2019-01-09 03:53:22[s: 0, u: 0] fusioncore_delete_unnecessary_polled_extras(): unable to execute: polled_data_id < 26045143, poll_key = dashlets_params_simple_hosts
2019-01-09 03:53:28[s: 0, u: 0] fusioncore_delete_unnecessary_polled_extras(): unable to execute: polled_data_id < 26045169, poll_key = dashlets_params_simple_hostgroups
2019-01-09 03:53:29[s: 0, u: 0] fusioncore_delete_unnecessary_polled_extras(): unable to execute: polled_data_id < 26044779, poll_key = dashlets_params_simple_hostgroups
2019-01-09 03:53:35[s: 0, u: 0] fusioncore_delete_unnecessary_polled_extras(): unable to execute: polled_data_id < 26045108, poll_key = dashlets_params_masked_services
2019-01-09 03:53:42[s: 0, u: 0] fusioncore_delete_unnecessary_polled_extras(): unable to execute: polled_data_id < 26045180, poll_key = service_status
2019-01-09 03:56:03[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_data
WHERE server_id NOT IN (3,1,8,5,7)
2019-01-09 03:56:57[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_data
WHERE UNIX_TIMESTAMP(polled_time) < 1546937765
2019-01-09 03:59:08[s: 0, u: 0] fusioncore_delete_unnecessary_polled_extras(): unable to execute: polled_data_id < 26045180, poll_key = hostgroup_members
2019-01-09 03:59:10[s: 0, u: 0] fusioncore_delete_unnecessary_polled_extras(): unable to execute: polled_data_id < 26044779, poll_key = dashlets_params_simple_servicegroups
2019-01-09 03:59:13[s: 0, u: 0] fusioncore_delete_unnecessary_polled_extras(): unable to execute: polled_data_id < 26045143, poll_key = dashlets_params_simple_hostgroups
2019-01-09 03:59:20[s: 0, u: 0] fusioncore_delete_unnecessary_polled_extras(): unable to execute: polled_data_id < 26045169, poll_key = dashlets_params_simple_servicegroups
2019-01-09 03:59:21[s: 0, u: 0] fusioncore_delete_unnecessary_polled_extras(): unable to execute: polled_data_id < 26045636, poll_key = host_status
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 589
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 589
2019-01-09 04:06:25[s: 0, u: 0] fusioncore_delete_unnecessary_polled_extras(): unable to execute: polled_data_id < 26045636, poll_key = service_status
2019-01-09 04:06:25[s: 0, u: 0] fusioncore_delete_unnecessary_polled_extras(): unable to execute: polled_data_id < 26045169, poll_key = dashlets_params_masked_hosts
2019-01-09 04:06:25[s: 0, u: 0] fusioncore_delete_unnecessary_polled_extras(): unable to execute: polled_data_id < 26045143, poll_key = dashlets_params_simple_servicegroups
2019-01-09 04:06:25[s: 0, u: 0] fusioncore_delete_unnecessary_polled_extras(): unable to execute: polled_data_id < 26045180, poll_key = servicegroup_members
2019-01-09 04:06:25[s: 0, u: 0] fusioncore_delete_unnecessary_polled_extras(): unable to execute: polled_data_id < 26044779, poll_key = dashlets_params_masked_hosts
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 589
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 589
2019-01-09 04:19:21[s: 0, u: 0] fusioncore_delete_unnecessary_polled_extras(): unable to execute: polled_data_id < 26045641, poll_key = host_status
2019-01-09 04:19:21[s: 0, u: 0] fusioncore_delete_unnecessary_polled_extras(): unable to execute: polled_data_id < 26045636, poll_key = hostgroup_members
2019-01-09 04:19:27[s: 0, u: 0] fusioncore_delete_unnecessary_polled_extras(): unable to execute: polled_data_id < 26045143, poll_key = dashlets_params_masked_hosts
2019-01-09 04:19:27[s: 0, u: 0] fusioncore_delete_unnecessary_polled_extras(): unable to execute: polled_data_id < 26044779, poll_key = dashlets_params_masked_services
2019-01-09 04:19:36[s: 0, u: 0] fusioncore_delete_unnecessary_polled_extras(): unable to execute: polled_data_id < 26045180, poll_key = users
PHP Fatal error: Call to a member function debugDumpParams() on a non-object in /usr/local/nagiosfusion/html/includes/classes/db.inc.php on line 589
2019-01-09 04:22:45[s: 0, u: 0] fusioncore_delete_unnecessary_polled_extras(): unable to execute: polled_data_id < 26045636, poll_key = servicegroup_members
2019-01-09 04:22:45[s: 0, u: 0] fusioncore_delete_unnecessary_polled_extras(): unable to execute: polled_data_id < 26045180, poll_key = dashlets_params_simple_hosts
2019-01-09 04:22:45[s: 0, u: 0] fusioncore_delete_unnecessary_polled_extras(): unable to execute: polled_data_id < 26045641, poll_key = service_status
2019-01-09 04:22:45[s: 0, u: 0] fusioncore_delete_unnecessary_polled_extras(): unable to execute: polled_data_id < 26045620, poll_key = host_status
2019-01-09 04:22:46[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE
FROM polled_extras
WHERE polled_data_id NOT IN
(SELECT polled_data_id FROM polled_data)
2019-01-09 04:22:47[s: 0, u: 0] dbmaint_query_and_log() unable to gather query data: DELETE

Lets increate the max connections and the open file limit for the Mariadb database.

Edit the /etc/my.cnf file and under the [mysqld] section ass the following options.

Those options are already set over those limits in the config.

Code: Select all

max_connections=818
open_files_limit = 20480

[tgriep]

Thanks for letting us know about the changes that are already done for the SQL server.

The errors that you posted are probably caused by the multiple dbmaint processes and the deadlocks.
We would have to go back further to see the first entries when the errors started to see what could of happened.

[martinnick]

It would appear that we only have rolled logs for dbmaint_subsys.log from September and the current log has information from today only. Is there a way to extract what you are looking for out of the database itself?

[tgriep]

With all of the database issues, there may not be any valid data but if you run the following command, it may show something.

Code: Select all

echo 'select * from log;' |mysql -t -u fusion -pfusion fusion

You can also see that same data in the Fusion GUI at the Admin > Fusion Logs menu.

Also, check the standard log files in the /var/log folder and see if there are any clues in there.