Nagios Support Forum

Posted: **Thu Jul 22, 2021 4:18 pm**

Hello,

I have a Nagios XI instance that's run into some difficulty. It's been under heavy load for a while. The last time I got assistance with it, it got its databases all converted to InnoDB. This helped immensely to stop the table crashes, but something else has happened. Now symptoms include:
Monitoring Engine Event Queue dashlet only says "Error: no output from backend!"
Hosts are displayed in the web UI, but no services are. This extends to the API as well:

Code: Select all

[root@*** scripts]# curl --insecure -XGET "https://***/nagiosxi/api/v1/objects/servicestatus?apikey=****&pretty=1"
{
    "recordcount": 0,
    "servicestatus": [

    ]
}

Configuration changes still appear to be working. There was an old install of Check MK, which still shows the services. I was able to deactivate a couple checks, apply the updated configuration, and see the checks disappear from the Check MK page.

To further complicate things, there was a failed attempt to upgrade from 5.8.4 to 5.8.5. The last few lines output were:

Code: Select all

UPGRADE: POST-UPGRADE: NDO post upgrade started...
Performing install...
   > Enter MySQL root password:  > Creating database with user
^[[D^[[D^[[C^[[D^[[D
 > Press <ENTER> to continue... > Using command line supplied credentials for account/db creation...
   > Username: ndoutils
   > Password: *****
   > Database: nagios

ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
Something went wrong creating database 'nagios'

The ^[[D were from me hitting keys to see if it was still alive. I had let the upgrade script run overnight but it never finished.

To even further complicate things, I moved the databases onto an external server to reduce the CPU load on the Nagios XI box. The doc warns about there being any errors, but the databases actually migrated just fine. The load on the Nagios XI box is much lower and putting the database on another VM seems to have allowed it to keep up with the check processing load. It's just that the service check results aren't showing up in the XI web UI or API (but like I mentioned, they're up-to-date as shown in the old Check MK web UI).

Thanks!

Posted: **Fri Jul 23, 2021 9:59 am**

Hi,

When you offloaded the database, did you follow our guide? According to the output posted, it looks like it cannot connect to the database. You can run a connection test from XI by running the following command.

Code: Select all

mysql -u nagios -p'nagiosP@ssw0rd' -h <IP_ADDRESS_OF_MYSQL_OR_MARIADB_SERVER> -e STATUS;

If that fails, please review the required changes in the guide below to make sure all the settings were updated correctly.
Nagios XI Offloading MySQL to Remote Server

Also, log into the Core interface, and let me know if you are setting current service check results.

Code: Select all

http://<IP address>/nagios

If this is not caused by a database connection setting, please send us the system profile. Thanks, Benjamin

To send us your system profile.
Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button

Posted: **Fri Jul 23, 2021 2:18 pm**

benjaminsmith wrote:Hi,

When you offloaded the database, did you follow our guide? According to the output posted, it looks like it cannot connect to the database.

Also, log into the Core interface, and let me know if you are setting current service check results.

If this is not caused by a database connection setting, please send us the system profile.

Hello,
I followed both that offloading guide and the STRICT_TRANS_TABLES guide. I verified the connections to all three databases before I performed the mysqldump step.

The Core interface shows recent check results, fortunately.

I submitted the profile via PM.

Thanks!

Posted: **Mon Jul 26, 2021 11:30 am**

This portion:

Code: Select all

UPGRADE: POST-UPGRADE: NDO post upgrade started...
Performing install...
   > Enter MySQL root password:  > Creating database with user
^[[D^[[D^[[C^[[D^[[D
> Press <ENTER> to continue... > Using command line supplied credentials for account/db creation...
   > Username: ndoutils
   > Password: *****
   > Database: nagios

The username from your /usr/local/nagiosxi/html/config.inc.php is nagios, not ndoutils.

This is likely why you are having issues:

Code: Select all

Database Error: Could not connect to database
Lost connection to MySQL server at 'reading initial communication packet', system error: 0
Database Error: Could not connect to database
Too many connections

You will need to set max_connections=800 (or higher) on your offloaded DB server. Is it running mariadb or mysql? What OS is it running?

Code: Select all

rpm -qa |grep -i mysql
rpm -qa | grep -i maria
uname -a
cat /etc/*release

Posted: **Thu Jul 29, 2021 12:03 pm**

ssax wrote:The username from your /usr/local/nagiosxi/html/config.inc.php is nagios, not ndoutils.

You will need to set max_connections=800 (or higher) on your offloaded DB server. Is it running mariadb or mysql? What OS is it running?

I double-checked my config.inc.php and started the update process over from scratch and it was successful this time. I don't know how I got the wrong ID the first time but I'm not going to lose any sleep trying to figure it out.

I checked the max_connections and I'd forgotten to bump that up. I looked at the MariaDB docs https://mariadb.com/kb/en/server-system ... onnections which said it could go up to 100k, so I tried 1000 even though the Nagios docs say 800. I still ran out of connections at 1000 according to "show global status like 'Max_used_connections';" (at 1001), so I bumped it up again to 2000. Max_used_connections hasn't gone much over 400 since I did that, though, so I don't yet know what the actual max needs to be.

Here's the output from the commands:

Code: Select all

[root@den-nagiosdb ~]# rpm -qa |grep -i mysql
perl-DBD-MySQL-4.046-3.module_el8.3.0+419+c2dec72b.x86_64
[root@den-nagiosdb ~]# rpm -qa | grep -i maria
mariadb-common-10.3.28-1.module_el8.3.0+757+d382997d.x86_64
mariadb-connector-c-3.1.11-2.el8_3.x86_64
mariadb-server-10.3.28-1.module_el8.3.0+757+d382997d.x86_64
mariadb-connector-c-config-3.1.11-2.el8_3.noarch
mariadb-errmsg-10.3.28-1.module_el8.3.0+757+d382997d.x86_64
mariadb-backup-10.3.28-1.module_el8.3.0+757+d382997d.x86_64
mariadb-server-utils-10.3.28-1.module_el8.3.0+757+d382997d.x86_64
mariadb-10.3.28-1.module_el8.3.0+757+d382997d.x86_64
mariadb-connector-c-devel-3.1.11-2.el8_3.x86_64
mariadb-gssapi-server-10.3.28-1.module_el8.3.0+757+d382997d.x86_64
mariadb-devel-10.3.28-1.module_el8.3.0+757+d382997d.x86_64
[root@den-nagiosdb ~]# uname -a
Linux den-nagiosdb 4.18.0-315.el8.x86_64 #1 SMP Mon Jun 28 19:09:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
[root@den-nagiosdb ~]# cat /etc/*release
CentOS Stream release 8
NAME="CentOS Stream"
VERSION="8"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Stream 8"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
CentOS Stream release 8
CentOS Stream release 8

Even though the version upgrade was successful, I've still got the same symptoms. The Monitoring Engine Event Queue dashlet still says "Error: No output from backend!", the Host Status TAC Summary dashlet looks fine but the Service Status TAC Summary dashlet is empty, and XI doesn't think any of the hosts have services that I can tell. But they do look okay in the Nagios Core interface.

Posted: **Thu Jul 29, 2021 7:13 pm**

Please send the output of this command:
- NOTE: You may need to adjust the -h 127.0.0.1, the -uroot, and -pnagiosxi in the command if your DB is offloaded to another server and/or you've changed the root mysql password

Code: Select all

echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -h 127.0.0.1 -uroot -pnagiosxi --table

PM me a fresh copy of your profile.zip as well.

Posted: **Fri Jul 30, 2021 9:03 am**

Here's the command output:

Code: Select all

+--------------------------------------------+------------+
| Table                                      | Size in MB |
+--------------------------------------------+------------+
| nagios_acknowledgements                    |       0.42 |
| nagios_commands                            |       0.06 |
| nagios_commenthistory                      |     463.42 |
| nagios_comments                            |      10.02 |
| nagios_configfiles                         |       0.03 |
| nagios_configfilevariables                 |       0.02 |
| nagios_conninfo                            |       0.02 |
| nagios_contact_addresses                   |       0.03 |
| nagios_contact_notificationcommands        |       0.03 |
| nagios_contactgroup_members                |       0.03 |
| nagios_contactgroups                       |       0.03 |
| nagios_contactnotificationmethods          |       3.34 |
| nagios_contactnotifications                |       3.66 |
| nagios_contacts                            |       0.03 |
| nagios_contactstatus                       |       0.03 |
| nagios_customvariables                     |       0.63 |
| nagios_customvariablestatus                |       0.63 |
| nagios_dbversion                           |       0.02 |
| nagios_downtimehistory                     |      18.03 |
| nagios_eventhandlers                       |       7.95 |
| nagios_externalcommands                    |       1.52 |
| nagios_flappinghistory                     |     189.67 |
| nagios_host_contactgroups                  |       0.17 |
| nagios_host_contacts                       |       0.03 |
| nagios_host_parenthosts                    |       0.03 |
| nagios_hostchecks                          |       1.61 |
| nagios_hostdependencies                    |       0.03 |
| nagios_hostescalation_contactgroups        |       0.30 |
| nagios_hostescalation_contacts             |       0.16 |
| nagios_hostescalations                     |       0.42 |
| nagios_hostgroup_members                   |       0.17 |
| nagios_hostgroups                          |       0.03 |
| nagios_hosts                               |       1.63 |
| nagios_hoststatus                          |       2.53 |
| nagios_instances                           |       0.02 |
| nagios_objects                             |      30.61 |
| nagios_processevents                       |       0.42 |
| nagios_programstatus                       |       0.03 |
| nagios_runtimevariables                    |       0.03 |
| nagios_scheduleddowntime                   |       0.03 |
| nagios_service_contactgroups               |       3.03 |
| nagios_service_contacts                    |       4.03 |
| nagios_service_parentservices              |       0.03 |
| nagios_servicechecks                       |      11.03 |
| nagios_servicedependencies                 |       0.03 |
| nagios_serviceescalation_contactgroups     |       3.03 |
| nagios_serviceescalation_contacts          |       0.03 |
| nagios_serviceescalations                  |       4.03 |
| nagios_servicegroup_members                |       0.27 |
| nagios_servicegroups                       |       0.03 |
| nagios_services                            |      24.58 |
| nagios_systemcommands                      |       0.05 |
| nagios_timedeventqueue                     |       0.09 |
| nagios_timedevents                         |       0.09 |
| nagios_timeperiod_timeranges               |       0.03 |
| nagios_timeperiods                         |       0.03 |
| tbl_command                                |       0.06 |
| tbl_contact                                |       0.03 |
| tbl_contactgroup                           |       0.03 |
| tbl_contacttemplate                        |       0.03 |
| tbl_domain                                 |       0.03 |
| tbl_host                                   |       0.50 |
| tbl_hostdependency                         |       0.03 |
| tbl_hostescalation                         |       0.03 |
| tbl_hostextinfo                            |       0.03 |
| tbl_hostgroup                              |       0.03 |
| tbl_hosttemplate                           |       0.03 |
| tbl_info                                   |       0.17 |
| tbl_lnkContactToCommandHost                |       0.02 |
| tbl_lnkContactToCommandService             |       0.02 |
| tbl_lnkContactToContactgroup               |       0.02 |
| tbl_lnkContactToContacttemplate            |       0.02 |
| tbl_lnkContactToVariabledefinition         |       0.02 |
| tbl_lnkContactgroupToContact               |       0.02 |
| tbl_lnkContactgroupToContactgroup          |       0.02 |
| tbl_lnkContacttemplateToCommandHost        |       0.02 |
| tbl_lnkContacttemplateToCommandService     |       0.02 |
| tbl_lnkContacttemplateToContactgroup       |       0.02 |
| tbl_lnkContacttemplateToContacttemplate    |       0.02 |
| tbl_lnkContacttemplateToVariabledefinition |       0.02 |
| tbl_lnkHostToContact                       |       0.02 |
| tbl_lnkHostToContactgroup                  |       0.02 |
| tbl_lnkHostToHost                          |       0.02 |
| tbl_lnkHostToHostgroup                     |       0.09 |
| tbl_lnkHostToHosttemplate                  |       0.11 |
| tbl_lnkHostToVariabledefinition            |       0.02 |
| tbl_lnkHostdependencyToHost_DH             |       0.02 |
| tbl_lnkHostdependencyToHost_H              |       0.02 |
| tbl_lnkHostdependencyToHostgroup_DH        |       0.02 |
| tbl_lnkHostdependencyToHostgroup_H         |       0.02 |
| tbl_lnkHostescalationToContact             |       0.02 |
| tbl_lnkHostescalationToContactgroup        |       0.02 |
| tbl_lnkHostescalationToHost                |       0.02 |
| tbl_lnkHostescalationToHostgroup           |       0.02 |
| tbl_lnkHostgroupToHost                     |       0.02 |
| tbl_lnkHostgroupToHostgroup                |       0.02 |
| tbl_lnkHosttemplateToContact               |       0.02 |
| tbl_lnkHosttemplateToContactgroup          |       0.02 |
| tbl_lnkHosttemplateToHost                  |       0.02 |
| tbl_lnkHosttemplateToHostgroup             |       0.02 |
| tbl_lnkHosttemplateToHosttemplate          |       0.02 |
| tbl_lnkHosttemplateToVariabledefinition    |       0.02 |
| tbl_lnkServiceToContact                    |       0.02 |
| tbl_lnkServiceToContactgroup               |       0.02 |
| tbl_lnkServiceToHost                       |       0.05 |
| tbl_lnkServiceToHostgroup                  |       0.02 |
| tbl_lnkServiceToServicegroup               |       0.02 |
| tbl_lnkServiceToServicetemplate            |       0.06 |
| tbl_lnkServiceToVariabledefinition         |       0.02 |
| tbl_lnkServicedependencyToHost_DH          |       0.02 |
| tbl_lnkServicedependencyToHost_H           |       0.02 |
| tbl_lnkServicedependencyToHostgroup_DH     |       0.02 |
| tbl_lnkServicedependencyToHostgroup_H      |       0.02 |
| tbl_lnkServicedependencyToService_DS       |       0.02 |
| tbl_lnkServicedependencyToService_S        |       0.02 |
| tbl_lnkServicedependencyToServicegroup_DS  |       0.02 |
| tbl_lnkServicedependencyToServicegroup_S   |       0.02 |
| tbl_lnkServiceescalationToContact          |       0.02 |
| tbl_lnkServiceescalationToContactgroup     |       0.02 |
| tbl_lnkServiceescalationToHost             |       0.02 |
| tbl_lnkServiceescalationToHostgroup        |       0.02 |
| tbl_lnkServiceescalationToService          |       0.02 |
| tbl_lnkServiceescalationToServicegroup     |       0.02 |
| tbl_lnkServicegroupToService               |       0.02 |
| tbl_lnkServicegroupToServicegroup          |       0.02 |
| tbl_lnkServicetemplateToContact            |       0.02 |
| tbl_lnkServicetemplateToContactgroup       |       0.02 |
| tbl_lnkServicetemplateToHost               |       0.02 |
| tbl_lnkServicetemplateToHostgroup          |       0.02 |
| tbl_lnkServicetemplateToServicegroup       |       0.02 |
| tbl_lnkServicetemplateToServicetemplate    |       0.02 |
| tbl_lnkServicetemplateToVariabledefinition |       0.02 |
| tbl_lnkTimeperiodToTimeperiod              |       0.02 |
| tbl_logbook                                |       0.02 |
| tbl_mainmenu                               |       0.02 |
| tbl_permission                             |       0.02 |
| tbl_permission_inactive                    |       0.02 |
| tbl_service                                |       0.13 |
| tbl_servicedependency                      |       0.03 |
| tbl_serviceescalation                      |       0.03 |
| tbl_serviceextinfo                         |       0.03 |
| tbl_servicegroup                           |       0.03 |
| tbl_servicetemplate                        |       0.03 |
| tbl_session                                |       0.02 |
| tbl_session_locks                          |       0.02 |
| tbl_settings                               |       0.03 |
| tbl_submenu                                |       0.02 |
| tbl_timedefinition                         |       0.02 |
| tbl_timeperiod                             |       0.03 |
| tbl_user                                   |       0.03 |
| tbl_variabledefinition                     |       0.02 |
| xi_auditlog                                |      53.63 |
| xi_auth_tokens                             |       0.03 |
| xi_cmp_ccm_backups                         |       0.02 |
| xi_cmp_favorites                           |       0.03 |
| xi_cmp_nagiosbpi_backups                   |       1.50 |
| xi_cmp_scheduledreports_log                |       0.02 |
| xi_cmp_trapdata                            |       0.50 |
| xi_cmp_trapdata_log                        |       0.03 |
| xi_commands                                |       0.02 |
| xi_deploy_agents                           |       0.02 |
| xi_deploy_jobs                             |       0.02 |
| xi_eventqueue                              |      18.20 |
| xi_events                                  |    1942.72 |
| xi_incidents                               |       0.02 |
| xi_meta                                    |   31427.00 |
| xi_mibs                                    |       0.05 |
| xi_options                                 |       0.06 |
| xi_sessions                                |       0.03 |
| xi_sysstat                                 |       0.03 |
| xi_usermeta                                |       0.17 |
| xi_users                                   |       0.03 |
+--------------------------------------------+------------+

And I have uploaded the profile via PM.

Thanks!

Posted: **Fri Jul 30, 2021 9:31 am**

These are temp tables that should be auto cleaned:

Code: Select all

| xi_events                                  |    1942.72 |
| xi_eventqueue                              |      18.20 |
| xi_meta                                    |   31427.00 |

Please run this command to fix it:

Code: Select all

echo "truncate table xi_events; truncate table xi_meta; truncate table xi_eventqueue;" | mysql -h X.X.X.X -uroot -pnagiosxi nagiosxi

Then reboot the XI server (recommended) or restart these services:

Code: Select all

systemctl restart crond nagios httpd

Then apply configuration and see if that resolves it.

If it doesn't, please create a ticket for this and include a link back to this forum thread so we can get a remote session setup:

https://support.nagios.com/tickets/

Nagios Support Forum

Error: no output from backend!

Error: no output from backend!

Re: Error: no output from backend!

Re: Error: no output from backend!

Re: Error: no output from backend!

Re: Error: no output from backend!

Re: Error: no output from backend!

Re: Error: no output from backend!

Re: Error: no output from backend!