SMP has the error “(Return code of 127 is out of bounds. Che

[ppalmer]

Hello,

We have the error message SMP has the error “(Return code of 127 is out of bounds. Check if plugin exists)” on one of our Nagios boxes, this follows a problem with the mysql database which I am not sure is related. We rebuilt the database using the script /usr/local/nagiosxi/scripts/repair_databases.sh and it returned a succeeded message.

nagios database repair succeeded
nagiosql database repair succeeded
nagiosxi database repair succeeded

200415 16:20:48 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
200415 16:20:48 InnoDB: Initializing buffer pool, size = 8.0M
200415 16:20:48 InnoDB: Completed initialization of buffer pool
200415 16:20:48 InnoDB: Started; log sequence number 0 356365245
200415 16:20:48 [Note] Event Scheduler: Loaded 0 events
200415 16:20:48 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution

What can I do next to trouble shoot this issue?

Thanks.

[jbrunkow]

One way to check whether the plugin exists, is to look in /usr/local/nagios/libexec/ to see if the corresponding script is present.

This message could also be referring to an undefined argument in the script. If the value of an argument has not been set ( "$ARG1$" / "$ARG2" , etc... ), then you can set it in the nrpe.cfg file.

Please refer to the following linked knowledge base article for more information on return code of 127.
https://support.nagios.com/kb/article/n ... g-613.html

[ppalmer]

Unfortunately it doesn't detail which plugin could be missing, I have checked the directory and compared it with another server and I can't see anything obvious. How would I determine which plugin it is?

Thanks,

Paul

[ppalmer]

I have followed the link above and gone to investigate and we have a database error again:

Database Error
A database connection error has been detected, please follow the repair prompt below. If the issue persists, please contact Nagios support.
Run the following from the CLI as root to attempt to repair the DB:

/usr/local/nagiosxi/scripts/repair_databases.sh

Because we ran this last week I guess there is an ongoing problem.

[ssax]

Please run the repair.

Then, PM me a copy of your profile, you can download it from Admin > System Profile > Download Profile.

Additionally, please send the output of these commands (as root):
- NOTE: You may need to adjust the -h 127.0.0.1, the -uroot, and -pnagiosxi in the first command if your DB is offloaded to another server and/or you've changed the root mysql password

Code: Select all

echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -h 127.0.0.1 -uroot -pnagiosxi --table

Let us know the exact hostname/servicename in question so that we can find out which plugin it's using and where to go from there.

[ppalmer]

Hi Stuart,

We keep getting database errors so I can't login to the front end at the moment to download the profile. We have run the database repair script twice now and the error message has come back a 3rd time. Once I can get back in again I'll get the profile, I'll also get the output of the command below.

Thanks,

Paul

[ppalmer]

I refreshed the page and managed to get in with the following errors:

SQL Error [nagiosxi] : Got error 28 from storage engine


SQL Error [nagiosxi] : Table 'xi_incidents' already exists

[ppalmer]

Database rebuilt:

recovering (with sort) MyISAM-table 'xi_users'
Data records: 3
- Fixing index 1
- Fixing index 2
Starting mysqld: [ OK ]
/usr/local/nagiosxi/scripts

===============
REPAIR COMPLETE
===============
Stopping ndo2db: done.
Starting ndo2db: done.
Stopping nagios: done.
Starting nagios: done.

=======================
nagios database repair succeeded
nagiosql database repair succeeded
nagiosxi database repair succeeded

root@smpnagios-kn-u1 #

root@smpnagios-kn-u1 # echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -h 127.0.0.1 -uroot -pnagiosxi --table
+--------------------------------------------+------------+
| Table | Size in MB |
+--------------------------------------------+------------+
| nagios_acknowledgements | 0.05 |
| nagios_commands | 0.03 |
| nagios_commenthistory | 2.89 |
| nagios_comments | 0.00 |
| nagios_configfiles | 0.00 |
| nagios_configfilevariables | 0.01 |
| nagios_conninfo | 0.03 |
| nagios_contact_addresses | 0.00 |
| nagios_contact_notificationcommands | 0.01 |
| nagios_contactgroup_members | 0.00 |
| nagios_contactgroups | 0.00 |
| nagios_contactnotificationmethods | 25.14 |
| nagios_contactnotifications | 26.64 |
| nagios_contacts | 0.00 |
| nagios_contactstatus | 0.00 |
| nagios_customvariables | 0.00 |
| nagios_customvariablestatus | 0.00 |
| nagios_dbversion | 0.00 |
| nagios_downtimehistory | 0.00 |
| nagios_eventhandlers | 0.00 |
| nagios_externalcommands | 0.00 |
| nagios_flappinghistory | 1.84 |
| nagios_host_contactgroups | 0.00 |
| nagios_host_contacts | 0.00 |
| nagios_host_parenthosts | 0.00 |
| nagios_hostchecks | 0.00 |
| nagios_hostdependencies | 0.00 |
| nagios_hostescalation_contactgroups | 0.00 |
| nagios_hostescalation_contacts | 0.00 |
| nagios_hostescalations | 0.00 |
| nagios_hostgroup_members | 0.00 |
| nagios_hostgroups | 0.00 |
| nagios_hosts | 0.01 |
| nagios_hoststatus | 0.03 |
| nagios_instances | 0.00 |
| nagios_logentries | 705.76 |
| nagios_notifications | 66.29 |
| nagios_objects | 0.24 |
| nagios_processevents | 0.01 |
| nagios_programstatus | 0.00 |
| nagios_runtimevariables | 0.00 |
| nagios_scheduleddowntime | 0.00 |
| nagios_service_contactgroups | 0.02 |
| nagios_service_contacts | 0.00 |
| nagios_service_parentservices | 0.00 |
| nagios_servicechecks | 0.00 |
| nagios_servicedependencies | 0.00 |
| nagios_serviceescalation_contactgroups | 0.00 |
| nagios_serviceescalation_contacts | 0.00 |
| nagios_serviceescalations | 0.00 |
| nagios_servicegroup_members | 0.02 |
| nagios_servicegroups | 0.00 |
| nagios_services | 0.11 |
| nagios_servicestatus | 0.22 |
| nagios_statehistory | 72.71 |
| nagios_systemcommands | 0.02 |
| nagios_timedeventqueue | 0.00 |
| nagios_timedevents | 0.00 |
| nagios_timeperiod_timeranges | 0.00 |
| nagios_timeperiods | 0.00 |
| tbl_command | 0.03 |
| tbl_contact | 0.01 |
| tbl_contactgroup | 0.01 |
| tbl_contacttemplate | 0.01 |
| tbl_domain | 0.01 |
| tbl_host | 0.01 |
| tbl_hostdependency | 0.00 |
| tbl_hostescalation | 0.00 |
| tbl_hostextinfo | 0.00 |
| tbl_hostgroup | 0.01 |
| tbl_hosttemplate | 0.01 |
| tbl_info | 0.13 |
| tbl_lnkContactToCommandHost | 0.00 |
| tbl_lnkContactToCommandService | 0.00 |
| tbl_lnkContactToContactgroup | 0.00 |
| tbl_lnkContactToContacttemplate | 0.00 |
| tbl_lnkContactToVariabledefinition | 0.00 |
| tbl_lnkContactgroupToContact | 0.00 |
| tbl_lnkContactgroupToContactgroup | 0.00 |
| tbl_lnkContacttemplateToCommandHost | 0.00 |
| tbl_lnkContacttemplateToCommandService | 0.00 |
| tbl_lnkContacttemplateToContactgroup | 0.00 |
| tbl_lnkContacttemplateToContacttemplate | 0.00 |
| tbl_lnkContacttemplateToVariabledefinition | 0.00 |
| tbl_lnkHostToContact | 0.00 |
| tbl_lnkHostToContactgroup | 0.00 |
| tbl_lnkHostToHost | 0.00 |
| tbl_lnkHostToHostgroup | 0.00 |
| tbl_lnkHostToHosttemplate | 0.00 |
| tbl_lnkHostToVariabledefinition | 0.00 |
| tbl_lnkHostdependencyToHost_DH | 0.00 |
| tbl_lnkHostdependencyToHost_H | 0.00 |
| tbl_lnkHostdependencyToHostgroup_DH | 0.00 |
| tbl_lnkHostdependencyToHostgroup_H | 0.00 |
| tbl_lnkHostescalationToContact | 0.00 |
| tbl_lnkHostescalationToContactgroup | 0.00 |
| tbl_lnkHostescalationToHost | 0.00 |
| tbl_lnkHostescalationToHostgroup | 0.00 |
| tbl_lnkHostgroupToHost | 0.00 |
| tbl_lnkHostgroupToHostgroup | 0.00 |
| tbl_lnkHosttemplateToContact | 0.00 |
| tbl_lnkHosttemplateToContactgroup | 0.00 |
| tbl_lnkHosttemplateToHost | 0.00 |
| tbl_lnkHosttemplateToHostgroup | 0.00 |
| tbl_lnkHosttemplateToHosttemplate | 0.00 |
| tbl_lnkHosttemplateToVariabledefinition | 0.00 |
| tbl_lnkServiceToContact | 0.00 |
| tbl_lnkServiceToContactgroup | 0.00 |
| tbl_lnkServiceToHost | 0.00 |
| tbl_lnkServiceToHostgroup | 0.00 |
| tbl_lnkServiceToServicegroup | 0.00 |
| tbl_lnkServiceToServicetemplate | 0.00 |
| tbl_lnkServiceToVariabledefinition | 0.00 |
| tbl_lnkServicedependencyToHost_DH | 0.00 |
| tbl_lnkServicedependencyToHost_H | 0.00 |
| tbl_lnkServicedependencyToHostgroup_DH | 0.00 |
| tbl_lnkServicedependencyToHostgroup_H | 0.00 |
| tbl_lnkServicedependencyToService_DS | 0.00 |
| tbl_lnkServicedependencyToService_S | 0.00 |
| tbl_lnkServicedependencyToServicegroup_DS | 0.02 |
| tbl_lnkServicedependencyToServicegroup_S | 0.02 |
| tbl_lnkServiceescalationToContact | 0.00 |
| tbl_lnkServiceescalationToContactgroup | 0.00 |
| tbl_lnkServiceescalationToHost | 0.00 |
| tbl_lnkServiceescalationToHostgroup | 0.00 |
| tbl_lnkServiceescalationToService | 0.00 |
| tbl_lnkServiceescalationToServicegroup | 0.02 |
| tbl_lnkServicegroupToService | 0.00 |
| tbl_lnkServicegroupToServicegroup | 0.00 |
| tbl_lnkServicetemplateToContact | 0.00 |
| tbl_lnkServicetemplateToContactgroup | 0.00 |
| tbl_lnkServicetemplateToHost | 0.00 |
| tbl_lnkServicetemplateToHostgroup | 0.00 |
| tbl_lnkServicetemplateToServicegroup | 0.00 |
| tbl_lnkServicetemplateToServicetemplate | 0.01 |
| tbl_lnkServicetemplateToVariabledefinition | 0.00 |
| tbl_lnkTimeperiodToTimeperiod | 0.00 |
| tbl_logbook | 0.00 |
| tbl_mainmenu | 0.00 |
| tbl_permission | 0.02 |
| tbl_permission_inactive | 0.02 |
| tbl_service | 0.00 |
| tbl_servicedependency | 0.00 |
| tbl_serviceescalation | 0.00 |
| tbl_serviceextinfo | 0.00 |
| tbl_servicegroup | 0.00 |
| tbl_servicetemplate | 0.02 |
| tbl_session | 0.02 |
| tbl_session_locks | 0.02 |
| tbl_settings | 0.00 |
| tbl_submenu | 0.00 |
| tbl_timedefinition | 0.00 |
| tbl_timeperiod | 0.01 |
| tbl_user | 0.01 |
| tbl_variabledefinition | 0.00 |
| xi_auditlog | 0.01 |
| xi_auth_tokens | 0.03 |
| xi_cmp_trapdata | 0.03 |
| xi_cmp_trapdata_log | 0.03 |
| xi_commands | 0.00 |
| xi_eventqueue | 0.03 |
| xi_events | 0.00 |
| xi_incidents | 0.00 |
| xi_meta | 0.01 |
| xi_mibs | 0.05 |
| xi_options | 0.02 |
| xi_sessions | 0.03 |
| xi_sysstat | 0.01 |
| xi_usermeta | 0.02 |
| xi_users | 0.01 |
+--------------------------------------------+------------+
root@smpnagios-kn-u1 #

After that I tried to download the profile and got this error message:

PROFILE BUILD FAILED
Array
(
)

CODE: 1

I can view the system info though.

[benjaminsmith]

Hi @ppalmer,

Thanks for posting the sql query. The following error message:

PROFILE BUILD FAILED
Array
(
)
CODE: 1

It is typically the result of the sudoers file being incorrect on the XI server. Please follow the steps in the guide below and let us know if you're able to download the system profile. Thanks.

Nagios XI Profile Build Failure

[ppalmer]

We managed to fix the problem stopping the profile build. I have sent the profile to ssax via pm.

Since restarting the host server the mySQL database seems stable, we only have the original error.