Return code of 127 for service-upgrade to NagiosXI 5.5.9

[vishfx]

Hi Nagios Team,

Recently upgraded from NagiosXI 5.5.9 ( Nagios Core 4.2.4 + mod_gearman 2 ) to NagiosXI 5.6.14 ( Nagios Core 4.4.5 + mod_gearman 3 )
on OS : RHEL 7.8

Post upgrade, the Nagios service check for Rabbitmq is failing with (Return code of 127 is out of bounds. Check if plugin exists)

Below are further details :

nagiosxi-issue.JPG

I have tried the below steps:

1. changed plugin file 1st line from #!/usr/bin/env perl to #!/bin/perl

2. changed commands.cfg from
define command {
command_name check_rabbitmq_queue
command_line $USER1$/check_rabbitmq_queue $ARG1$
}

to

define command {
command_name check_rabbitmq_queue
command_line /bin/perl $USER1$/check_rabbitmq_queue $ARG1$
}


Both didn't help.

Kindly assist.

Regards,
Vish.

[cdienger]

Is cpsnxiptst01 the XI machine or one of the workers? Have you checked that the plugin is available on XI and all workers at /usr/local/nagios/libexec/check_rabbitmq_queue ?

Also, is https://github.com/nagios-plugins-rabbi ... itmq_queue the plugin?

[vishfx]

Hi ,

These service checks were working before upgrade.
The plugins are indeed on both nagiosxi & worker nodes.
But for testing, I have added the host group to be part of localhostgroup in gearman3.
So service check is running on nagiosxi node.

Yes, this is the plugin.

[cdienger]

Is there anything being logged in /usr/local/nagios/var/nagios.log or /var/log/messages? Check the xi logs as well by running:

Code: Select all

tail -f /usr/local/nagiosxi/var/*.log

[vishfx]

I only see the below message in the logs :

nagiosxi-issue-logs.JPG

Is there a way enable additional debuging options to see what exactly is failing ?
Were you able to replicate this issue ?

Just that you know the plugin works fine from bash cli.

Regards,
Vish.

[ssax]

Try changing this in your worker.conf and see if that works for you:

Code: Select all

enable_embedded_perl=off

Then restart the worker service.

You could enable debug logging in the /usr/local/nagios/etc/nagios.cfg:

Code: Select all

debug_file=/usr/local/nagios/var/nagios.debug
debug_level=-1
debug_verbosity=2

Then restart the nagios service.

Code: Select all

systemctl restart nagios

What is the output of this command?

Code: Select all

ls -l /usr/local/nagios/libexec/check_rabbitmq_queue

You can also enable debug on the worker as well.

Code: Select all

# use debug to increase the verbosity of the module.
# Possible values are:
#     0 = only errors
#     1 = debug messages
#     2 = trace messages
#     3 = trace and all gearman related logs are going to stdout.
# Default is 0.
debug=2

Please PM me a copy of your profile as well so I can look through the configs, you can download it from Admin > System Profile > Download Profile button.

[vishfx]

Try changing this in your worker.conf and see if that works for you:

Code: Select all

enable_embedded_perl=off

Then restart the worker service.

I have excluded the hostgroup from gearman checks. so the service checks are run locally on nagiosxi web node for easier troubleshooting.

PM'd you the settings.

You could enable debug logging in the /usr/local/nagios/etc/nagios.cfg:

Code: Select all

debug_file=/usr/local/nagios/var/nagios.debug
debug_level=-1
debug_verbosity=2

Have PM's nagios.debug & nagios.log

Then restart the nagios service.

Code: Select all

systemctl restart nagios

What is the output of this command?

Code: Select all

ls -l /usr/local/nagios/libexec/check_rabbitmq_queue

$ ls -l /usr/local/nagios/libexec/check_rabbitmq_queue
-rwxr-xr-x 1 nagios nagios 10670 Jun 10 04:58 /usr/local/nagios/libexec/check_rabbitmq_queue

You can also enable debug on the worker as well.

Code: Select all

# use debug to increase the verbosity of the module.
# Possible values are:
#     0 = only errors
#     1 = debug messages
#     2 = trace messages
#     3 = trace and all gearman related logs are going to stdout.
# Default is 0.
debug=2

Please PM me a copy of your profile as well so I can look through the configs, you can download it from Admin > System Profile > Download Profile button.

After clicking download button, Am getting the below error :
PROFILE BUILD FAILED
Array
(
)
CODE: 1

Have PM'd you systeminfo file.

We need to address this issue urgently, since this is blocking critical NagiosXI upgrades.
Can I do a screenshare session and show you the issue ?

Kindly assist.

Regards,
Vish.

[benjaminsmith]

Hi Vish,

Can I do a screenshare session and show you the issue ?

Let's get a support ticket opened for this issue, which will make it easier to book a remote session if necessary.

To open a support ticket, go to:

https://support.nagios.com/tickets/

After clicking download button, Am getting the below error :
PROFILE BUILD FAILED
Array
(
)
CODE: 1

This error is almost always caused by an incorrect sudoers file. Please follow the steps in the article below to resolve, and attach the system profile to the ticket. Thanks.

https://support.nagios.com/kb/article/n ... d-533.html