Nagios service runs from CLI but not from UI?

[brandondash]

I get output if I run the script from the command line; however, if I run the same script using the "Run Check Command", I get no output and no RRD gets updated/created. This was a service/command/script that was working fine last week and no changes were made AFAIK. Any idea how something like this could happen?

[pbroste]

Hello,

Thanks for reaching out and would like to take a look at your Nagios XI System Profile so we can see what is going on.

To send us your system profile.

• Login to the Nagios XI GUI using a web browser.
• Click the "Admin" > "System Profile" Menu
• Click the "Download Profile" button
• Save the profile.zip file and share

Thanks,
Perry

[benjaminsmith]

HI,

The "Run Check Command" button can fail if the command string has special characters. We usually recommend double-checking by running the check command from the CLI. If that works (as the Nagios user), it will run execute properly when called by the nagios service.

In addition to getting the system profile, let us know the exact name of the host or service so we can review the logs/configurations.

Besides performance graphs not updating properly, is the service monitoring as expected?

Regards,
Benjamin

[brandondash]

Thanks for the reply,

Unfortunately this is a closed circuit classified enclave, so I will not be able to provide the data files for which you are asking; however, I am willing to search them and pass on information as needed. What am I looking for? Generalized stack traces? Any sort of specific warning or error related to the host/service/rrd in question?

[ssax]

The Run Check Command button won't cause the backend performance graph RRDs to be updated unless that is specifically what your script is doing, for normal checks the performance data will only be submitted when the check is run from in the nagios backend and the check outputs performance data.

I would also setup the check and apply configuration to see if it works from the backend as there were some recent CCM Run Check Command hardening that could be impacting the Run Check Command functionality.

What plugin is it using? Can you attach it? What is the full command with arguments (sanitized information is fine).

Usually this would be related to environmental variables being missing that the script expects or permissions on something.

When testing from the CLI, did you su - to nagios first?

Code: Select all

su - nagios
/run/the/command

[brandondash]

So... this has grown in scope. Let me answer your questions first:

1.) This is a custom script that outputs perfdata as per the specification found here https://assets.nagios.com/downloads/nag ... fdata.html. I want to stress that this worked for years up until I posted my query.
2.) The script can run from CLI as root or nagios both. In both cases the output to stdout is exactly as we would expect. Also, since the script is run from the CLI the RRD does not get updated. This conforms with what we expect as well. I've never seen an RRD get updated or created when running an underlying script from CLI. I always thought that functionality was tied to something "under the hood" in the NagiosXI perfdata infrastructure.
3.) I'll be happy to link the service/command structure and configuration (sanitized of course) but it will be quite boring to see. They look like every other service or command that comes "out of the box" with NagiosXI.
4.) I want to restate the specific problem. When we run this service via the "Run Check Command" inside a service we get NO OUTPUT WHATSOEVER... not even the standard output. We have even put debug in the underlying script on the file system and get nothing happens at all.

Now... before we move forward with that I want to copy/paste another (probably much bigger) problem that may or may not be related. I had the production engineer run the steps you asked for to get the profile so that they could investigate and forward information. Here is their response (XXXX and YYYY are the NagiosXI servers at our various sites):

Brandon et al,
I followed the instructions you’ve listed below and was unable to retrieve the profile. When I clicked on Download profile on XXXX, I received “PROFILE BUILD FAILED, Array (), CODE: 1”, when I tried it on YYYY I didn’t receive anything, it looked like it was just sitting there. So I went back to XXXX and clicked on “View System info”. As I write this e-mail, it is going on 10 minutes and it still has not provided any output to the screen, it just keeps spinning.

Related? Unrelated? Easy to fix? New thread? Need guidance.

[pbroste]

Hello,

Since we will be hands-off on this one, could you take the command that you are using to output to the .rrd and watch it to see if you are receiving perfdata.

Example;

Code: Select all

/bin/echo -e "$LASTSERVICECHECK$\t$HOSTNAME$\t$SERVICEDESC$\t$SERVICESTATE$\t$SERVICEATTEMPT$\t$SERVICESTATETYPE$\t$SERVICEEXECUTIONTIME$\t$SERVICELATENCY$\t$SERVICEOUTPUT$\t$SERVICEPERFDATA$" >> /usr/local/nagios/var/service-perfdata.dat

And

Code: Select all

tail -F /usr/local/nagios/var/service-perfdata.dat

Should see data coming across and make sure the permissions and ownership on the service-perfdata.dat directory are associated with nagios:nagios.

We also understand that you are unable to pull the System Profile and want to provide the command line option:

Code: Select all

rm -rf /usr/local/nagiosxi/var/components/profile*
/usr/local/nagiosxi/scripts/components/getprofile.sh SUPPORT

The system profile can be found here /usr/local/nagiosxi/var/components/profile.zip

Thanks,
Perry

[brandondash]

After creating the profile manually and investigating, we found the problem. I'm not quite sure HOW this happened since it was only 2 out of 50 services, but it turns out the server was too low on space! Funny how something so obvious can get missed. You can close this thread. Thank you for the information and assistance.