Unable to get output on custom check

[lrnnetops]

Hi Team,

Nagios server OS - Centos 6.5 (64bit)
Nagios XI version - 5.4.8

I created custom check to monitor my oracle DB. in this case i have my own perl script which does the job.

1 - Set environment variable on nagios server.

-------------------
[nagios@laxco6nxi01 ~]$ echo $ORACLE_HOME
/usr/lib/oracle/11.2/client64
[nagios@laxco6nxi01 ~]$ echo $LD_LIBRARY_PATH
/usr/lib/oracle/11.2/client64/lib
[nagios@laxco6nxi01 ~]$ echo $ORACLE_BASE
/usr/lib/oracle
[nagios@laxco6nxi01 ~]$ echo $TNS_ADMIN
/usr/lib/oracle/11.2/client64/network/admin
-------------------

2 - I created command.

-------------------
Command Name = check_oracle.pl
Command line =$USER1$/check_oracle.pl -u $ARG1$ -p $ARG2$ -c $ARG3$ $ARG4$
-------------------

Applied the changes. All successfully submitted.

3 - Created a service.

Check command = check_oracle.pl
ARG1 = username
ARG2 = password
ARG3 = service name (tnsnames.ora)
ARG4 = thresholds

After that i clicked "Run Check command" & typed IP address of my DB server. Now the issue is its not showing me any output. where as i copied the command which i got after i clicked on "Run Check command" & executed on nagios server with user root as well as nagios & i got the desired output.

-------------------
# output from nagios webpage.

nagios-webpage-output.JPG

-------------------

# output from command line (user - root)

command-exec-root-user.JPG

-------------------

# output from command line (user - nagios)

command-exec-nagios-user.JPG

-------------------

I am not able to find out why the same command works when i execute it on nagios server from command line & why it don't work when gets executed by nagios check (webpage).

Please help.

Regards,
Rohan Talkar

[mcapra]

Can you share the complete contents of check_oracle.pl? It may be that the script is writing to something other than stdout, which is what Nagios reads for status information.

[scottwilkerson]

Also, with our other Oracle plugins you usually need to add the environment items to the command as nagios doesn't load plugins with a full user environment

Example commands for some of our other Oracle plugins

Code: Select all

/usr/bin/env LD_LIBRARY_PATH=/usr/lib/oracle/11.2/client64/lib ORACLE_HOME=/usr/lib/oracle/11.2/client64 $USER1$/check_oracle_health $ARG1$

[lrnnetops]

Hi mcapra,

Attached check_oracle.pl.

Hello scottwilkerson,

I have added environment variable you mentioned, but those worked for my few checks only. thats the reason we were using our custom check by "check_oracle.pl". issue i am facing is same command gets executed from nagios web page not producing output where as same command producing output when executes from shell.

Regards,
Rohan Talkar

[scottwilkerson]

Hi mcapra,

Attached check_oracle.pl.

Hello scottwilkerson,

I have added environment variable you mentioned, but those worked for my few checks only. thats the reason we were using our custom check by "check_oracle.pl". issue i am facing is same command gets executed from nagios web page not producing output where as same command producing output when executes from shell.

Regards,
Rohan Talkar

Unfortunately there is one thing we cannot overcome with running the commands from the web page, and that is the commands run as the apache user, vs. the nagios user. If the plugin required nagios user environment variables, it will only run from the command line, or through nagios which run as the nagios user.

[lrnnetops]

Hi mcapra,

Attached check_oracle.pl.

Hello scottwilkerson,

I have added environment variable you mentioned, but those worked for my few checks only. thats the reason we were using our custom check by "check_oracle.pl". issue i am facing is same command gets executed from nagios web page not producing output where as same command producing output when executes from shell.

Regards,
Rohan Talkar

Unfortunately there is one thing we cannot overcome with running the commands from the web page, and that is the commands run as the apache user, vs. the nagios user. If the plugin required nagios user environment variables, it will only run from the command line, or through nagios which run as the nagios user.

Hi scottwilkerson,

I wanted to say when i run command / check from nagios xi UI. I tested created service check from service management by clicking on "Run Check Command". but i got no output as below.

run_check_command.JPG

As mentioned earlier when we execute same check from linux shell (command line) with user "root" & "nagios" we receive desired output, but same configured check we execute from nagiosxi UI we get no response / output.

[root@laxco6nxi01 lib]# /usr/local/nagios/libexec/check_oracle.pl -u username -p password -c darlagp -d80/90
Max Extents OK 0.02 < 80 WARN< 90 CRITICAL

[nagios@laxco6nxi01 lib]$ /usr/local/nagios/libexec/check_oracle.pl -u username -p password -c darlagp -d80/90
Max Extents OK 0.02 < 80 WARN< 90 CRITICAL

our "check_oracle.pl" is provided earlier. permission for the "check_oracle.pl" as below.

[nagios@laxco6nxi01 ~]$ ll /usr/local/nagios/libexec/check_oracle.pl
-rwxr-xr-x 1 apache nagios 39080 Aug 28 08:07 /usr/local/nagios/libexec/check_oracle.pl

We are unable to trace the issue. Please assist.

Regards,
Rohan

[scottwilkerson]

As mentioned earlier when we execute same check from linux shell (command line) with user "root" & "nagios" we receive desired output, but same configured check we execute from nagiosxi UI we get no response / output.

As I had mentioned earlier

Unfortunately there is one thing we cannot overcome with running the commands from the web page, and that is the commands run as the apache user, vs. the nagios user. If the plugin required nagios user environment variables, it will only run from the command line, or through nagios which run as the nagios user.

So if the apache user doesn't have access to run the check you will have the issue you describe

[lrnnetops]

So if the apache user doesn't have access to run the check you will have the issue you describe

Hi Scott,

We have old nagios xi server from where we monitor mentioned check which we are unable to monitor from our newly configured nagios xi.

We replicated almost all settings eg. environment variable, permissions, checks templates etc... but still we not able to get the output on UI.

Sample output screenshot as below.

# Old server check 1 & 2 working

old_server-check1_and2-worked.JPG

# new server check 1 & 2 not working

new_server-check1_and2-not-worked.jpg

Here i am unable to understand when we click "Run check command" from nagios UI we see that command got executed with nagios user.

[nagios@laxco6nxi01 ~]$ /usr/local/nagios/libexec/check_oracle.pl -u username -p password -c alliagp1 -t85/90

Can you help us how can we directly execute below command with apache user so we can check for errors if a permission issue & can fix.

/usr/local/nagios/libexec/check_oracle.pl -u username -p password -c alliagp1 -t85/90

also let us know where can we find logs of the execution which we do by clicking on "Run check command" from UI.

also let us know location & name of a log file where "Run check command" write logs.


Regards,
Rohan

[npolovenko]

@lrnnetops, You could try to tail the mysql log while you execute the plugin check from the web UI:

tail -f /var/log/mysql.log

Also, can you post the screenshots of:

1)your plugin's permissions by running "ls -la" in /usr/local/nagios/libexec
2)run ls-la in /usr/lib/oracle
3)ls -la in /usr/lib/oracle/11.2/client64
4)ls -la in /usr/lib/oracle/11.2/client64/network/admin

Apache is a part of Nagios usergroup, so we need to make sure that nagios user and user group have access to all of the above directories, and have the proper permissions set up.

The plugin may have worked from the command line because you ran it as a root user.

[dwasswa]

Hi @lrnnetops,

Code: Select all

/usr/local/nagiosxi/var/cmdsubsys.log

This file logs the process of any commands passed to the Nagios XI backend/subsystem through the cmdsubsys cron. This
includes “Apply Configuration” or other Nagios XI specific commands.