Page 1 of 2

Unable to get output on custom check

Posted: Mon Aug 28, 2017 5:35 pm
by lrnnetops
Hi Team,

Nagios server OS - Centos 6.5 (64bit)
Nagios XI version - 5.4.8

I created custom check to monitor my oracle DB. in this case i have my own perl script which does the job.

1 - Set environment variable on nagios server.

-------------------
[nagios@laxco6nxi01 ~]$ echo $ORACLE_HOME
/usr/lib/oracle/11.2/client64
[nagios@laxco6nxi01 ~]$ echo $LD_LIBRARY_PATH
/usr/lib/oracle/11.2/client64/lib
[nagios@laxco6nxi01 ~]$ echo $ORACLE_BASE
/usr/lib/oracle
[nagios@laxco6nxi01 ~]$ echo $TNS_ADMIN
/usr/lib/oracle/11.2/client64/network/admin
-------------------

2 - I created command.

-------------------
Command Name = check_oracle.pl
Command line =$USER1$/check_oracle.pl -u $ARG1$ -p $ARG2$ -c $ARG3$ $ARG4$
-------------------

Applied the changes. All successfully submitted.

3 - Created a service.

Check command = check_oracle.pl
ARG1 = username
ARG2 = password
ARG3 = service name (tnsnames.ora)
ARG4 = thresholds

After that i clicked "Run Check command" & typed IP address of my DB server. Now the issue is its not showing me any output. where as i copied the command which i got after i clicked on "Run Check command" & executed on nagios server with user root as well as nagios & i got the desired output.

-------------------
# output from nagios webpage.
nagios-webpage-output.JPG
-------------------

# output from command line (user - root)
command-exec-root-user.JPG
-------------------

# output from command line (user - nagios)
command-exec-nagios-user.JPG
-------------------

I am not able to find out why the same command works when i execute it on nagios server from command line & why it don't work when gets executed by nagios check (webpage).

Please help.

Regards,
Rohan Talkar

Re: Unable to get output on custom check

Posted: Tue Aug 29, 2017 8:34 am
by mcapra
Can you share the complete contents of check_oracle.pl? It may be that the script is writing to something other than stdout, which is what Nagios reads for status information.

Re: Unable to get output on custom check

Posted: Tue Aug 29, 2017 9:39 am
by scottwilkerson
Also, with our other Oracle plugins you usually need to add the environment items to the command as nagios doesn't load plugins with a full user environment

Example commands for some of our other Oracle plugins

Code: Select all

/usr/bin/env LD_LIBRARY_PATH=/usr/lib/oracle/11.2/client64/lib ORACLE_HOME=/usr/lib/oracle/11.2/client64 $USER1$/check_oracle_health $ARG1$

Re: Unable to get output on custom check

Posted: Fri Sep 01, 2017 4:36 pm
by lrnnetops
Hi mcapra,

Attached check_oracle.pl.

Hello scottwilkerson,

I have added environment variable you mentioned, but those worked for my few checks only. thats the reason we were using our custom check by "check_oracle.pl". issue i am facing is same command gets executed from nagios web page not producing output where as same command producing output when executes from shell.

Regards,
Rohan Talkar

Re: Unable to get output on custom check

Posted: Fri Sep 01, 2017 4:41 pm
by scottwilkerson
lrnnetops wrote:Hi mcapra,

Attached check_oracle.pl.

Hello scottwilkerson,

I have added environment variable you mentioned, but those worked for my few checks only. thats the reason we were using our custom check by "check_oracle.pl". issue i am facing is same command gets executed from nagios web page not producing output where as same command producing output when executes from shell.

Regards,
Rohan Talkar
Unfortunately there is one thing we cannot overcome with running the commands from the web page, and that is the commands run as the apache user, vs. the nagios user. If the plugin required nagios user environment variables, it will only run from the command line, or through nagios which run as the nagios user.

Re: Unable to get output on custom check

Posted: Tue Oct 10, 2017 11:22 am
by lrnnetops
scottwilkerson wrote:
lrnnetops wrote:Hi mcapra,

Attached check_oracle.pl.

Hello scottwilkerson,

I have added environment variable you mentioned, but those worked for my few checks only. thats the reason we were using our custom check by "check_oracle.pl". issue i am facing is same command gets executed from nagios web page not producing output where as same command producing output when executes from shell.

Regards,
Rohan Talkar
Unfortunately there is one thing we cannot overcome with running the commands from the web page, and that is the commands run as the apache user, vs. the nagios user. If the plugin required nagios user environment variables, it will only run from the command line, or through nagios which run as the nagios user.
Hi scottwilkerson,

I wanted to say when i run command / check from nagios xi UI. I tested created service check from service management by clicking on "Run Check Command". but i got no output as below.
run_check_command.JPG
As mentioned earlier when we execute same check from linux shell (command line) with user "root" & "nagios" we receive desired output, but same configured check we execute from nagiosxi UI we get no response / output.

[root@laxco6nxi01 lib]# /usr/local/nagios/libexec/check_oracle.pl -u username -p password -c darlagp -d80/90
Max Extents OK 0.02 < 80 WARN< 90 CRITICAL

[nagios@laxco6nxi01 lib]$ /usr/local/nagios/libexec/check_oracle.pl -u username -p password -c darlagp -d80/90
Max Extents OK 0.02 < 80 WARN< 90 CRITICAL

our "check_oracle.pl" is provided earlier. permission for the "check_oracle.pl" as below.

[nagios@laxco6nxi01 ~]$ ll /usr/local/nagios/libexec/check_oracle.pl
-rwxr-xr-x 1 apache nagios 39080 Aug 28 08:07 /usr/local/nagios/libexec/check_oracle.pl

We are unable to trace the issue. Please assist.

Regards,
Rohan

Re: Unable to get output on custom check

Posted: Tue Oct 10, 2017 12:00 pm
by scottwilkerson
lrnnetops wrote:As mentioned earlier when we execute same check from linux shell (command line) with user "root" & "nagios" we receive desired output, but same configured check we execute from nagiosxi UI we get no response / output.
As I had mentioned earlier
scottwilkerson wrote:Unfortunately there is one thing we cannot overcome with running the commands from the web page, and that is the commands run as the apache user, vs. the nagios user. If the plugin required nagios user environment variables, it will only run from the command line, or through nagios which run as the nagios user.
So if the apache user doesn't have access to run the check you will have the issue you describe

Re: Unable to get output on custom check

Posted: Wed Oct 11, 2017 5:19 am
by lrnnetops

So if the apache user doesn't have access to run the check you will have the issue you describe
Hi Scott,

We have old nagios xi server from where we monitor mentioned check which we are unable to monitor from our newly configured nagios xi.

We replicated almost all settings eg. environment variable, permissions, checks templates etc... but still we not able to get the output on UI.

Sample output screenshot as below.

# Old server check 1 & 2 working
old_server-check1_and2-worked.JPG
# new server check 1 & 2 not working
new_server-check1_and2-not-worked.jpg
Here i am unable to understand when we click "Run check command" from nagios UI we see that command got executed with nagios user.

[nagios@laxco6nxi01 ~]$ /usr/local/nagios/libexec/check_oracle.pl -u username -p password -c alliagp1 -t85/90

Can you help us how can we directly execute below command with apache user so we can check for errors if a permission issue & can fix.

/usr/local/nagios/libexec/check_oracle.pl -u username -p password -c alliagp1 -t85/90

also let us know where can we find logs of the execution which we do by clicking on "Run check command" from UI.

also let us know location & name of a log file where "Run check command" write logs.


Regards,
Rohan

Re: Unable to get output on custom check

Posted: Wed Oct 11, 2017 2:54 pm
by npolovenko
@lrnnetops, You could try to tail the mysql log while you execute the plugin check from the web UI:
tail -f /var/log/mysql.log
Also, can you post the screenshots of:

1)your plugin's permissions by running "ls -la" in /usr/local/nagios/libexec
2)run ls-la in /usr/lib/oracle
3)ls -la in /usr/lib/oracle/11.2/client64
4)ls -la in /usr/lib/oracle/11.2/client64/network/admin

Apache is a part of Nagios usergroup, so we need to make sure that nagios user and user group have access to all of the above directories, and have the proper permissions set up.

The plugin may have worked from the command line because you ran it as a root user.

Re: Unable to get output on custom check

Posted: Wed Oct 11, 2017 3:15 pm
by dwasswa
Hi @lrnnetops,

Code: Select all

/usr/local/nagiosxi/var/cmdsubsys.log
This file logs the process of any commands passed to the Nagios XI backend/subsystem through the cmdsubsys cron. This
includes “Apply Configuration” or other Nagios XI specific commands.