cannot read error from plugin to XI

[nag6200]

For all support requests, we need to know:

Answers to 1 and 2 apply to Nagios XI server and the remote linux OS

1. Linux Distribution and version?
$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.7 (Maipo)


2. 32 or 64bit?
$ uname -i
x86_64


3. VMware Image or Manual Install of XI?
Manual. This situation is only happening on one remote linux system.

4. Are there special configurations on your system, ie; is Gnome installed? Are you using a proxy? Are you using SSL?
No to all. Our security team did just move some IDs from local linux OS IDs to active directory IDs. We’ve not had any issues and just wanted to mention that.


5. **If you are encountering multiple issues that may not be related, start a thread for each issue



Hello Nagios Support,
I am receiving an unexpected permissions error in Nagios XI (version 5.6.6) for a new active check calling the check_log3.pl plugin using an NCPA agent on a remote linux server. The error message in XI is:
Cannot read ‘/tmp/nord’

When running the plugin locally from the remote linux server the check works:
$ pwd
/tmp
$ /usr/local/ncpa/plugins/check_log3.pl -l /tmp/nord -p hello
OK: Found 0 lines (limit=1/0): No matches found.|lines=0


Permissions are open on the target directory and the target file:
$ ls -ld /tmp/nord
-rw-rw-r--. 1 nagios nagios 7 Nov 1 16:50 /tmp/nord
$ ls -ld /tmp
drwxrwxrwt. 14 root root 4096 Nov 4 10:47 /tmp


And if works from the Nagios XI server from command line:

$ /usr/local/nagios/libexec/check_ncpa.py -H remote_linux_machine -t 'redacted' -P port -M 'plugins/check_log3.pl' -q 'args=-l /tmp/nord -p hello -show-filename -w 1 -c 1'
OK: Found 0 lines (limit=1/1): No matches found. [/tmp/nord]|lines=0

Works from web browser API page:
./check_ncpa.py -H remote_linux_machine -P port -t '<your token>' -M 'plugins/check_log3.pl' -q 'args=-l /tmp/nord -p hello'

OK: Found 0 lines (limit=1/0): No matches found.|lines=0

And it works from Nagios core config manager web page:
$ /usr/local/nagios/libexec/check_ncpa.py -H redacted -t 'redacted' -P port -M 'plugins/check_log3.pl' -q 'args=-l /tmp/nord -p WFLYHC0030: Connection to remote host -show-filename -w 1 -c 1'
OK: Found 0 lines (limit=1/1): No matches found. [/tmp/nord]|lines=0

The passive and local NCPA processes on the remote linux machine have been restarted and that did not help.

Any help would be appreciated.

Thanks,
Doug Nordhausen

System Profile
A system profile makes it easier for our support techs to understand the system that you are running on. Including a downloaded system profile with your support ticket is always recommended.
View System Info Download Profile
Nagios XI - System Info
System
Nagios XI version: 5.6.6
XI installed from: source
XI UUID: c61195ca-4507-4ac6-b2c1-d68c1b60d334
Release info: lnxvnaghq001.cinfin.com 3.10.0-1062.el7.x86_64 x86_64
Red Hat Enterprise Linux Server release 7.7 (Maipo)
Gnome is not installed

Apache Information
PHP Version: 5.4.16
Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36
Server Name: lnxvnaghq001.cinfin.com
Server Address: 10.224.4.32
Server Port: 443

Date/Time
PHP Timezone: US/Eastern
PHP Time: Mon, 04 Nov 2019 12:18:00 -0500
System Time: Mon, 04 Nov 2019 12:18:00 -0500

Nagios XI Data
License ends in: NNSQOV
UUID: c61195ca-4507-4ac6-b2c1-d68c1b60d334
Install Type: source

Nov 04 12:17:38 lnxvnaghq001.cinfin.com nagios[4304]: job 9877 (pid=33896): read() returned error 11
Nov 04 05:00:14 lnxvnaghq001.cinfin.com systemd[1]: Started SYSV: Visit the Website at http://sourceforge.net/projects/pnp4nagios/.
Nov 04 12:17:22 lnxvnaghq001.cinfin.com ndo2db[4310]: Trimming eventhandlers.
CPU Load 15: 6.26
Total Hosts: 2653
Total Services: 32706

Function get_base_uri() returns: https://lnxvnaghq001.cinfin.com/nagiosxi/
Function get_base_url() returns: https://lnxvnaghq001.cinfin.com/nagiosxi/
Function get_backend_url(internal_call=false) returns: https://lnxvnaghq001.cinfin.com/nagiosx ... rofile.php
Function get_backend_url(internal_call=true) returns: https://localhost/nagiosxi/backend/


Ping Test localhost
Running:
/bin/ping -c 3 localhost 2>&1
PING localhost (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.032 ms
64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.022 ms
64 bytes from localhost (127.0.0.1): icmp_seq=3 ttl=64 time=0.045 ms

--- localhost ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.022/0.033/0.045/0.009 ms

Test wget To localhost
WGET From URL: https://localhost/nagiosxi/includes/components/ccm/
Running:
/usr/bin/wget https://localhost/nagiosxi/includes/components/ccm/
--2019-11-04 12:18:02-- https://localhost/nagiosxi/includes/components/ccm/
Resolving localhost (localhost)... 127.0.0.1, ::1
Connecting to localhost (localhost)|127.0.0.1|:443... connected.
ERROR: cannot verify localhost's certificate, issued by '/DC=local/DC=CIC/CN=CIC-Intermediate-Authority-2':
Unable to locally verify the issuer's authority.
ERROR: no certificate subject alternative name matches
requested host name 'localhost'.
To connect to localhost insecurely, use `--no-check-certificate'.

Network Settings
1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo

valid_lft forever preferred_lft forever

2: ens192: mtu 1500 qdisc mq state UP group default qlen 1000

link/ether 00:50:56:b6:54:62 brd ff:ff:ff:ff:ff:ff

inet 10.224.4.32/24 brd 10.224.4.255 scope global noprefixroute ens192

valid_lft forever preferred_lft forever

3: ens224: mtu 1500 qdisc mq state UP group default qlen 1000

link/ether 00:50:56:b6:63:84 brd ff:ff:ff:ff:ff:ff

inet 10.224.26.35/22 brd 10.224.27.255 scope global noprefixroute ens224

valid_lft forever preferred_lft forever



default via 10.224.4.1 dev ens192 proto static metric 100

10.224.4.0/24 dev ens192 proto kernel scope link src 10.224.4.32 metric 100

10.224.24.0/22 dev ens224 proto kernel scope link src 10.224.26.35 metric 101



Nagios XI Components
actions 2.2.1
alertcloud 1.2.1
alertstream 2.1.1
autodiscovery 2.2.6
backendapiurl 1.0.5
bandwidthreport 1.8.1
bbmap 1.2.1
birdseye 3.2.4
bulkmodifications 2.2.0
capacityplanning 2.3.0
ccm 3.0.3
custom-includes 1.0.5
customlogin 1.0.0
customlogo 1.2.0
deploydashboard 1.3.0
deploynotification 1.3.3
duo 1.0.2
escalationwizard 1.5.1
freevariabletab 1.0.1
globaleventhandler 1.3.0
graphexplorer 2.2.2
helpsystem 2.0.1
highcharts 4.0.1
homepagemod 1.1.11
hypermap 1.2.1
hypermap_replay 1.2.0
isms 1.2.3
latestalerts 1.2.7
ldap_ad_integration 1.1.1
map 1.0.0
massacknowledge 2.2.1
massimmediatecheck 1.0.0
metrics 1.3.3
minemap 1.2.5
msp 1.1.0
mtr 1.0.1
nagiosbpi 2.8.0
nagioscore
nagioscorecfg
nagiosim 2.2.6
nagiosna 1.4.1
nagiosql
nagvis 2.0.3
nocscreen 1.3.1
nrdsconfigmanager 1.6.6
nxti 1.0.2
opscreen 1.8.0
perfdata
pingaction 1.1.2
pnp
profile 1.4.1
proxy 1.1.5
rdp 1.0.5
rename 1.7.0
scheduledbackups 1.2.0
scheduledreporting
similetimeline 1.5.1
snmptrapsender 1.6.1
statusmap 1.0.3
tracerouteaction 1.1.2
twilio 1.0.0
usermacros 1.1.0
xicore
Nagios XI Config Wizards
activedirectory 1.3.4
ec2 1.1.2
s3 1.1.0
java_tomcat 1.0.0
autodiscovery 1.4.2
bpiwizard 1.1.4
bulkhostimport 2.1.2
capacity-planning 1.0.1
dhcp 1.1.6
dnsquery 1.1.4
digitalocean 1.0.1
docker 1.1.0
domain_expiration 1.1.6
email-delivery 2.0.5
esensors_websensor 1.1.6
exchange 1.3.3
ftpserver 1.5.7
folder_watch 1.0.6
genericnetdevice 1.0.4
java_glassfish 1.0.0
google-cloud 1.0.1
hyperv 1.0.0
java_jboss 1.0.0
java_jetty 1.0.0
ldapserver 1.3.4
linode 1.0.1
linux_snmp 1.5.7
linux-server 1.5.7
mssql_database 1.6.4
mssql_query 1.6.6
mssql_server 1.9.2
macosx 1.3.2
mailserver 1.2.6
microsoft-azure 1.0.1
mongodb_database 1.1.3
mongodbserver 1.1.3
mountpoint 1.0.3
mysqlquery 1.2.4
mysqlserver 1.3.4
ncpa 2.0.6
nrpe 1.5.3
nagioslogserver 1.0.7
nna 1.0.6
nagiosxiserver 1.3.2
nagiostats 1.2.3
switch 2.5.1
oraclequery 1.3.5
oracleserverspace 1.5.5
oracletablespace 1.5.6
passivecheck 1.2.5
postgresdb 1.5.4
postgresquery 1.2.4
postgresserver 1.3.5
printer 1.1.4
radiusserver 2.0.3
rackspace 1.0.1
sla 1.3.4
snmp 1.6.4
snmp_trap 1.5.4
snmpwalk 1.4.2
sshproxy 1.5.8
solaris 1.3.1
tcpudpport 1.3.4
tftp 1.0.3
passiveobject 1.1.3
vmware 1.7.3
watchguard 1.4.6
webtransaction 1.2.5
java_weblogic 1.0.0
website 1.4.0
website_defacement 1.2.0
websiteurl 1.4.0
windowsdesktop 1.6.4
windowseventlog 2.0.0
windowssnmp 1.5.4
windowsserver 1.6.4
windowswmi 2.1.2
Nagios XI Dashlets
alertcloud
bbmap
capacityplanning
graphexplorer
hypermap
latestalerts
metrics
metricsguage
minemap
xicore_xi_news_feed
xicore_getting_started
xicore_admin_tasks
xicore_eventqueue_chart
xicore_component_status
xicore_server_stats
xicore_monitoring_stats
xicore_monitoring_perf
xicore_monitoring_process
xicore_perfdata_chart
xicore_host_status_summary
xicore_service_status_summary
xicore_comments
xicore_hostgroup_status_overview
xicore_hostgroup_status_grid
xicore_servicegroup_status_overview
xicore_servicegroup_status_grid
xicore_hostgroup_status_summary
xicore_servicegroup_status_summary
xicore_available_updates
xicore_network_outages
xicore_network_outages_summary
xicore_network_health
xicore_host_status_tac_summary
xicore_service_status_tac_summary
xicore_feature_status_tac_summary
availability
custom_dashlet 1.0.7
gauges 1.2.2
googlemapdashlet 1.1.0
internettrafficreport
rss_dashlet 1.1.5
sansrisingports 2.0
servicegroupservices 1.0
sla
statusinfo 2016-08-22
worldtimeserver 2.0.0

[lmiltchev]

It is very strange that you could run the command successfully from the command line, and from the CCM, but it fails in the GUI. Can you go to the CCM > Services > <your services> View Config, and show us the actual config?

[nag6200]

Sure, here it is, thanks.
Doug


###############################################################################
#
# Services configuration file
#
# Created by: Nagios CCM 3.0.3
# Date: 2019-11-05 11:33:33
# Version: Nagios Core 4.x
#
# --- DO NOT EDIT THIS FILE BY HAND ---
# Nagios CCM will overwrite all manual settings during the next update if you
# would like to edit files manually, place them in the 'static' directory or
# import your configs into the CCM by placing them in the 'import' directory.
#
###############################################################################

define service {
host_name host.domain.com
service_description CL3 PATROL - WFLYHC0030: Connection to remote host - /home/bmcpatrl/nord
check_command check_xi_ncpa!-t 'redacted' -P 5666 -M 'plugins/check_log3.pl' -q 'args=-l /home/bmcpatrl/nord -p WFLYHC0030: Connection to remote host -show-filename -w 1 -c 1'!!!!!!!
max_check_attempts 1
check_interval 2
retry_interval 1
active_checks_enabled 1
passive_checks_enabled 0
check_period xi_timeperiod_24x7
event_handler ServiceIssue_Event_Handler_Nagios
event_handler_enabled 1
notification_interval 60
notification_period xi_timeperiod_24x7
notifications_enabled 0
contacts nagiosadmin
_xiwizard ncpa
register 1
}

define service {
host_name host.domain.com
service_description look at /tmp/nord
check_command check_xi_ncpa!-t 'redacted' -P 5666 -M 'plugins/check_log3.pl' -q 'args=-l /tmp/nord -p WFLYHC0030: Connection to remote host -show-filename -w 1 -c 1'!!!!!!!
max_check_attempts 1
check_interval 2
retry_interval 1
active_checks_enabled 1
passive_checks_enabled 0
check_period xi_timeperiod_24x7
event_handler ServiceIssue_Event_Handler_Nagios
event_handler_enabled 1
notification_interval 60
notification_period xi_timeperiod_24x7
notifications_enabled 0
contacts nagiosadmin
_xiwizard ncpa
register 1
}

define service {
host_name host.domain.com
service_description NIC ens192 bytes_recv
use xiwizard_passive_service
max_check_attempts 1
check_interval 1
retry_interval 1
check_period xi_timeperiod_24x7
notification_interval 60
notification_period xi_timeperiod_24x7
notifications_enabled 0
contact_groups Linux_Server_Team
stalking_options n
_xiwizard passiveobject
register 1
}

define service {
host_name host.domain.com
service_description NIC ens192 bytes_sent
use xiwizard_passive_service
max_check_attempts 1
check_interval 1
retry_interval 1
check_period xi_timeperiod_24x7
notification_interval 60
notification_period xi_timeperiod_24x7
notifications_enabled 0
contact_groups Linux_Server_Team
stalking_options n
_xiwizard passiveobject
register 1
}

define service {
host_name host.domain.com
service_description NIC ens224 bytes_recv
use xiwizard_passive_service
max_check_attempts 1
check_interval 1
retry_interval 1
check_period xi_timeperiod_24x7
notification_interval 60
notification_period xi_timeperiod_24x7
notifications_enabled 0
contact_groups Linux_Server_Team
stalking_options n
_xiwizard passiveobject
register 1
}

define service {
host_name host.domain.com
service_description NIC ens224 bytes_sent
use xiwizard_passive_service
max_check_attempts 1
check_interval 1
retry_interval 1
check_period xi_timeperiod_24x7
notification_interval 60
notification_period xi_timeperiod_24x7
notifications_enabled 0
contact_groups Linux_Server_Team
stalking_options n
_xiwizard passiveobject
register 1
}

define service {
host_name host.domain.com
service_description Ping
use xiwizard_linuxserver_ping_service
check_command check-host-alive!!!!!!!!
max_check_attempts 5
check_interval 5
retry_interval 1
check_period xi_timeperiod_24x7
notification_interval 60
notification_period xi_timeperiod_24x7
notifications_enabled 0
contacts bpennin
_xiwizard linux_snmp
register 1
}

###############################################################################
#
# Services configuration file
#
# END OF FILE
#
###############################################################################

[lmiltchev]

Try wrapping the search string in double quotes:

Code: Select all

check_command check_xi_ncpa!-t 'redacted' -P 5666 -M 'plugins/check_log3.pl' -q 'args=-l /home/bmcpatrl/nord -p "WFLYHC0030: Connection to remote host" -show-filename -w 1 -c 1'!!!!!!!

Let us know if this fixes your issue.

[nag6200]

unfortunately no that didn't help -

check_command check_xi_ncpa!-t 'xxx' -P 1234 -M 'plugins/check_log3.pl' -q 'args=-l /tmp/nord -p "WFLYHC0030: Connection to remote host" -show-filename -w 1 -c 1'!!!!!!!


Regards,
Doug

[lmiltchev]

It is possible that when you were testing the plugin, a temp file (e.g."tmp-nord.seek") was created, which was owned by root.root, instead of nagios.nagios. If you find one on your remote system, remove it, then schedule an immediate check from the Nagios XI web UI to see if it is going to work. The "seek" file should be in the /tmp directory, but you can search the whole box by running:

Code: Select all

find / -name "*.seek"

Note: I tested a similar scenario (when the "nord" file was in the /tmp directory). I haven't tested one, with the file in someone's home directory... What are the permissions of "/home/bmcpatrl/" and "/home/bmcpatrl/nord"?

[nag6200]

Good grief.

We figured out the problem after realizing that all active checks to this server were failing. There was an incorrect IP address in the XI config for the remote server. Once that was corrected the check_log3 and all the other active checks started worked.

Thanks a lot for your help!
Doug

[mbellerue]

Ohh, that's the worst when that happens! But thank you for coming back with the info on what happened. I'll go ahead and lock the thread.