Solaris 11 x86 NCPA Installation Failure

[Sargento]

Hello,

On two separate Solaris 11 x86 servers the official Solaris 11 NCPA Package fails to successfully run. 70% of the time the package says that it was installed successfully. The other 30% of the time the ncpa package fails to install. After the successful installations I see these services running.

Code: Select all

$ svcs | grep ncpa
> online         15:39:59 svc:/site/ncpa_passive:default
> maintenance    15:39:54 svc:/site/ncpa_listener:default

I attempt to clear and refresh this service..

Code: Select all

 
$ svcadm refresh ncpa_listener
$ svcadm clear ncpa_listener

The maintenance mode does not clear. I cannot add this client in Nagios XI as the service is not running. As stated before, I thought that perhaps this was an issue on a single server but the exact same thing has happened on two separate Solaris 11 servers. I get the below error in mailx after attempting the above clear.

Code: Select all

SUNW-MSG-ID: SMF-8000-YX, TYPE: defect, VER: 1, SEVERITY: major
EVENT-TIME: Wed Mar 24 11:37:14 CDT 2021
PLATFORM: (omitted)
SOURCE: software-diagnosis, REV: 0.1
EVENT-ID: 60d95d5d-37c9-e101-d618-9b61ad98b0db
DESC: A service failed - a method is failing in a retryable manner but too often.
AUTO-RESPONSE: The service has been placed into the maintenance state.
IMPACT: svc:/site/ncpa_passive:default is unavailable.
REC-ACTION: Run 'svcs -xv svc:/site/ncpa_passive:default' to determine the generic reason why the service failed, the location
of any logfiles, and a list of other services impacted. Please refer to the associated reference document at http://support.ora
cle.com/msg/SMF-8000-YX for the latest service procedures and policies regarding this diagnosis.

I cannot monitor Solaris servers as of right now due to the faulty packages.

I downloaded the ncpa packages from the official package repository at https://www.nagios.org/ncpa/#downloads
Thank you

[vtrac]

Hi,
First, if you could reboot your Solaris machine.

List installed NCPA package:

Code: Select all

pkg list ncpa*

Now, can you please run the followings:

Code: Select all

svcadm clear ncpa_listener

svcadm enable ncpa_listener

svcs -xv ncpa_listener

Please take a look at the log file, if ncpa_listener would not start:

Code: Select all

tail -100 /var/svc/log/application-ncpa_listener:default.log

Please upload all outputs from above commands here.


Regards,
Vinh

[Sargento]

We cannot afford an outage for the Solaris servers. I am sorry but it just cannot be done. I will provide you with the requested information though.

Here is the ncpa package information

Code: Select all

pkginfo -i ncpa
system      ncpa A monitoring agent (Nagios Cross-Platform Agent) for Nagios.

pkginfo -l ncpa
   PKGINST:  ncpa
      NAME:  A monitoring agent (Nagios Cross-Platform Agent) for Nagios.
  CATEGORY:  system
      ARCH:  i86pc
   VERSION:  Dev Release 02/12/2021
   BASEDIR:  /usr/local
    PSTAMP:  solaris20210212151544
  INSTDATE:  Mar 25 2021 16:19
    STATUS:  completely installed
     FILES:      170 installed pathnames
                  18 directories
                  72 executables
               75740 blocks used (approx)

Here is a look at the services. I have cleared and enabled them as you have noted for me to do. The maintenance did not clear.

Code: Select all

svcs | grep ncpa
maintenance    Mar_24   svc:/site/ncpa_passive:default
maintenance    16:21:34 svc:/site/ncpa_listener:default

svcadm clear ncpa_passive
svcadm enable ncpa_listener
svcadm enable ncpa_passive

svcs | grep ncpa
maintenance    16:21:34 svc:/site/ncpa_listener:default
maintenance    16:21:50 svc:/site/ncpa_passive:default

The attempted clear caused the below message on our server (as mail)

Code: Select all

From noaccess@(serverfqdn) Thu Mar 25 16:21:50 2021
Date: Thu, 25 Mar 2021 16:21:50 -0500 (CDT)
From: No Access User <(serverfqdn)>
X-FMEV-HOSTNAME: (hostname)
X-FMEV-CLASS: list.suspect
X-FMEV-UUID: f6de8a1f-8f8c-4b8c-9b4e-c51408acdff8
X-FMEV-CODE: SMF-8000-YX
X-FMEV-SEVERITY: major
Subject: Fault Management Event: (hostname):SMF-8000-YX
To: root@(server fqdn)

SUNW-MSG-ID: SMF-8000-YX, TYPE: defect, VER: 1, SEVERITY: major
EVENT-TIME: Thu Mar 25 16:21:50 CDT 2021
PLATFORM: Ultra-24, CSN: 0903FMB01M, HOSTNAME: (hostname)
SOURCE: software-diagnosis, REV: 0.1
EVENT-ID: f6de8a1f-8f8c-4b8c-9b4e-c51408acdff8
DESC: A service failed - a method is failing in a retryable manner but too often.
AUTO-RESPONSE: The service has been placed into the maintenance state.
IMPACT: svc:/site/ncpa_passive:default is unavailable.
REC-ACTION: Run 'svcs -xv svc:/site/ncpa_passive:default' to determine the generic reason why the service failed,
 the location of any logfiles, and a list of other services impacted. Please refer to the associated reference do
cument at http://support.oracle.com/msg/SMF-8000-YX for the latest service procedures and policies regarding this
 diagnosis.

The process is not running but to be thorough I ran the requested svcs command.

Code: Select all

svcs -xv ncpa_listener
svc:/site/ncpa_listener:default (?)
 State: maintenance since Thu Mar 25 16:21:34 2021
Reason: Start method failed repeatedly, last exited with status 1.
   See: http://support.oracle.com/msg/SMF-8000-KS
   See: /var/svc/log/site-ncpa_listener:default.log
Impact: This service is not running.

There appears to be no log for any ncpa service as the below command returns no text

Code: Select all

ls -1 /var/svc/log/application* | grep ncpa

[vtrac]

Hi,
Based on the "svc -vx ncpa_listener", the log is at:

Code: Select all

/var/svc/log/site-ncpa_listener:default.log

Please open the log for more details.


Regards,
Vinh

[Sargento]

Vinh,

Here is the log output you requested. Thank you for finding the location.

Code: Select all

[ Mar 17 15:22:45 Enabled. ]
[ Mar 17 15:22:45 Rereading configuration. ]
[ Mar 17 15:22:45 Executing start method ("/usr/local/ncpa/ncpa_listener"). ]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/cx_Freeze/initscripts/Console.py", line 27, in <module>
  File "ncpa_listener.py", line 11, in <module>
  File "/root/ncpa/agent/listener/certificate.py", line 2, in <module>
  File "/usr/local/lib/python2.7/site-packages/OpenSSL/__init__.py", line 8, in <module>
  File "/usr/local/lib/python2.7/site-packages/OpenSSL/rand.py", line 12, in <module>
  File "/usr/local/lib/python2.7/site-packages/OpenSSL/_util.py", line 6, in <module>
  File "/usr/local/lib/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 12, in <module>
  File "ExtensionLoader_cryptography_hazmat_bindings__openssl.py", line 25, in <module>
  File "ExtensionLoader_cryptography_hazmat_bindings__openssl.py", line 17, in __bootstrap__
ImportError: ld.so.1: ncpa_listener: fatal: relocation error: file /usr/local/ncpa/cryptography.hazmat.bindings._openssl.so: symbol TLSv1_2
_server_method: referenced symbol not found
[ Mar 17 15:22:46 Method "start" exited with status 1. ]
[ Mar 17 15:22:46 Executing start method ("/usr/local/ncpa/ncpa_listener"). ]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/cx_Freeze/initscripts/Console.py", line 27, in <module>
  File "ncpa_listener.py", line 11, in <module>
  File "/root/ncpa/agent/listener/certificate.py", line 2, in <module>
  File "/usr/local/lib/python2.7/site-packages/OpenSSL/__init__.py", line 8, in <module>
  File "/usr/local/lib/python2.7/site-packages/OpenSSL/rand.py", line 12, in <module>
  File "/usr/local/lib/python2.7/site-packages/OpenSSL/_util.py", line 6, in <module>
  File "/usr/local/lib/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 12, in <module>
  File "ExtensionLoader_cryptography_hazmat_bindings__openssl.py", line 25, in <module>
  File "ExtensionLoader_cryptography_hazmat_bindings__openssl.py", line 17, in __bootstrap__
ImportError: ld.so.1: ncpa_listener: fatal: relocation error: file /usr/local/ncpa/cryptography.hazmat.bindings._openssl.so: symbol TLSv1_2
_server_method: referenced symbol not found
[ Mar 17 15:22:46 Method "start" exited with status 1. ]
[ Mar 17 15:22:46 Executing start method ("/usr/local/ncpa/ncpa_listener"). ]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/cx_Freeze/initscripts/Console.py", line 27, in <module>
  File "ncpa_listener.py", line 11, in <module>
  File "/root/ncpa/agent/listener/certificate.py", line 2, in <module>
  File "/usr/local/lib/python2.7/site-packages/OpenSSL/__init__.py", line 8, in <module>
  File "/usr/local/lib/python2.7/site-packages/OpenSSL/rand.py", line 12, in <module>
  File "/usr/local/lib/python2.7/site-packages/OpenSSL/_util.py", line 6, in <module>
  File "/usr/local/lib/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 12, in <module>
  File "ExtensionLoader_cryptography_hazmat_bindings__openssl.py", line 25, in <module>
  File "ExtensionLoader_cryptography_hazmat_bindings__openssl.py", line 17, in __bootstrap__
ImportError: ld.so.1: ncpa_listener: fatal: relocation error: file /usr/local/ncpa/cryptography.hazmat.bindings._openssl.so: symbol TLSv1_2
_server_method: referenced symbol not found
[ Mar 17 15:22:47 Method "start" exited with status 1. ]
[ Mar 23 15:29:00 Leaving maintenance because disable requested. ]
[ Mar 23 15:29:00 Disabled. ]
[ Mar 23 15:33:18 Enabled. ]
[ Mar 23 15:33:18 Rereading configuration. ]
[ Mar 23 15:33:18 Executing start method ("/usr/local/ncpa/ncpa_listener"). ]
ld.so.1: ncpa_listener: fatal: libpython2.7.so.1.0: open failed: No such file or directory
[ Mar 23 15:33:18 Method "start" failed due to signal KILL. ]
[ Mar 23 15:33:18 Executing start method ("/usr/local/ncpa/ncpa_listener"). ]
ld.so.1: ncpa_listener: fatal: libpython2.7.so.1.0: open failed: No such file or directory
[ Mar 23 15:33:18 Method "start" failed due to signal KILL. ]
[ Mar 23 15:33:18 Executing start method ("/usr/local/ncpa/ncpa_listener"). ]
ld.so.1: ncpa_listener: fatal: libpython2.7.so.1.0: open failed: No such file or directory
[ Mar 23 15:33:18 Method "start" failed due to signal KILL. ]
[ Mar 23 15:39:11 Leaving maintenance because disable requested. ]
[ Mar 23 15:39:11 Disabled. ]
[ Mar 23 17:10:13 Enabled. ]
[ Mar 23 17:10:13 Rereading configuration. ]
[ Mar 23 17:10:14 Executing start method ("/usr/local/ncpa/ncpa_listener"). ]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/cx_Freeze/initscripts/Console.py", line 27, in <module>
  File "ncpa_listener.py", line 11, in <module>
  File "/root/ncpa/agent/listener/certificate.py", line 2, in <module>
  File "/usr/local/lib/python2.7/site-packages/OpenSSL/__init__.py", line 8, in <module>
  File "/usr/local/lib/python2.7/site-packages/OpenSSL/rand.py", line 12, in <module>
  File "/usr/local/lib/python2.7/site-packages/OpenSSL/_util.py", line 6, in <module>
  File "/usr/local/lib/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 12, in <module>
  File "ExtensionLoader_cryptography_hazmat_bindings__openssl.py", line 25, in <module>
  File "ExtensionLoader_cryptography_hazmat_bindings__openssl.py", line 17, in __bootstrap__
ImportError: ld.so.1: ncpa_listener: fatal: relocation error: file /usr/local/ncpa/cryptography.hazmat.bindings._openssl.so: symbol TLSv1_2
_server_method: referenced symbol not found
[ Mar 23 17:10:14 Method "start" exited with status 1. ]
[ Mar 23 17:10:14 Executing start method ("/usr/local/ncpa/ncpa_listener"). ]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/cx_Freeze/initscripts/Console.py", line 27, in <module>
  File "ncpa_listener.py", line 11, in <module>
  File "/root/ncpa/agent/listener/certificate.py", line 2, in <module>
  File "/usr/local/lib/python2.7/site-packages/OpenSSL/__init__.py", line 8, in <module>
  File "/usr/local/lib/python2.7/site-packages/OpenSSL/rand.py", line 12, in <module>
  File "/usr/local/lib/python2.7/site-packages/OpenSSL/_util.py", line 6, in <module>
  File "/usr/local/lib/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 12, in <module>
  File "ExtensionLoader_cryptography_hazmat_bindings__openssl.py", line 25, in <module>
  File "ExtensionLoader_cryptography_hazmat_bindings__openssl.py", line 17, in __bootstrap__
ImportError: ld.so.1: ncpa_listener: fatal: relocation error: file /usr/local/ncpa/cryptography.hazmat.bindings._openssl.so: symbol TLSv1_2
_server_method: referenced symbol not found
[ Mar 23 17:10:14 Method "start" exited with status 1. ]
[ Mar 23 17:10:14 Executing start method ("/usr/local/ncpa/ncpa_listener"). ]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/cx_Freeze/initscripts/Console.py", line 27, in <module>
  File "ncpa_listener.py", line 11, in <module>
  File "/root/ncpa/agent/listener/certificate.py", line 2, in <module>
  File "/usr/local/lib/python2.7/site-packages/OpenSSL/__init__.py", line 8, in <module>
  File "/usr/local/lib/python2.7/site-packages/OpenSSL/rand.py", line 12, in <module>
  File "/usr/local/lib/python2.7/site-packages/OpenSSL/_util.py", line 6, in <module>
  File "/usr/local/lib/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 12, in <module>
  File "ExtensionLoader_cryptography_hazmat_bindings__openssl.py", line 25, in <module>
  File "ExtensionLoader_cryptography_hazmat_bindings__openssl.py", line 17, in __bootstrap__
ImportError: ld.so.1: ncpa_listener: fatal: relocation error: file /usr/local/ncpa/cryptography.hazmat.bindings._openssl.so: symbol TLSv1_2
_server_method: referenced symbol not found
[ Mar 23 17:10:15 Method "start" exited with status 1. ]
[ Mar 24 08:33:18 Rereading configuration. ]
[ Mar 24 08:33:48 Leaving maintenance because clear requested. ]
[ Mar 24 08:33:48 Enabled. ]
[ Mar 24 08:33:48 Executing start method ("/usr/local/ncpa/ncpa_listener"). ]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/cx_Freeze/initscripts/Console.py", line 27, in <module>
  File "ncpa_listener.py", line 11, in <module>
  File "/root/ncpa/agent/listener/certificate.py", line 2, in <module>
  File "/usr/local/lib/python2.7/site-packages/OpenSSL/__init__.py", line 8, in <module>
  File "/usr/local/lib/python2.7/site-packages/OpenSSL/rand.py", line 12, in <module>
  File "/usr/local/lib/python2.7/site-packages/OpenSSL/_util.py", line 6, in <module>
  File "/usr/local/lib/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 12, in <module>
  File "ExtensionLoader_cryptography_hazmat_bindings__openssl.py", line 25, in <module>
  File "ExtensionLoader_cryptography_hazmat_bindings__openssl.py", line 17, in __bootstrap__
ImportError: ld.so.1: ncpa_listener: fatal: relocation error: file /usr/local/ncpa/cryptography.hazmat.bindings._openssl.so: symbol TLSv1_2
_server_method: referenced symbol not found
[ Mar 24 08:33:48 Method "start" exited with status 1. ]
[ Mar 24 08:44:19 Leaving maintenance because clear requested. ]
[ Mar 24 08:44:19 Enabled. ]
[ Mar 24 08:44:19 Executing start method ("/usr/local/ncpa/ncpa_listener"). ]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/cx_Freeze/initscripts/Console.py", line 27, in <module>
  File "ncpa_listener.py", line 11, in <module>
  File "/root/ncpa/agent/listener/certificate.py", line 2, in <module>
  File "/usr/local/lib/python2.7/site-packages/OpenSSL/__init__.py", line 8, in <module>
  File "/usr/local/lib/python2.7/site-packages/OpenSSL/rand.py", line 12, in <module>
  File "/usr/local/lib/python2.7/site-packages/OpenSSL/_util.py", line 6, in <module>
  File "/usr/local/lib/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 12, in <module>
  File "ExtensionLoader_cryptography_hazmat_bindings__openssl.py", line 25, in <module>
  File "ExtensionLoader_cryptography_hazmat_bindings__openssl.py", line 17, in __bootstrap__
ImportError: ld.so.1: ncpa_listener: fatal: relocation error: file /usr/local/ncpa/cryptography.hazmat.bindings._openssl.so: symbol TLSv1_2
_server_method: referenced symbol not found
[ Mar 24 08:44:20 Method "start" exited with status 1. ]
[ Mar 24 08:45:30 Leaving maintenance because clear requested. ]
[ Mar 24 08:45:30 Enabled. ]
[ Mar 24 08:45:30 Executing start method ("/usr/local/ncpa/ncpa_listener"). ]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/cx_Freeze/initscripts/Console.py", line 27, in <module>
  File "ncpa_listener.py", line 11, in <module>
  File "/root/ncpa/agent/listener/certificate.py", line 2, in <module>
  File "/usr/local/lib/python2.7/site-packages/OpenSSL/__init__.py", line 8, in <module>
  File "/usr/local/lib/python2.7/site-packages/OpenSSL/rand.py", line 12, in <module>
  File "/usr/local/lib/python2.7/site-packages/OpenSSL/_util.py", line 6, in <module>
  File "/usr/local/lib/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 12, in <module>
  File "ExtensionLoader_cryptography_hazmat_bindings__openssl.py", line 25, in <module>
  File "ExtensionLoader_cryptography_hazmat_bindings__openssl.py", line 17, in __bootstrap__
ImportError: ld.so.1: ncpa_listener: fatal: relocation error: file /usr/local/ncpa/cryptography.hazmat.bindings._openssl.so: symbol TLSv1_2
_server_method: referenced symbol not found
[ Mar 24 08:45:30 Method "start" exited with status 1. ]
[ Mar 24 11:36:32 Leaving maintenance because disable requested. ]
[ Mar 24 11:36:32 Disabled. ]
[ Mar 24 11:37:08 Enabled. ]
[ Mar 24 11:37:08 Rereading configuration. ]
[ Mar 24 11:37:08 Executing start method ("/usr/local/ncpa/ncpa_listener"). ]
ld.so.1: ncpa_listener: fatal: libpython2.7.so.1.0: open failed: No such file or directory
[ Mar 24 11:37:08 Method "start" failed due to signal KILL. ]
[ Mar 24 11:37:08 Executing start method ("/usr/local/ncpa/ncpa_listener"). ]
ld.so.1: ncpa_listener: fatal: libpython2.7.so.1.0: open failed: No such file or directory
[ Mar 24 11:37:08 Method "start" failed due to signal KILL. ]
[ Mar 24 11:37:08 Executing start method ("/usr/local/ncpa/ncpa_listener"). ]
ld.so.1: ncpa_listener: fatal: libpython2.7.so.1.0: open failed: No such file or directory
[ Mar 24 11:37:08 Method "start" failed due to signal KILL. ]
[ Mar 25 16:19:01 Rereading configuration. ]
[ Mar 25 16:21:34 Leaving maintenance because clear requested. ]
[ Mar 25 16:21:34 Enabled. ]
[ Mar 25 16:21:34 Executing start method ("/usr/local/ncpa/ncpa_listener"). ]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/cx_Freeze/initscripts/Console.py", line 27, in <module>
  File "ncpa_listener.py", line 8, in <module>
  File "/root/ncpa/agent/ncpadaemon.py", line 14, in <module>
  File "/root/ncpa/agent/listener/database.py", line 5, in <module>
  File "/root/ncpa/agent/listener/server.py", line 18, in <module>
  File "/usr/local/lib/python2.7/site-packages/gevent/__init__.py", line 51, in <module>
  File "/usr/local/lib/python2.7/site-packages/gevent/hub.py", line 13, in <module>
ImportError: ld.so.1: ncpa_listener: fatal: libpython2.7.so.1.0: open failed: No such file or directory
[ Mar 25 16:21:34 Method "start" exited with status 1. ]

[vtrac]

Hi,
Here's the error stated in the log:

Code: Select all

ImportError: ld.so.1: ncpa_listener: fatal: libpython2.7.so.1.0: open failed: No such file or directory

Looks like you are missing the "libpython2.7.so.1.0" library.

Please talk to your sys-admin to install
libpython2.7.so.1.0
or
python-lib

https://www.opencsw.org/packages/libpython2_7_1_0/

If you could, install the whole "python-lib" package would be better.

I could not create a Solaris 11 VM today, but my "libpython2.7.so.1.0" is located at (below) and belong to the "python-libs-2.7.5-88.el7.x86_64" RPM package .
(NOTE: just an example on my CentOS 7, Solaris might be located differently):

Code: Select all

/usr/lib64/libpython2.7.so.1.0

Regards,
Vinh

[Sargento]

We do have python on the Solaris server in question but it is not python 2.7. Are you saying that Python 2.7 is a requirement for NCPA to run on Solaris?

[vtrac]

Hi,
Here's your error:

Code: Select all

[ Mar 25 16:21:34 Enabled. ]
[ Mar 25 16:21:34 Executing start method ("/usr/local/ncpa/ncpa_listener"). ]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/cx_Freeze/initscripts/Console.py", line 27, in <module>
  File "ncpa_listener.py", line 8, in <module>
  File "/root/ncpa/agent/ncpadaemon.py", line 14, in <module>
  File "/root/ncpa/agent/listener/database.py", line 5, in <module>
  File "/root/ncpa/agent/listener/server.py", line 18, in <module>
  File "/usr/local/lib/python2.7/site-packages/gevent/__init__.py", line 51, in <module>
  File "/usr/local/lib/python2.7/site-packages/gevent/hub.py", line 13, in <module>
ImportError: ld.so.1: ncpa_listener: fatal: libpython2.7.so.1.0: open failed: No such file or directory
[ Mar 25 16:21:34 Method "start" exited with status 1. ]

As you can see, it is picking up "/usr/local/lib/python2.7/site-package" path.

You might have mutliple python installed.

What do you have under:

Code: Select all

/usr/local/lib/python2.7/site-packages/

Regards,
Vinh

[Sargento]

That directory does not exist. We only have Python 2.6 on these test servers.

[vtrac]

Hi,
Please upgrade your python to at least v2.7.

The official support for Python 2.6 has ended since October 29, 2013.


Regards,
Vinh