AD LDAP Logins Fail after YUM update

[alexfertmann]

Hello,

I am running Nagios XI 5.5.2 on Centos 7.8 (installed on Physical HW, not a Virtual Appliance). We have AD LDAP configured for authentication with TLS. This has been working for some time. I have just done a yum update of the server and rebooted. After the server came back up, I was no longer able to login using AD LDAP. After some troubleshooting I found that the issue was related specifically to using TLS. When I disable TLS, I am able to login successfully. I following this troubleshooting guide to turn on debug logging: https://support.nagios.com/kb/article/a ... n-600.html

It seems that nagios cannot read the cert file. I thought maybe that the cert file become courrpt. I deleted it from nagios and while watching the logs it seems that there was an error deleting it from the filesystem but the web ui shows it as deleted. When I go to re-import it via the webui it shows as imported successfully but the debug log shows it could not create the file permission denied. Here are the debug logs from a login and adding/removing the cert:

==> /var/log/httpd/error_log <==
ldap_url_parse_ext(ldap://localhost/)
ldap_init: trying /etc/openldap/ldap.conf
ldap_init: using /etc/openldap/ldap.conf
ldap_init: HOME env is NULL
ldap_init: LDAPCONF env is NULL
ldap_init: LDAPRC env is NULL
ldap_create
ldap_extended_operation_s
ldap_extended_operation
ldap_send_initial_request
ldap_new_connection 1 1 0
ldap_int_open_connection
ldap_connect_to_host: TCP huemp16mce.employees.hofstra.univ:389
ldap_new_socket: 20
ldap_prepare_socket: 20
ldap_connect_to_host: Trying 10.20.4.78:389
ldap_pvt_connect: fd: 20 tm: -1 async: 0
attempting to connect:
connect success
ldap_open_defconn: successful
ldap_send_server_request
ldap_result ld 0x562923515960 msgid 1
wait4msg ld 0x562923515960 msgid 1 (infinite timeout)
wait4msg continue ld 0x562923515960 msgid 1 all 1
** ld 0x562923515960 Connections:
* host: huemp16mce.employees.hofstra.univ port: 389 (default)
refcnt: 2 status: Connected
last used: Mon May 18 14:22:30 2020


** ld 0x562923515960 Outstanding Requests:
* msgid 1, origid 1, status InProgress
outstanding referrals 0, parent count 0
ld 0x562923515960 request count 1 (abandoned 0)
** ld 0x562923515960 Response Queue:
Empty
ld 0x562923515960 response count 0
ldap_chkResponseList ld 0x562923515960 msgid 1 all 1
ldap_chkResponseList returns ld 0x562923515960 NULL
ldap_int_select
read1msg: ld 0x562923515960 msgid 1 all 1
read1msg: ld 0x562923515960 msgid 1 message type extended-result
read1msg: ld 0x562923515960 0 new referrals
read1msg: mark request completed, ld 0x562923515960 msgid 1
request done: ld 0x562923515960 msgid 1
res_errno: 0, res_error: <>, res_matched: <>
ldap_free_request (origid 1, msgid 1)
ldap_parse_extended_result
ldap_parse_result
ldap_msgfree
TLSMC: MozNSS compatibility interception begins.
tlsmc_intercept_initialization: INFO: entry options follow:
tlsmc_intercept_initialization: INFO: cacertdir = `/cacerts'
tlsmc_intercept_initialization: INFO: certfile = `(null)'
tlsmc_intercept_initialization: INFO: keyfile = `(null)'
tlsmc_convert: INFO: trying to open NSS DB with CACertDir = `/cacerts'.
tlsmc_open_nssdb: INFO: trying to initialize moznss using security dir `` prefix `cacerts`.
tlsmc_open_nssdb: WARN: could not initialize MozNSS context - error -8015.
tlsmc_convert: INFO: cannot open the NSS DB, expecting PEM configuration is present.
tlsmc_intercept_initialization: INFO: altered options follow:
tlsmc_intercept_initialization: INFO: cacertdir = `'
tlsmc_intercept_initialization: INFO: certfile = `(null)'
tlsmc_intercept_initialization: INFO: keyfile = `(null)'
tlsmc_intercept_initialization: INFO: successfully intercepted TLS initialization. Continuing with OpenSSL only.
TLSMC: MozNSS compatibility interception ends.
TLS: could not load verify locations (file:`/etc/openldap/certs/57ae1fc05c196.pem',dir:`').
TLS: error:0606506D:digital envelope routines:EVP_DecryptFinal_ex:wrong final block length evp_enc.c:581
TLS: error:02001002:system library:fopen:No such file or directory bss_file.c:175
TLS: error:2006D080:BIO routines:BIO_new_file:no such file bss_file.c:182
TLS: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib by_file.c:258
ldap_err2string

==> /var/log/httpd/ssl_error_log <==
[Mon May 18 14:22:30.460502 2020] [:error] [pid 8270] [client 10.20.16.26:23295] PHP Warning: ldap_start_tls(): Unable to start TLS: Connect error in /usr/local/nagiosxi/html/includes/components/ldap_ad_integration/adLDAP/src/adLDAP.php on line 641, referer: https://nagios2.hofstra.edu/nagiosxi/login.php

==> /var/log/httpd/error_log <==
ldap_bind_s
ldap_simple_bind_s
ldap_sasl_bind_s
ldap_sasl_bind
ldap_send_initial_request
ldap_send_server_request
ldap_result ld 0x562923515960 msgid 2
wait4msg ld 0x562923515960 msgid 2 (infinite timeout)
wait4msg continue ld 0x562923515960 msgid 2 all 1
** ld 0x562923515960 Connections:
* host: huemp16mce.employees.hofstra.univ port: 389 (default)
refcnt: 2 status: Connected
last used: Mon May 18 14:22:30 2020


** ld 0x562923515960 Outstanding Requests:
* msgid 2, origid 2, status InProgress
outstanding referrals 0, parent count 0
ld 0x562923515960 request count 1 (abandoned 0)
** ld 0x562923515960 Response Queue:
Empty
ld 0x562923515960 response count 0
ldap_chkResponseList ld 0x562923515960 msgid 2 all 1
ldap_chkResponseList returns ld 0x562923515960 NULL
ldap_int_select
read1msg: ld 0x562923515960 msgid 2 all 1
ldap_err2string

==> /var/log/httpd/ssl_error_log <==
[Mon May 18 14:22:30.461331 2020] [:error] [pid 8270] [client 10.20.16.26:23295] PHP Warning: ldap_bind(): Unable to bind to server: Can't contact LDAP server in /usr/local/nagiosxi/html/includes/components/ldap_ad_integration/adLDAP/src/adLDAP.php on line 714, referer: https://nagios2.hofstra.edu/nagiosxi/login.php

==> /var/log/httpd/error_log <==
ldap_free_request (origid 2, msgid 2)
ldap_free_connection 1 1
ldap_free_connection: actually freed
Error opening Certificate /etc/openldap/certs/5ec2d20559bd5.pem
140180742825872:error:02001002:system library:fopen:No such file or directory:bss_file.c:402:fopen('/etc/openldap/certs/5ec2d20559bd5.pem','r')
140180742825872:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:404:
unable to load certificate

==> /var/log/httpd/ssl_error_log <==
[Mon May 18 14:24:05.368684 2020] [:error] [pid 11592] [client 10.20.16.26:23336] PHP Warning: file_put_contents(/etc/openldap/certs/5ec2d2c559fd7.crt): failed to open stream: Permission denied in /usr/local/nagiosxi/html/includes/components/ldap_ad_integration/ajax.php on line 97, referer: https://nagios2.hofstra.edu/nagiosxi/in ... manage.php

==> /var/log/httpd/error_log <==
sh: /etc/openldap/certs/5ec2d2c559fd7.pem: Permission denied
Error opening Certificate /etc/openldap/certs/5ec2d2c559fd7.pem
139651854718864:error:02001002:system library:fopen:No such file or directory:bss_file.c:402:fopen('/etc/openldap/certs/5ec2d2c559fd7.pem','r')
139651854718864:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:404:
unable to load certificate

==> /var/log/httpd/ssl_error_log <==
[Mon May 18 14:24:05.401890 2020] [:error] [pid 11592] [client 10.20.16.26:23336] PHP Warning: opendir(/etc/openldap/cacerts): failed to open dir: No such file or directory in /usr/local/nagiosxi/html/includes/components/ldap_ad_integration/ajax.php on line 105, referer: https://nagios2.hofstra.edu/nagiosxi/in ... manage.php

==> /var/log/httpd/error_log <==
sh: line 0: cd: /etc/openldap/cacerts: No such file or directory
ln: failed to create symbolic link '.0': Permission denied
[Mon May 18 14:24:05.602636 2020] [mpm_prefork:notice] [pid 8259] AH00170: caught SIGWINCH, shutting down gracefully
[Mon May 18 14:24:08.911545 2020] [suexec:notice] [pid 13876] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Mon May 18 14:24:08.962476 2020] [lbmethod_heartbeat:notice] [pid 13876] AH02282: No slotmem from mod_heartmonitor
[Mon May 18 14:24:09.027281 2020] [mpm_prefork:notice] [pid 13876] AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/5.4.16 configured -- resuming normal operations
[Mon May 18 14:24:09.027331 2020] [core:notice] [pid 13876] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
Error opening Certificate /etc/openldap/certs/5ec2d2c559fd7.pem
139855208114064:error:02001002:system library:fopen:No such file or directory:bss_file.c:402:fopen('/etc/openldap/certs/5ec2d2c559fd7.pem','r')
139855208114064:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:404:
unable to load certificate

[root@nagios certs]#
[root@nagios certs]# tail -f /var/log/httpd/error_log /var/log/httpd/ssl_error_log
==> /var/log/httpd/error_log <==
ln: failed to create symbolic link '.0': Permission denied
[Mon May 18 14:24:05.602636 2020] [mpm_prefork:notice] [pid 8259] AH00170: caught SIGWINCH, shutting down gracefully
[Mon May 18 14:24:08.911545 2020] [suexec:notice] [pid 13876] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Mon May 18 14:24:08.962476 2020] [lbmethod_heartbeat:notice] [pid 13876] AH02282: No slotmem from mod_heartmonitor
[Mon May 18 14:24:09.027281 2020] [mpm_prefork:notice] [pid 13876] AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/5.4.16 configured -- resuming normal operations
[Mon May 18 14:24:09.027331 2020] [core:notice] [pid 13876] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
Error opening Certificate /etc/openldap/certs/5ec2d2c559fd7.pem
139855208114064:error:02001002:system library:fopen:No such file or directory:bss_file.c:402:fopen('/etc/openldap/certs/5ec2d2c559fd7.pem','r')
139855208114064:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:404:
unable to load certificate

==> /var/log/httpd/ssl_error_log <==
[Mon May 18 14:10:57.389103 2020] [:error] [pid 9529] [client 10.20.16.26:22995] PHP Warning: ldap_bind(): Unable to bind to server: Can't contact LDAP server in /usr/local/nagiosxi/html/includes/components/ldap_ad_integration/adLDAP/src/adLDAP.php on line 714, referer: https://nagios2.hofstra.edu/nagiosxi/login.php
[Mon May 18 14:15:05.086985 2020] [:error] [pid 9836] [client 10.20.16.26:23103] PHP Warning: ldap_bind(): Unable to bind to server: Can't contact LDAP server in /usr/local/nagiosxi/html/includes/components/ldap_ad_integration/adLDAP/src/adLDAP.php on line 714, referer: https://nagios2.hofstra.edu/nagiosxi/login.php
[Mon May 18 14:15:59.747547 2020] [:error] [pid 21245] [client 10.20.16.26:23129] PHP Warning: ldap_start_tls(): Unable to start TLS: Connect error in /usr/local/nagiosxi/html/includes/components/ldap_ad_integration/adLDAP/src/adLDAP.php on line 641, referer: https://nagios2.hofstra.edu/nagiosxi/login.php
[Mon May 18 14:15:59.760753 2020] [:error] [pid 21245] [client 10.20.16.26:23129] PHP Warning: ldap_bind(): Unable to bind to server: Can't contact LDAP server in /usr/local/nagiosxi/html/includes/components/ldap_ad_integration/adLDAP/src/adLDAP.php on line 714, referer: https://nagios2.hofstra.edu/nagiosxi/login.php
[Mon May 18 14:20:53.367650 2020] [:error] [pid 11437] [client 10.20.16.26:23240] PHP Warning: file_put_contents(/etc/openldap/certs/5ec2d20559bd5.crt): failed to open stream: Permission denied in /usr/local/nagiosxi/html/includes/components/ldap_ad_integration/ajax.php on line 97, referer: https://nagios2.hofstra.edu/nagiosxi/in ... manage.php
[Mon May 18 14:20:53.394569 2020] [:error] [pid 11437] [client 10.20.16.26:23240] PHP Warning: opendir(/etc/openldap/cacerts): failed to open dir: No such file or directory in /usr/local/nagiosxi/html/includes/components/ldap_ad_integration/ajax.php on line 105, referer: https://nagios2.hofstra.edu/nagiosxi/in ... manage.php
[Mon May 18 14:22:30.460502 2020] [:error] [pid 8270] [client 10.20.16.26:23295] PHP Warning: ldap_start_tls(): Unable to start TLS: Connect error in /usr/local/nagiosxi/html/includes/components/ldap_ad_integration/adLDAP/src/adLDAP.php on line 641, referer: https://nagios2.hofstra.edu/nagiosxi/login.php
[Mon May 18 14:22:30.461331 2020] [:error] [pid 8270] [client 10.20.16.26:23295] PHP Warning: ldap_bind(): Unable to bind to server: Can't contact LDAP server in /usr/local/nagiosxi/html/includes/components/ldap_ad_integration/adLDAP/src/adLDAP.php on line 714, referer: https://nagios2.hofstra.edu/nagiosxi/login.php
[Mon May 18 14:24:05.368684 2020] [:error] [pid 11592] [client 10.20.16.26:23336] PHP Warning: file_put_contents(/etc/openldap/certs/5ec2d2c559fd7.crt): failed to open stream: Permission denied in /usr/local/nagiosxi/html/includes/components/ldap_ad_integration/ajax.php on line 97, referer: https://nagios2.hofstra.edu/nagiosxi/in ... manage.php
[Mon May 18 14:24:05.401890 2020] [:error] [pid 11592] [client 10.20.16.26:23336] PHP Warning: opendir(/etc/openldap/cacerts): failed to open dir: No such file or directory in /usr/local/nagiosxi/html/includes/components/ldap_ad_integration/ajax.php on line 105, referer: https://nagios2.hofstra.edu/nagiosxi/in ... manage.php
[Mon May 18 14:30:03.970819 2020] [:error] [pid 16241] [client 10.20.16.26:23501] PHP Warning: file_put_contents(/etc/openldap/certs/5ec2d42becfda.crt): failed to open stream: Permission denied in /usr/local/nagiosxi/html/includes/components/ldap_ad_integration/ajax.php on line 97, referer: https://nagios2.hofstra.edu/nagiosxi/in ... manage.php

==> /var/log/httpd/error_log <==
sh: /etc/openldap/certs/5ec2d42becfda.pem: Permission denied
Error opening Certificate /etc/openldap/certs/5ec2d42becfda.pem
139749564106640:error:02001002:system library:fopen:No such file or directory:bss_file.c:402:fopen('/etc/openldap/certs/5ec2d42becfda.pem','r')
139749564106640:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:404:
unable to load certificate

==> /var/log/httpd/ssl_error_log <==
[Mon May 18 14:30:04.001926 2020] [:error] [pid 16241] [client 10.20.16.26:23501] PHP Warning: opendir(/etc/openldap/cacerts): failed to open dir: No such file or directory in /usr/local/nagiosxi/html/includes/components/ldap_ad_integration/ajax.php on line 105, referer: https://nagios2.hofstra.edu/nagiosxi/in ... manage.php

==> /var/log/httpd/error_log <==
sh: line 0: cd: /etc/openldap/cacerts: No such file or directory
ln: failed to create symbolic link '.0': Permission denied
[Mon May 18 14:30:04.412342 2020] [mpm_prefork:notice] [pid 13876] AH00170: caught SIGWINCH, shutting down gracefully
[Mon May 18 14:30:09.517507 2020] [suexec:notice] [pid 24722] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Mon May 18 14:30:09.568902 2020] [lbmethod_heartbeat:notice] [pid 24722] AH02282: No slotmem from mod_heartmonitor
[Mon May 18 14:30:09.633038 2020] [mpm_prefork:notice] [pid 24722] AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/5.4.16 configured -- resuming normal operations
[Mon May 18 14:30:09.633091 2020] [core:notice] [pid 24722] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
^C
[root@nagios certs]# cat /etc/centos-release
CentOS Linux release 7.8.2003 (Core)
[root@nagios certs]#

[cdienger]

What do the permissions look like if you run:

Code: Select all

ls -alh /etc/openldap/certs

Try setting the permissions:

Code: Select all

chown apache:nagios /etc/openldap/certs
chown apache:apache /etc/openldap/certs/*.crt
chown apache:apache /etc/openldap/certs/*.pem
chmod 644 /etc/openldap/certs/*.crt
chmod 644 /etc/openldap/certs/*.pem

[alexfertmann]

Hello -

The permissions were all set to root:root. I applied the commands you sent and it now looks like this:

[root@nagios openldap]# ls -alh /etc/openldap/certs/
total 64K
drwxr-xr-x. 2 apache nagios 86 May 20 10:23 .
drwxr-xr-x. 4 root root 63 Jan 29 2019 ..
-rw-r--r--. 1 root root 64K Aug 12 2016 cert8.db
-rw-r--r--. 1 root root 16K Aug 12 2016 key3.db
-r--------. 1 root root 45 Aug 12 2016 password
-rw-r--r--. 1 root root 16K Aug 12 2016 secmod.db
[root@nagios openldap]#


I then re-added our cert for ldap via the Web Interface. After that the folder looks like this:

[root@nagios openldap]# ls -alh /etc/openldap/certs/
total 80K
drwxr-xr-x. 2 apache nagios 4.0K May 20 10:26 .
drwxr-xr-x. 4 root root 63 Jan 29 2019 ..
-rw-r--r-- 1 apache apache 1.3K May 20 10:26 5ec53e06064cd.crt
-rw-r--r-- 1 apache apache 4.3K May 20 10:26 5ec53e06064cd.pem
-rw-r--r--. 1 root root 64K Aug 12 2016 cert8.db
-rw-r--r--. 1 root root 16K Aug 12 2016 key3.db
-r--------. 1 root root 45 Aug 12 2016 password
-rw-r--r--. 1 root root 16K Aug 12 2016 secmod.db
[root@nagios openldap]#


While re-adding the cert via the web, I noticed the following appear in the tailed log:

==> /var/log/httpd/error_log <==
sh: line 0: cd: /etc/openldap/cacerts: No such file or directory
ln: failed to create symbolic link '0469299f.0': Permission denied
[Wed May 20 10:26:14.924794 2020] [mpm_prefork:notice] [pid 6777] AH00170: caught SIGWINCH, shutting down gracefully
[Wed May 20 10:26:21.046014 2020] [suexec:notice] [pid 20846] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Wed May 20 10:26:21.088901 2020] [lbmethod_heartbeat:notice] [pid 20846] AH02282: No slotmem from mod_heartmonitor
[Wed May 20 10:26:21.148558 2020] [mpm_prefork:notice] [pid 20846] AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/5.4.16 configured -- resuming normal operations
[Wed May 20 10:26:21.148603 2020] [core:notice] [pid 20846] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'


==================================
==================================
I re-enabled TLS and attempted to login and still was unsuccessful. Below are the logs from the failed attempt.
==================================


==> /var/log/httpd/error_log <==
ldap_url_parse_ext(ldap://localhost/)
ldap_init: trying /etc/openldap/ldap.conf
ldap_init: using /etc/openldap/ldap.conf
ldap_init: HOME env is NULL
ldap_init: LDAPCONF env is NULL
ldap_init: LDAPRC env is NULL
ldap_create
ldap_extended_operation_s
ldap_extended_operation
ldap_send_initial_request
ldap_new_connection 1 1 0
ldap_int_open_connection
ldap_connect_to_host: TCP huemp16mce.employees.hofstra.univ:389
ldap_new_socket: 20
ldap_prepare_socket: 20
ldap_connect_to_host: Trying 10.20.4.22:389
ldap_pvt_connect: fd: 20 tm: -1 async: 0
attempting to connect:
connect success
ldap_open_defconn: successful
ldap_send_server_request
ldap_result ld 0x555d5136cf80 msgid 1
wait4msg ld 0x555d5136cf80 msgid 1 (infinite timeout)
wait4msg continue ld 0x555d5136cf80 msgid 1 all 1
** ld 0x555d5136cf80 Connections:
* host: huemp16mce.employees.hofstra.univ port: 389 (default)
refcnt: 2 status: Connected
last used: Wed May 20 10:30:10 2020


** ld 0x555d5136cf80 Outstanding Requests:
* msgid 1, origid 1, status InProgress
outstanding referrals 0, parent count 0
ld 0x555d5136cf80 request count 1 (abandoned 0)
** ld 0x555d5136cf80 Response Queue:
Empty
ld 0x555d5136cf80 response count 0
ldap_chkResponseList ld 0x555d5136cf80 msgid 1 all 1
ldap_chkResponseList returns ld 0x555d5136cf80 NULL
ldap_int_select
read1msg: ld 0x555d5136cf80 msgid 1 all 1
read1msg: ld 0x555d5136cf80 msgid 1 message type extended-result
read1msg: ld 0x555d5136cf80 0 new referrals
read1msg: mark request completed, ld 0x555d5136cf80 msgid 1
request done: ld 0x555d5136cf80 msgid 1
res_errno: 0, res_error: <>, res_matched: <>
ldap_free_request (origid 1, msgid 1)
ldap_parse_extended_result
ldap_parse_result
ldap_msgfree
TLSMC: MozNSS compatibility interception begins.
tlsmc_intercept_initialization: INFO: entry options follow:
tlsmc_intercept_initialization: INFO: cacertdir = `/cacerts'
tlsmc_intercept_initialization: INFO: certfile = `(null)'
tlsmc_intercept_initialization: INFO: keyfile = `(null)'
tlsmc_convert: INFO: trying to open NSS DB with CACertDir = `/cacerts'.
tlsmc_open_nssdb: INFO: trying to initialize moznss using security dir `` prefix `cacerts`.
tlsmc_open_nssdb: WARN: could not initialize MozNSS context - error -8015.
tlsmc_convert: INFO: cannot open the NSS DB, expecting PEM configuration is present.
tlsmc_intercept_initialization: INFO: altered options follow:
tlsmc_intercept_initialization: INFO: cacertdir = `'
tlsmc_intercept_initialization: INFO: certfile = `(null)'
tlsmc_intercept_initialization: INFO: keyfile = `(null)'
tlsmc_intercept_initialization: INFO: successfully intercepted TLS initialization. Continuing with OpenSSL only.
TLSMC: MozNSS compatibility interception ends.
TLS: could not load verify locations (file:`/etc/openldap/certs/57ae1fc05c196.pem',dir:`').
TLS: error:0606506D:digital envelope routines:EVP_DecryptFinal_ex:wrong final block length evp_enc.c:581
TLS: error:02001002:system library:fopen:No such file or directory bss_file.c:175
TLS: error:2006D080:BIO routines:BIO_new_file:no such file bss_file.c:182
TLS: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib by_file.c:258
ldap_err2string

==> /var/log/httpd/ssl_error_log <==
[Wed May 20 10:30:10.226390 2020] [:error] [pid 20991] [client 147.4.169.14:64015] PHP Warning: ldap_start_tls(): Unable to start TLS: Connect error in /usr/local/nagiosxi/html/includes/components/ldap_ad_integration/adLDAP/src/adLDAP.php on line 641, referer: https://nagios2.hofstra.edu/nagiosxi/login.php

==> /var/log/httpd/error_log <==
ldap_bind_s
ldap_simple_bind_s
ldap_sasl_bind_s
ldap_sasl_bind
ldap_send_initial_request
ldap_send_server_request
ldap_result ld 0x555d5136cf80 msgid 2
wait4msg ld 0x555d5136cf80 msgid 2 (infinite timeout)
wait4msg continue ld 0x555d5136cf80 msgid 2 all 1
** ld 0x555d5136cf80 Connections:
* host: huemp16mce.employees.hofstra.univ port: 389 (default)
refcnt: 2 status: Connected
last used: Wed May 20 10:30:10 2020


** ld 0x555d5136cf80 Outstanding Requests:
* msgid 2, origid 2, status InProgress
outstanding referrals 0, parent count 0
ld 0x555d5136cf80 request count 1 (abandoned 0)
** ld 0x555d5136cf80 Response Queue:
Empty
ld 0x555d5136cf80 response count 0
ldap_chkResponseList ld 0x555d5136cf80 msgid 2 all 1
ldap_chkResponseList returns ld 0x555d5136cf80 NULL
ldap_int_select
read1msg: ld 0x555d5136cf80 msgid 2 all 1
ldap_err2string
ldap_free_request (origid 2, msgid 2)
ldap_free_connection 1 1
ldap_free_connection: actually freed

==> /var/log/httpd/ssl_error_log <==
[Wed May 20 10:30:10.227100 2020] [:error] [pid 20991] [client 147.4.169.14:64015] PHP Warning: ldap_bind(): Unable to bind to server: Can't contact LDAP server in /usr/local/nagiosxi/html/includes/components/ldap_ad_integration/adLDAP/src/adLDAP.php on line 714, referer: https://nagios2.hofstra.edu/nagiosxi/login.php

==> /var/log/httpd/error_log <==


=============================================

From a quick look at the error it seems nagios is trying to use an old cert file that no longer exists:

TLS: could not load verify locations (file:`/etc/openldap/certs/57ae1fc05c196.pem',dir:`').
TLS: error:0606506D:digital envelope routines:EVP_DecryptFinal_ex:wrong final block length evp_enc.c:581
TLS: error:02001002:system library:fopen:No such file or directory bss_file.c:175
TLS: error:2006D080:BIO routines:BIO_new_file:no such file bss_file.c:182
TLS: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib by_file.c:258

***
/etc/openldap/certs/57ae1fc05c196.pem
***

I believe the proper cert file should be: 5ec53e06064cd.pem that got generated in the /etc/openldap/certs directory after re-adding the cert.


How do we fix this?

Thanks!

[cdienger]

I blieve it is having a problem creating the link in /etc/openldap/cacerts. Check the permissions:

Code: Select all

ls -alh /etc/openldap/cacerts

and set:

Code: Select all

chown apache:nagios /etc/openldap/cacerts
chmod 775 /etc/openldap/cacerts

[alexfertmann]

Hello -

The /etc/openldap/cacerts directory was missing. I checked a backup from before the upgrade and the cacerts directory didnt exist before the upgrade as well.

I created the cacerts directory using the commands and permissions you sent:

=====================================================================
[root@nagios openldap]# ls -alh /etc/openldap/cacerts
ls: cannot access /etc/openldap/cacerts: No such file or directory
[root@nagios openldap]#
[root@nagios openldap]# mkdir cacerts
[root@nagios openldap]# chown apache:nagios /etc/openldap/cacerts
[root@nagios openldap]# chmod 775 /etc/openldap/cacerts
[root@nagios openldap]# ls -alh
total 20K
drwxr-xr-x. 5 root root 81 May 21 13:36 .
drwxr-xr-x. 154 root root 8.0K May 18 13:47 ..
drwxrwxr-x 2 apache nagios 10 May 21 13:36 cacerts
drwxr-xr-x. 2 apache nagios 4.0K May 20 10:26 certs
-rw-rw-r--. 1 apache nagios 435 Aug 12 2016 ldap.conf
drwxr-xr-x. 2 root root 33 May 18 11:42 schema

============================================

After I did this, I removed the cert from the web UI and re-added it. After doing this I saw the cacerts folder populated with the link:

============================================
[root@nagios cacerts]# pwd
/etc/openldap/cacerts
[root@nagios cacerts]# ls -alh
total 0
drwxrwxr-x 2 apache nagios 31 May 21 13:41 .
drwxr-xr-x. 5 root root 81 May 21 13:36 ..
lrwxrwxrwx 1 apache apache 37 May 21 13:41 0469299f.0 -> /etc/openldap/certs/5ec6bd2d9b35e.pem
[root@nagios cacerts]#
============================================

Additionally, the debug log only generated the following during the import of the cert:

============================================
==> /var/log/httpd/error_log <==
[Thu May 21 13:41:02.512999 2020] [mpm_prefork:notice] [pid 4412] AH00170: caught SIGWINCH, shutting down gracefully
[Thu May 21 13:41:10.624007 2020] [suexec:notice] [pid 11551] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu May 21 13:41:10.676798 2020] [lbmethod_heartbeat:notice] [pid 11551] AH02282: No slotmem from mod_heartmonitor
[Thu May 21 13:41:10.744726 2020] [mpm_prefork:notice] [pid 11551] AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/5.4.16 configured -- resuming normal operations
[Thu May 21 13:41:10.744772 2020] [core:notice] [pid 11551] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
============================================

I then re-enabled TLS and attempted a login. The login still failed. See the debug log below:

============================================
ldap_url_parse_ext(ldap://localhost/)
ldap_init: trying /etc/openldap/ldap.conf
ldap_init: using /etc/openldap/ldap.conf
ldap_init: HOME env is NULL
ldap_init: LDAPCONF env is NULL
ldap_init: LDAPRC env is NULL
ldap_create
ldap_extended_operation_s
ldap_extended_operation
ldap_send_initial_request
ldap_new_connection 1 1 0
ldap_int_open_connection
ldap_connect_to_host: TCP huemp16syr.employees.hofstra.univ:389
ldap_new_socket: 20
ldap_prepare_socket: 20
ldap_connect_to_host: Trying 172.17.2.16:389
ldap_pvt_connect: fd: 20 tm: -1 async: 0
attempting to connect:
connect success
ldap_open_defconn: successful
ldap_send_server_request
ldap_result ld 0x559b171d1080 msgid 1
wait4msg ld 0x559b171d1080 msgid 1 (infinite timeout)
wait4msg continue ld 0x559b171d1080 msgid 1 all 1
** ld 0x559b171d1080 Connections:
* host: huemp16syr.employees.hofstra.univ port: 389 (default)
refcnt: 2 status: Connected
last used: Thu May 21 13:42:18 2020


** ld 0x559b171d1080 Outstanding Requests:
* msgid 1, origid 1, status InProgress
outstanding referrals 0, parent count 0
ld 0x559b171d1080 request count 1 (abandoned 0)
** ld 0x559b171d1080 Response Queue:
Empty
ld 0x559b171d1080 response count 0
ldap_chkResponseList ld 0x559b171d1080 msgid 1 all 1
ldap_chkResponseList returns ld 0x559b171d1080 NULL
ldap_int_select
read1msg: ld 0x559b171d1080 msgid 1 all 1
read1msg: ld 0x559b171d1080 msgid 1 message type extended-result
read1msg: ld 0x559b171d1080 0 new referrals
read1msg: mark request completed, ld 0x559b171d1080 msgid 1
request done: ld 0x559b171d1080 msgid 1
res_errno: 0, res_error: <>, res_matched: <>
ldap_free_request (origid 1, msgid 1)
ldap_parse_extended_result
ldap_parse_result
ldap_msgfree
TLSMC: MozNSS compatibility interception begins.
tlsmc_intercept_initialization: INFO: entry options follow:
tlsmc_intercept_initialization: INFO: cacertdir = `/cacerts'
tlsmc_intercept_initialization: INFO: certfile = `(null)'
tlsmc_intercept_initialization: INFO: keyfile = `(null)'
tlsmc_convert: INFO: trying to open NSS DB with CACertDir = `/cacerts'.
tlsmc_open_nssdb: INFO: trying to initialize moznss using security dir `` prefix `cacerts`.
tlsmc_open_nssdb: WARN: could not initialize MozNSS context - error -8015.
tlsmc_convert: INFO: cannot open the NSS DB, expecting PEM configuration is present.
tlsmc_intercept_initialization: INFO: altered options follow:
tlsmc_intercept_initialization: INFO: cacertdir = `'
tlsmc_intercept_initialization: INFO: certfile = `(null)'
tlsmc_intercept_initialization: INFO: keyfile = `(null)'
tlsmc_intercept_initialization: INFO: successfully intercepted TLS initialization. Continuing with OpenSSL only.
TLSMC: MozNSS compatibility interception ends.
TLS: could not load verify locations (file:`/etc/openldap/certs/57ae1fc05c196.pem',dir:`').
TLS: error:0606506D:digital envelope routines:EVP_DecryptFinal_ex:wrong final block length evp_enc.c:581
TLS: error:02001002:system library:fopen:No such file or directory bss_file.c:175
TLS: error:2006D080:BIO routines:BIO_new_file:no such file bss_file.c:182
TLS: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib by_file.c:258
ldap_err2string

==> /var/log/httpd/ssl_error_log <==
[Thu May 21 13:42:18.642865 2020] [:error] [pid 11734] [client 10.20.16.26:61331] PHP Warning: ldap_start_tls(): Unable to start TLS: Connect error in /usr/local/nagiosxi/html/includes/components/ldap_ad_integration/adLDAP/src/adLDAP.php on line 641, referer: https://nagios2.hofstra.edu/nagiosxi/lo ... f&noauth=1

==> /var/log/httpd/error_log <==
ldap_bind_s
ldap_simple_bind_s
ldap_sasl_bind_s
ldap_sasl_bind
ldap_send_initial_request
ldap_send_server_request
ldap_result ld 0x559b171d1080 msgid 2
wait4msg ld 0x559b171d1080 msgid 2 (infinite timeout)
wait4msg continue ld 0x559b171d1080 msgid 2 all 1
** ld 0x559b171d1080 Connections:
* host: huemp16syr.employees.hofstra.univ port: 389 (default)
refcnt: 2 status: Connected
last used: Thu May 21 13:42:18 2020


** ld 0x559b171d1080 Outstanding Requests:
* msgid 2, origid 2, status InProgress
outstanding referrals 0, parent count 0
ld 0x559b171d1080 request count 1 (abandoned 0)
** ld 0x559b171d1080 Response Queue:
Empty
ld 0x559b171d1080 response count 0
ldap_chkResponseList ld 0x559b171d1080 msgid 2 all 1
ldap_chkResponseList returns ld 0x559b171d1080 NULL
ldap_int_select
read1msg: ld 0x559b171d1080 msgid 2 all 1
ldap_err2string
ldap_free_request (origid 2, msgid 2)
ldap_free_connection 1 1
ldap_free_connection: actually freed

==> /var/log/httpd/ssl_error_log <==
[Thu May 21 13:42:18.655712 2020] [:error] [pid 11734] [client 10.20.16.26:61331] PHP Warning: ldap_bind(): Unable to bind to server: Can't contact LDAP server in /usr/local/nagiosxi/html/includes/components/ldap_ad_integration/adLDAP/src/adLDAP.php on line 714, referer: https://nagios2.hofstra.edu/nagiosxi/lo ... f&noauth=1

==> /var/log/httpd/error_log <==

==================================================

Looking at the debug log, nagios is still looking at a non-existant certificate file:
===================
TLS: could not load verify locations (file:`/etc/openldap/certs/57ae1fc05c196.pem',dir:`').
===================

The cert file should be: /etc/openldap/certs/5ec6bd2d9b35e.pem


As a test, I copied 5ec6bd2d9b35e.pem and 5ec6bd2d9b35e.crt to create 57ae1fc05c196.pem and 57ae1fc05c196.crt.
I then changed the ownership and permissions to match that of 5ec6bd2d9b35e.pem and 5ec6bd2d9b35e.crt:

======================================================
[root@nagios certs]# cp 5ec6bd2d9b35e.crt 57ae1fc05c196.crt
[root@nagios certs]# cp 5ec6bd2d9b35e.pem 57ae1fc05c196.pem
[root@nagios certs]# ls -alh
total 92K
drwxr-xr-x. 2 apache nagios 4.0K May 21 13:56 .
drwxr-xr-x. 5 root root 81 May 21 13:36 ..
-rw-r--r-- 1 root root 1.3K May 21 13:56 57ae1fc05c196.crt
-rw-r--r-- 1 root root 4.3K May 21 13:56 57ae1fc05c196.pem
-rw-r--r-- 1 apache apache 1.3K May 21 13:41 5ec6bd2d9b35e.crt
-rw-r--r-- 1 apache apache 4.3K May 21 13:41 5ec6bd2d9b35e.pem
-rw-r--r--. 1 root root 64K Aug 12 2016 cert8.db
-rw-r--r--. 1 root root 16K Aug 12 2016 key3.db
-r--------. 1 root root 45 Aug 12 2016 password
-rw-r--r--. 1 root root 16K Aug 12 2016 secmod.db
[root@nagios certs]# chown apache:nagios 57ae1fc05c196.*
[root@nagios certs]# chmod 644 57ae1fc05c196.*
[root@nagios certs]# ls -alh
total 92K
drwxr-xr-x. 2 apache nagios 4.0K May 21 13:56 .
drwxr-xr-x. 5 root root 81 May 21 13:36 ..
-rw-r--r-- 1 apache nagios 1.3K May 21 13:56 57ae1fc05c196.crt
-rw-r--r-- 1 apache nagios 4.3K May 21 13:56 57ae1fc05c196.pem
-rw-r--r-- 1 apache apache 1.3K May 21 13:41 5ec6bd2d9b35e.crt
-rw-r--r-- 1 apache apache 4.3K May 21 13:41 5ec6bd2d9b35e.pem
-rw-r--r--. 1 root root 64K Aug 12 2016 cert8.db
-rw-r--r--. 1 root root 16K Aug 12 2016 key3.db
-r--------. 1 root root 45 Aug 12 2016 password
-rw-r--r--. 1 root root 16K Aug 12 2016 secmod.db
[root@nagios certs]#
==========================================================
I then re-attempted a login. I still could not login and the error looks similar:

=====================================
ldap_url_parse_ext(ldap://localhost/)
ldap_init: trying /etc/openldap/ldap.conf
ldap_init: using /etc/openldap/ldap.conf
ldap_init: HOME env is NULL
ldap_init: LDAPCONF env is NULL
ldap_init: LDAPRC env is NULL
ldap_create
ldap_extended_operation_s
ldap_extended_operation
ldap_send_initial_request
ldap_new_connection 1 1 0
ldap_int_open_connection
ldap_connect_to_host: TCP huemp16sc.employees.hofstra.univ:389
ldap_new_socket: 20
ldap_prepare_socket: 20
ldap_connect_to_host: Trying 10.21.4.12:389
ldap_pvt_connect: fd: 20 tm: -1 async: 0
attempting to connect:
connect success
ldap_open_defconn: successful
ldap_send_server_request
ldap_result ld 0x559b16f3cde0 msgid 1
wait4msg ld 0x559b16f3cde0 msgid 1 (infinite timeout)
wait4msg continue ld 0x559b16f3cde0 msgid 1 all 1
** ld 0x559b16f3cde0 Connections:
* host: huemp16sc.employees.hofstra.univ port: 389 (default)
refcnt: 2 status: Connected
last used: Thu May 21 14:02:44 2020


** ld 0x559b16f3cde0 Outstanding Requests:
* msgid 1, origid 1, status InProgress
outstanding referrals 0, parent count 0
ld 0x559b16f3cde0 request count 1 (abandoned 0)
** ld 0x559b16f3cde0 Response Queue:
Empty
ld 0x559b16f3cde0 response count 0
ldap_chkResponseList ld 0x559b16f3cde0 msgid 1 all 1
ldap_chkResponseList returns ld 0x559b16f3cde0 NULL
ldap_int_select
read1msg: ld 0x559b16f3cde0 msgid 1 all 1
read1msg: ld 0x559b16f3cde0 msgid 1 message type extended-result
read1msg: ld 0x559b16f3cde0 0 new referrals
read1msg: mark request completed, ld 0x559b16f3cde0 msgid 1
request done: ld 0x559b16f3cde0 msgid 1
res_errno: 0, res_error: <>, res_matched: <>
ldap_free_request (origid 1, msgid 1)
ldap_parse_extended_result
ldap_parse_result
ldap_msgfree
TLSMC: MozNSS compatibility interception begins.
tlsmc_intercept_initialization: INFO: entry options follow:
tlsmc_intercept_initialization: INFO: cacertdir = `/cacerts'
tlsmc_intercept_initialization: INFO: certfile = `(null)'
tlsmc_intercept_initialization: INFO: keyfile = `(null)'
tlsmc_convert: INFO: trying to open NSS DB with CACertDir = `/cacerts'.
tlsmc_open_nssdb: INFO: trying to initialize moznss using security dir `` prefix `cacerts`.
tlsmc_open_nssdb: WARN: could not initialize MozNSS context - error -8015.
tlsmc_convert: INFO: cannot open the NSS DB, expecting PEM configuration is present.
tlsmc_intercept_initialization: INFO: altered options follow:
tlsmc_intercept_initialization: INFO: cacertdir = `'
tlsmc_intercept_initialization: INFO: certfile = `(null)'
tlsmc_intercept_initialization: INFO: keyfile = `(null)'
tlsmc_intercept_initialization: INFO: successfully intercepted TLS initialization. Continuing with OpenSSL only.
TLSMC: MozNSS compatibility interception ends.
TLS: could not load verify locations (file:`/etc/openldap/certs/57ae1fc05c196.pem',dir:`').
TLS: error:0B064071:x509 certificate routines:ADD_CERT_DIR:invalid directory by_dir.c:206
ldap_err2string

==> /var/log/httpd/ssl_error_log <==
[Thu May 21 14:02:44.791232 2020] [:error] [pid 17898] [client 10.20.16.26:63690] PHP Warning: ldap_start_tls(): Unable to start TLS: Connect error in /usr/local/nagiosxi/html/includes/components/ldap_ad_integration/adLDAP/src/adLDAP.php on line 641, referer: https://nagios2.hofstra.edu/nagiosxi/lo ... f&noauth=1

==> /var/log/httpd/error_log <==
ldap_bind_s
ldap_simple_bind_s
ldap_sasl_bind_s
ldap_sasl_bind
ldap_send_initial_request
ldap_send_server_request
ldap_result ld 0x559b16f3cde0 msgid 2
wait4msg ld 0x559b16f3cde0 msgid 2 (infinite timeout)
wait4msg continue ld 0x559b16f3cde0 msgid 2 all 1
** ld 0x559b16f3cde0 Connections:
* host: huemp16sc.employees.hofstra.univ port: 389 (default)
refcnt: 2 status: Connected
last used: Thu May 21 14:02:44 2020


** ld 0x559b16f3cde0 Outstanding Requests:
* msgid 2, origid 2, status InProgress
outstanding referrals 0, parent count 0
ld 0x559b16f3cde0 request count 1 (abandoned 0)
** ld 0x559b16f3cde0 Response Queue:
Empty
ld 0x559b16f3cde0 response count 0
ldap_chkResponseList ld 0x559b16f3cde0 msgid 2 all 1
ldap_chkResponseList returns ld 0x559b16f3cde0 NULL
ldap_int_select
read1msg: ld 0x559b16f3cde0 msgid 2 all 1
ldap_err2string

==> /var/log/httpd/ssl_error_log <==
[Thu May 21 14:02:44.792140 2020] [:error] [pid 17898] [client 10.20.16.26:63690] PHP Warning: ldap_bind(): Unable to bind to server: Can't contact LDAP server in /usr/local/nagiosxi/html/includes/components/ldap_ad_integration/adLDAP/src/adLDAP.php on line 714, referer: https://nagios2.hofstra.edu/nagiosxi/lo ... f&noauth=1

==> /var/log/httpd/error_log <==
ldap_free_request (origid 2, msgid 2)
ldap_free_connection 1 1
ldap_free_connection: actually freed
=================================================


I'm not sure whats going on here. Seems to me that nagios isn't updating the config somewhere when I re-import the cert. But it's also odd that when I created the files it was looking for, it still kicked the same error.

[cdienger]

I'd like to take a closer look over a remote if possible. Please create a ticket at https://support.nagios.com/tickets/ and reference this thread and we can set something up.