Page 1 of 2

backend error?

Posted: Tue Jan 05, 2016 5:50 pm
by emartine
I added a new SSL Cert, restarted httpd and ended up getting a backend error issue. Apparently the monitoring engine process was unavailable and neither was the monitoring engine event queue. In the past I've seen this cause orphaned check issues with gearman when two of these spawn. I am unable to kill them

ps -ef | grep nagios.cfg
nagios 6904 1 8 16:39 ? 00:00:25 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios 6964 6904 0 16:39 ? 00:00:00 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg

I rebooted the server and I am getting the same result. Any idea what might be going on?

Re: backend error?

Posted: Tue Jan 05, 2016 6:02 pm
by emartine
I am running Nagios XI 2014R2.6

Re: backend error?

Posted: Wed Jan 06, 2016 10:36 am
by ssax
Are you seeing errors in the logs (apache or nagios logs)? If so, please post them.

You are supposed to see two of those, it's a problem if there are more than two though.

Re: backend error?

Posted: Fri Jan 08, 2016 12:59 pm
by emartine
I was able to kill the second primary process after some fuss... there was apparently a local report that held on to it and it no longer spawned another.

Re: backend error?

Posted: Fri Jan 08, 2016 1:45 pm
by bwallace
Thanks for the update. Are we in the clear now that you were able to kill that second primary process or are there still some lingering issues?

Re: backend error?

Posted: Tue Jan 12, 2016 12:58 pm
by emartine
All good for this. Just one question. What are the first steps to take if I run into this again... See screenshot.

Re: backend error?

Posted: Tue Jan 12, 2016 5:41 pm
by ssax
You should see two of them, here is the output of a freshly installed XI 5.2.3 server (installed today):

Code: Select all

[root@ssc66xidev ~]# ps -ef | grep nagios.cfg
nagios    2335     1  0 15:25 ?        00:00:07 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios    2363  2335  0 15:25 ?        00:00:02 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
Usually when you run into the issue you are experiencing it IS because of multiple nagios processes running (more than 2).

Are you seeing anything related in your /usr/local/nagios/var/nagios.log?

The only other times I've seen this is when the nagios user has expired and when someone changed the Apache allowed ciphers.

Code: Select all

chage -l nagios

Re: backend error?

Posted: Wed Jan 13, 2016 11:53 am
by emartine
Good catch. Yes it was the ciphers.
Ciphers were changed in the advent of new security policies that went into effect due in part to the new browser security updates that went out.

Two changes were made:
Default : SSLProtocol all -SSLv2
Default : SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW

I change this back to the defaults. Restarted apache and it worked fine. With these settings in place the security scan we ran on the server reported the below vulnerabilities. What do you suggest we do regarding this?



1)SSL/TLS use of weak RC4 cipher
THREAT:
Secure Sockets Layer (SSL v2/v3) and Transport Layer Security (TLS ) protocols provide integrity, confidentiality and authenticity services to other
protocols that lack these features.
SSL/TLS protocols use ciphers such as AES,DES, 3DES and RC4 to encrypt the content of the higher layer protocols and thus provide the
confidentiality service. Normally the output of an encryption process is a sequence of random looking bytes. It was known that RC4 output has some
bias in the output. Recently a group of researches has discovered that the there is a stronger bias in RC4, which make statistical analysis of
ciphertext more practical.
The described attack is to inject a malicious javascript into the victim's browser that would ensure that there are multiple connections being
established with a target website and the same HTTP cookie is sent multiple times to the website in encrypted form. This provides the attacker a
large set of ciphertext samples, that can be used for statistical analysis.
NOTE: On 3/12/15 NVD changed the CVSS v2 access complicity from high to medium. As a result Qualys revised the CVSS score to 4.3
immediately. On 5/4/15 Qualys is also revising the severity to level 3.
IMPACT:
If this attack is carried out and an HTTP cookie is recovered, then the attacker can then use the cookie to impersonate the user whose cookie was
recovered.
This attack is not very practical as it requires the attacker to have access to millions of samples of ciphertext, but there are certain assumptions that
an attacker can make to improve the chances of recovering the cleartext from cihpertext. For examples HTTP cookies are either base64 encoded or
hex digits. This information can help the attacker in their efforts to recover the cookie.
SOLUTION:
RC4 should not be used where possible. One reason that RC4 was still being used was BEAST and Lucky13 attacks against CBC mode ciphers in
SSL and TLS. However, TLSv 1.2 or later address these issues.

2) SSLv3 Padding Oracle Attack Information Disclosure Vulnerability (POODLE)
THREAT:
The SSL protocol 3.0 design error, uses nondeterministic CBC padding, which makes it easier for man-in-the-middle attacks.
The target supports SSLv3, which makes it vulnerable to POODLE (Padding Oracle On Downgraded Legacy Encryption), even if it also supports
more recent versions of TLS. It's subject to a downgrade attack, in which the attacker tricks the browser into connecting with SSLv3.
IMPACT:
An attacker who can take a man-in-the-middle (MitM) position can exploit this vulnerability and gain access to encrypted communication between a
client and server.
SOLUTION:
Disable SSLv3 support to avoid this vulnerability.
Examples to disable SSLv3.
nginx: list specific allowed protocols in the "ssl_protocols" line. Make sure SSLv2 and SSLv3 is not listed. For example: ssl_protocols TLSv2 TLSv1.1
TLSv1.2;
Apache: Add -SSLv3 to the "SSLProtocol" line.


3) SSL Server Has SSLv3 Enabled Vulnerability
THREAT:
Secure Sockets Layer (SSL v2/v3) and Transport Layer Security (TLS ) protocols provide integrity, confidentiality and authenticity services to other
protocols that lack these features.
SSL/TLS protocols use ciphers such as AES,DES, 3DES and RC4 to encrypt the content of the higher layer protocols and thus provide the
confidentiality service. Normally the output of an encryption process is a sequence of random looking bytes. It was known that RC4 output has some
bias in the output. Recently a group of researches has discovered that the there is a stronger bias in RC4, which make statistical analysis of
ciphertext more practical.
The described attack is to inject a malicious javascript into the victim's browser that would ensure that there are multiple connections being
established with a target website and the same HTTP cookie is sent multiple times to the website in encrypted form. This provides the attacker a
large set of ciphertext samples, that can be used for statistical analysis.
NOTE: On 3/12/15 NVD changed the CVSS v2 access complicity from high to medium. As a result Qualys revised the CVSS score to 4.3
immediately. On 5/4/15 Qualys is also revising the severity to level 3.
IMPACT:
If this attack is carried out and an HTTP cookie is recovered, then the attacker can then use the cookie to impersonate the user whose cookie was
recovered.
This attack is not very practical as it requires the attacker to have access to millions of samples of ciphertext, but there are certain assumptions that
an attacker can make to improve the chances of recovering the cleartext from cihpertext. For examples HTTP cookies are either base64 encoded or
hex digits. This information can help the attacker in their efforts to recover the cookie.
SOLUTION:
RC4 should not be used where possible. One reason that RC4 was still being used was BEAST and Lucky13 attacks against CBC mode ciphers in
SSL and TLS. However, TLSv 1.2 or later address these issues.

4) SSLv3.0/TLSv1.0 Protocol Weak CBC Mode Server Side Vulnerability (BEAST)

THREAT:
SSLv 3.0 and TLS v1.0 protocols are used to provide integrity, authenticity and privacy to other protocols such as HTTP and LDAP. They provide
these services by using encryption for privacy, x509 certificates for authenticity and one-way hash functions for integrity. To encrypt data SSL and
TLS can use block ciphers, which are encryption algorithms that can encrypt only a fixed block of original data to an encrypted block of the same
size. Note that these ciphers will always obtain the same resulting block for the same original block of data. To achieve difference in the output the
output of encryption is XORed with yet another block of the same size referred to as initialization vectors (IV). A special mode of operation for block
ciphers known as CBC (cipher block chaining) uses one IV for the initial block and the result of the previous block for each subsequent block to
obtain difference in the output of block cipher encryption.
In SSLv3.0 and TLSv1.0 implementation the choice CBC mode usage was poor because the entire traffic shares one CBC session with single set of
initial IVs. The rest of the IV are as mentioned above results of the encryption of the previous blocks. The subsequent IV are available to the
eavesdroppers. This allows an attacker with the capability to inject arbitrary traffic into the plain-text stream (to be encrypted by the client) to verify
their guess of the plain-text preceding the injected block. If the attackers guess is correct then the output of the encryption will be the same for two
blocks.
Scan Results page 6
For low entropy data it is possible to guess the plain-text block with relatively few number of attempts. For example for data that has 1000
possibilities the number of attempts can be 500.
For more information please see a paper by Gregory V. Bard. (http://eprint.iacr.org/2006/136.pdf)
NOTE:
The CVSS access complexity assigned by NIST to CVE-2011-3389 is 'Medium' which makes the base score 4.3. But Qualys has assigned access
complexity to 'High' for server side, because Javascipt injection and MiTM capabilities and a vulnerable client are required to exploit this vulnerability.
Therefore the Qualys CVSS score is 2.6.
IMPACT:
Recently attacks against the web authentication cookies have been described which used this vulnerability. If the authentication cookie is guessed
by the attacker then the attacker can impersonate the legitimate user on the Web site which accepts the authentication cookie.
SOLUTION:
This attack was identified in 2004 and later revisions of TLS protocol which contain a fix for this. If possible, upgrade to TLSv1.1 or TLSv1.2. If
upgrading to TLSv1.1 or TLSv1.2 is not possible, then disabling CBC mode ciphers will remove the vulnerability.
Setting your SSL server to prioritize RC4 ciphers mitigates this vulnerability. Microsoft has posted information including workarounds for IIS at
KB2588513 (http://technet.microsoft.com/en-us/secu ... ry/2588513).
Using the following SSL configuration in Apache mitigates this vulnerability:
SSLHonorCipherOrder On
SSLCipherSuite RC4-SHA:HIGH:!ADH
Qualys SSL/TLS Deployment Best Practices can be found here (https://www.ssllabs.com/projects/best-practices/).
Note: RC4 recommendation is only in situations where upgrade to TLSv1.2 is not possible. RC4 in TLS v1.0 has output bias problem as described in
QID 38601. Therefore it is recommended to upgrade to TLS v1.2 or later.

Re: backend error?

Posted: Wed Jan 13, 2016 6:25 pm
by tmcdonald
I can send you some reading material relating to the listed vulnerabilities, however security patching is not something we can do for you as it is not covered by our support contract. For this you will need to discuss with your Security or Linux team.

https://raymii.org/s/tutorials/Strong_S ... ache2.html
https://www.digitalocean.com/community/ ... nerability

The above links address many weaknesses in the SSL and Apache/nginx realms, as well as provide mitigation procedures. I do need to advise you to take proper backups before making any of the listed changes. If you have questions about the specifics, I will answer to the best of my ability, but security is something that needs to be handled by a proper security team.

Re: backend error?

Posted: Thu Jan 14, 2016 11:31 am
by emartine
Of course. The concern here is what does Nagios XI need to prevent the no output error.