Twilio stopped working

[SavaSC]

Hello,

A while back I was working on getting Twilio going on my 2 NagiosXI boxes. The support thread is here: https://support.nagios.com/forum/viewto ... 16&t=38789

We noticed this weekend when we had an outage that one of the NagiosXI boxes sent it's text alerts through Twilio just fine but the other did not. They both were working at one point.

I went back and looked, and they both appear to have the exact same setup. The files are in the same places and the configurations are the same. Even the users they should be texting are the same.

I'm at a loss at how to proceed. Any ideas where I can look?

Thanks!

*EDIT*

Forgot to say that I tested from both boxes using the following command:
php /usr/local/etc/sendTxtMsg.php 123456789 "Test"
(using my telephone #, of course)

When I did that test, the one box sent the test fine (as expected). The broken box gave an error saying that there was a certificate error.

[rkennedy]

The broken box gave an error saying that there was a certificate error.

Could you post this exact error? The Twilio plugin includes the certificate that needs to be used. Was it altered by any chance?
https://github.com/JustinCarmony/Nagios ... p/Services

[SavaSC]

object(Services_Twilio_TinyHttpException)#44 (7) {
["message:protected"]=>
string(146) "SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed"

Yes, I supposed the exact error would help. Sorry about that.

This box doesn't text alert for much so I'm not sure how long it has been down. The recent upgrade in Nagios versions is the only config change I've done to this box since getting Twilio working on it.

[rkennedy]

Let's check a few things on both systems. Could you show us the output for the following ran on BOTH machines? This will help to compare the differences and see what's going on under the hood.

Code: Select all

curl -V
curl -v https://api.twilio.com

Here's how the headers are coming up on my stock XI system -

Code: Select all

[root@localhost libexec]# curl -v https://api.twilio.com
* About to connect() to api.twilio.com port 443 (#0)
*   Trying 54.175.50.42... connected
* Connected to api.twilio.com (54.175.50.42) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=*.twilio.com,OU=api,O="Twilio, Inc.",L=San Francisco,ST=California,C=US
*       start date: Aug 06 00:00:00 2016 GMT
*       expire date: Sep 16 23:59:59 2017 GMT
*       common name: *.twilio.com
*       issuer: CN=thawte SSL CA - G2,O="thawte, Inc.",C=US
> GET / HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.21 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: api.twilio.com
> Accept: */*

Another coworker mentioned that Twilio did update their certificate a few months back, so this could be related - https://support.twilio.com/hc/en-us/art ... ay-31-2016

[SavaSC]

OK, here is the output from the two boxes.

Working box

Code: Select all

[root@ltc099l ~]# curl -V
curl 7.15.5 (i686-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
Protocols: tftp ftp telnet dict ldap http file https ftps
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz
[root@ltc099l ~]# curl -v https://api.twilio.com
* About to connect() to api.twilio.com port 443
*   Trying 54.209.197.21... connected
* Connected to api.twilio.com (54.209.197.21) port 443
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSLv2, Client hello (1):
SSLv3, TLS handshake, Server hello (2):
SSLv3, TLS handshake, CERT (11):
SSLv3, TLS handshake, Server finished (14):
SSLv3, TLS handshake, Client key exchange (16):
SSLv3, TLS change cipher, Client hello (1):
SSLv3, TLS handshake, Finished (20):
SSLv3, TLS change cipher, Client hello (1):
SSLv3, TLS handshake, Finished (20):
SSL connection using AES128-SHA
* Server certificate:
*        subject: /C=US/ST=California/L=San Francisco/O=Twilio, Inc./OU=api/CN=*.twilio.com
*        start date: 2016-08-06 00:00:00 GMT
*        expire date: 2017-09-16 23:59:59 GMT
*        subjectAltName: api.twilio.com matched
*        issuer: /C=US/O=thawte, Inc./CN=thawte SSL CA - G2
* SSL certificate verify ok.
> GET / HTTP/1.1
> User-Agent: curl/7.15.5 (i686-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
> Host: api.twilio.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Headers: Accept, Authorization, Content-Type, If-Match, If-Modified-Since, If-None-Match, If-Unmodified-Since
< Access-Control-Allow-Methods: GET, POST, DELETE, OPTIONS
< Access-Control-Allow-Origin: *
< Access-Control-Expose-Headers: ETag
< Content-Type: application/xml
< Date: Tue, 04 Oct 2016 13:46:06 GMT
< Strict-Transport-Security: max-age=15768000
< Twilio-Request-Duration: 0.001
< Twilio-Request-Id: RQ824addccbe904ce8b553ea48fbfbebdb
< X-Powered-By: AT-5000
< X-Shenanigans: none
< Content-Length: 558
< Connection: keep-alive
<?xml version='1.0' encoding='UTF-8'?>
Connection #0 to host api.twilio.com left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
<TwilioResponse><Versions firstpageuri="/?Page=0&PageSize=50" numpages="1" end="1" total="2" previouspageuri="" lastpageuri="/?Page=0&PageSize=50" uri="/" pagesize="50" start="0" nextpageuri="" page="0"><Version><Name>2008-08-01</Name><Uri>/2008-08-01</Uri><SubresourceUris><Accounts>/2008-08-01/Accounts</Accounts></SubresourceUris></Version><Version><Name>2010-04-01</Name><Uri>/2010-04-01</Uri><SubresourceUris><Accounts>/2010-04-01/Accounts</Accounts></SubresourceUris></Version></Versions></TwilioResponse>[root@ltc099l ~]#

Non-working box

Code: Select all

[root@ltc231l ~]# curl -V
curl 7.15.5 (i686-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
Protocols: tftp ftp telnet dict ldap http file https ftps
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz
[root@ltc231l ~]# curl -v https://api.twilio.com
* About to connect() to api.twilio.com port 443
*   Trying 52.2.38.199... connected
* Connected to api.twilio.com (52.2.38.199) port 443
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSLv2, Client hello (1):
SSLv3, TLS handshake, Server hello (2):
SSLv3, TLS handshake, CERT (11):
SSLv3, TLS alert, Server hello (2):
SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
* Closing connection #0
curl: (60) SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). The default
 bundle is named curl-ca-bundle.crt; you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

[dwhitfield]

On the non-working machine, please run

Code: Select all

yum update -y ca-certificates

.

It looks like the Twilio API certificate issue is a separate issue, so the fact that you are getting 2015 certs should not matter.

[SavaSC]

OK, it's almost working now. I looked at /etc/pki/tls/certs/ca-bundle.crt on both servers and the one on the working server was newer than the one on the non-working one. I copied the cert from the working one to the non-working one. Now sending a test message at the command line works. However, I'm having issues with Nagios actually sending the text. Going to go back over my setup and make sure I didn't miss anything.

[SavaSC]

On the non-working machine, please run

Code: Select all

yum update -y ca-certificates

.

It looks like the Twilio API certificate issue is a separate issue, so the fact that you are getting 2015 certs should not matter.

OH! Didn't know about that. Good info.

[tmcdonald]

Going to go back over my setup and make sure I didn't miss anything.

We'll leave this open for you in case you have further questions.

[SavaSC]

I (think I) have identified the issue with our Twillio settings. The problem wasn't that Nagios wasn't sending but the command line would. The problem was that I didn't completely understand the timing issues with the notification escalation. I had it set for texting on the 4th notification. Since my Host alert notification interval was set to 1 hour, that means the Host had to be down for 3 hours before the text went out. Since the down times were never that long, we never received the messages.

It's all about the fine print!

Thanks for keeping this open for so long. Now that everything is working correctly with my Twillio integration, you can close this thread if you want.