Page 1 of 1
Twilio stopped working
Posted: Mon Oct 03, 2016 2:59 pm
by SavaSC
Hello,
A while back I was working on getting Twilio going on my 2 NagiosXI boxes. The support thread is here:
https://support.nagios.com/forum/viewto ... 16&t=38789
We noticed this weekend when we had an outage that one of the NagiosXI boxes sent it's text alerts through Twilio just fine but the other did not. They both were working at one point.
I went back and looked, and they both appear to have the exact same setup. The files are in the same places and the configurations are the same. Even the users they should be texting are the same.
I'm at a loss at how to proceed. Any ideas where I can look?
Thanks!
*EDIT*
Forgot to say that I tested from both boxes using the following command:
php /usr/local/etc/sendTxtMsg.php 123456789 "Test"
(using my telephone #, of course)
When I did that test, the one box sent the test fine (as expected). The broken box gave an error saying that there was a certificate error.
Re: Twilio stopped working
Posted: Mon Oct 03, 2016 3:25 pm
by rkennedy
The broken box gave an error saying that there was a certificate error.
Could you post this exact error? The Twilio plugin includes the certificate that needs to be used. Was it altered by any chance?
https://github.com/JustinCarmony/Nagios ... p/Services
Re: Twilio stopped working
Posted: Mon Oct 03, 2016 3:34 pm
by SavaSC
object(Services_Twilio_TinyHttpException)#44 (7) {
["message:protected"]=>
string(146) "SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed"
Yes, I supposed the exact error would help. Sorry about that.
This box doesn't text alert for much so I'm not sure how long it has been down. The recent upgrade in Nagios versions is the only config change I've done to this box since getting Twilio working on it.
Re: Twilio stopped working
Posted: Mon Oct 03, 2016 4:36 pm
by rkennedy
Let's check a few things on both systems. Could you show us the output for the following ran on BOTH machines? This will help to compare the differences and see what's going on under the hood.
Code: Select all
curl -V
curl -v https://api.twilio.com
Here's how the headers are coming up on my stock XI system -
Code: Select all
[root@localhost libexec]# curl -v https://api.twilio.com
* About to connect() to api.twilio.com port 443 (#0)
* Trying 54.175.50.42... connected
* Connected to api.twilio.com (54.175.50.42) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* subject: CN=*.twilio.com,OU=api,O="Twilio, Inc.",L=San Francisco,ST=California,C=US
* start date: Aug 06 00:00:00 2016 GMT
* expire date: Sep 16 23:59:59 2017 GMT
* common name: *.twilio.com
* issuer: CN=thawte SSL CA - G2,O="thawte, Inc.",C=US
> GET / HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.21 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: api.twilio.com
> Accept: */*
Another coworker mentioned that Twilio did update their certificate a few months back, so this could be related -
https://support.twilio.com/hc/en-us/art ... ay-31-2016
Re: Twilio stopped working
Posted: Tue Oct 04, 2016 8:49 am
by SavaSC
OK, here is the output from the two boxes.
Working box
Code: Select all
[root@ltc099l ~]# curl -V
curl 7.15.5 (i686-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
Protocols: tftp ftp telnet dict ldap http file https ftps
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz
[root@ltc099l ~]# curl -v https://api.twilio.com
* About to connect() to api.twilio.com port 443
* Trying 54.209.197.21... connected
* Connected to api.twilio.com (54.209.197.21) port 443
* successfully set certificate verify locations:
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* SSLv2, Client hello (1):
SSLv3, TLS handshake, Server hello (2):
SSLv3, TLS handshake, CERT (11):
SSLv3, TLS handshake, Server finished (14):
SSLv3, TLS handshake, Client key exchange (16):
SSLv3, TLS change cipher, Client hello (1):
SSLv3, TLS handshake, Finished (20):
SSLv3, TLS change cipher, Client hello (1):
SSLv3, TLS handshake, Finished (20):
SSL connection using AES128-SHA
* Server certificate:
* subject: /C=US/ST=California/L=San Francisco/O=Twilio, Inc./OU=api/CN=*.twilio.com
* start date: 2016-08-06 00:00:00 GMT
* expire date: 2017-09-16 23:59:59 GMT
* subjectAltName: api.twilio.com matched
* issuer: /C=US/O=thawte, Inc./CN=thawte SSL CA - G2
* SSL certificate verify ok.
> GET / HTTP/1.1
> User-Agent: curl/7.15.5 (i686-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
> Host: api.twilio.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Headers: Accept, Authorization, Content-Type, If-Match, If-Modified-Since, If-None-Match, If-Unmodified-Since
< Access-Control-Allow-Methods: GET, POST, DELETE, OPTIONS
< Access-Control-Allow-Origin: *
< Access-Control-Expose-Headers: ETag
< Content-Type: application/xml
< Date: Tue, 04 Oct 2016 13:46:06 GMT
< Strict-Transport-Security: max-age=15768000
< Twilio-Request-Duration: 0.001
< Twilio-Request-Id: RQ824addccbe904ce8b553ea48fbfbebdb
< X-Powered-By: AT-5000
< X-Shenanigans: none
< Content-Length: 558
< Connection: keep-alive
<?xml version='1.0' encoding='UTF-8'?>
Connection #0 to host api.twilio.com left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
<TwilioResponse><Versions firstpageuri="/?Page=0&PageSize=50" numpages="1" end="1" total="2" previouspageuri="" lastpageuri="/?Page=0&PageSize=50" uri="/" pagesize="50" start="0" nextpageuri="" page="0"><Version><Name>2008-08-01</Name><Uri>/2008-08-01</Uri><SubresourceUris><Accounts>/2008-08-01/Accounts</Accounts></SubresourceUris></Version><Version><Name>2010-04-01</Name><Uri>/2010-04-01</Uri><SubresourceUris><Accounts>/2010-04-01/Accounts</Accounts></SubresourceUris></Version></Versions></TwilioResponse>[root@ltc099l ~]#
Non-working box
Code: Select all
[root@ltc231l ~]# curl -V
curl 7.15.5 (i686-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
Protocols: tftp ftp telnet dict ldap http file https ftps
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz
[root@ltc231l ~]# curl -v https://api.twilio.com
* About to connect() to api.twilio.com port 443
* Trying 52.2.38.199... connected
* Connected to api.twilio.com (52.2.38.199) port 443
* successfully set certificate verify locations:
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* SSLv2, Client hello (1):
SSLv3, TLS handshake, Server hello (2):
SSLv3, TLS handshake, CERT (11):
SSLv3, TLS alert, Server hello (2):
SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
* Closing connection #0
curl: (60) SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
More details here: http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). The default
bundle is named curl-ca-bundle.crt; you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
Re: Twilio stopped working
Posted: Tue Oct 04, 2016 9:33 am
by dwhitfield
On the non-working machine, please run
.
It looks like the Twilio API certificate issue is a separate issue, so the fact that you are getting 2015 certs should not matter.
Re: Twilio stopped working
Posted: Tue Oct 04, 2016 10:11 am
by SavaSC
OK, it's almost working now. I looked at /etc/pki/tls/certs/ca-bundle.crt on both servers and the one on the working server was newer than the one on the non-working one. I copied the cert from the working one to the non-working one. Now sending a test message at the command line works. However, I'm having issues with Nagios actually sending the text. Going to go back over my setup and make sure I didn't miss anything.
Re: Twilio stopped working
Posted: Tue Oct 04, 2016 10:12 am
by SavaSC
dwhitfield wrote:On the non-working machine, please run
.
It looks like the Twilio API certificate issue is a separate issue, so the fact that you are getting 2015 certs should not matter.
OH! Didn't know about that. Good info.
Re: Twilio stopped working
Posted: Tue Oct 04, 2016 10:58 am
by tmcdonald
SavaSC wrote:Going to go back over my setup and make sure I didn't miss anything.
We'll leave this open for you in case you have further questions.
Re: Twilio stopped working
Posted: Wed Nov 30, 2016 11:21 am
by SavaSC
I (think I) have identified the issue with our Twillio settings. The problem
wasn't that Nagios wasn't sending but the command line would. The problem was that I didn't completely understand the timing issues with the notification escalation. I had it set for texting on the 4th notification. Since my Host alert notification interval was set to 1 hour, that means the Host had to be down for 3 hours before the text went out. Since the down times were never that long, we never received the messages.
It's all about the fine print!
Thanks for keeping this open for so long. Now that everything is working correctly with my Twillio integration, you can close this thread if you want.