Page 1 of 1
Domain Expiration check with global domain support
Posted: Wed Jan 20, 2016 11:02 am
by CWSI
Hey guys,
I'm looking at monitoring domains from quite a number of different territories, including .my, .ae, .co.za, .au etc. Is there a plugin that can reliably check all domains that others are using successfully? I know it's a bit of a challenging thing to do due to some domains not offering WHOIS services, offering them in strange formats etc., but wondering if maybe there's something I've missed. For these domains I'm getting various errors from plugin timeouts to WHOIS run errors, though they work fine for .com, .co.uk, .ie etc.
Any thoughts gratefully received
Re: Domain Expiration check with global domain support
Posted: Wed Jan 20, 2016 11:14 am
by rkennedy
What plugin are you using currently to check the expiration?
Re: Domain Expiration check with global domain support
Posted: Wed Jan 20, 2016 1:22 pm
by CWSI
Hi,
I'm using check_domain.php 1.0.0, the one that came builtin with XI I believe. From what I've read part of the problem is that the Linux whois command relies on a list of whois servers which is not always up-to-date or has entries missing. There also doesn't seem to be quite enough error checking in check_domain.php as it will return things like "-16328 days to expiry" if it finds the word "expir" when it greps but the word isn't used in the context it expects.
Re: Domain Expiration check with global domain support
Posted: Wed Jan 20, 2016 1:48 pm
by rkennedy
Can you post a sample command where it returns the -16328 days to expiry so that I can replicate it on my end and submit a bug?
As for the check_domain.php scripts - it does include a -s variable that you can use to specify which whois server to use. I don't think the script is the issue at this point for what you're looking to accomplish.
The limitation, is the whois service on some domains. See below -
Code: Select all
[root@suse11 libexec]# ./check_domain.php -d google.ae -s whois.aeda.net.ae
Error running whois:
[root@suse11 libexec]# ./check_domain.php -d google.ae
Error running whois:
Does not work at all.. hmm. Let's try a whois -
Code: Select all
[root@suse11 libexec]# whois google.ae
[Querying whois.aeda.net.ae]
[whois.aeda.net.ae]
Domain Name: google.ae
Registrar ID: markmonitor
Registrar Name: MarkMonitor
Status: ok
Registrant Contact ID: GOOGLE
Registrant Contact Name: Google Inc.
Registrant Contact Email: Visit whois.aeda.ae for Web based WhoIs
Registrant Contact Organisation: Google Inc.
Tech Contact ID: GOOGLE
Tech Contact Name: Google Inc.
Tech Contact Email: Visit whois.aeda.ae for Web based WhoIs
Tech Contact Organisation: Google Inc.
Name Server: ns1.google.com
Name Server: ns2.google.com
Does not return any dates at all. So after navigating to the whois domain,
http://whois.aeda.net.ae/ - you'll notice this policy -
To address user concerns about privacy and spam, and in line with international best practice, .ae Domain Administration (aeDA) has implemented Image Verification Check (IVC) for all email addresses on the web-based Whois service. The purpose of IVC is to prevent or hinder unauthorised access by automated data mining programs or scripts. For consistency, aeDA has removed all email addresses from Port 43 Whois responses; users of Port 43 Whois will be referred to the web-based Whois service to access email addresses via IVC.
Please note that there are restrictions in place on how many queries you can make. Your query limit is 20 per hour and 200 per day from the same IP address. If you exceed this you will be banned for 24 hours.
If you need to use a facility to check for the availability of a Domain Name, you can use our Domain Availability tool (Whois Check).
With all of that said, I think the issue at this point is the additional security in play for some TLD's.
Re: Domain Expiration check with global domain support
Posted: Thu Jan 21, 2016 2:55 pm
by CWSI
Hey,
Thanks for the reply. Agreed, WHOIS seems to be troublesome and I guess my first query was whether anybody had found a workaround for this preferably without having to use the -s switch as it isn't too scalable. Looks like my other problem with more day-to-day domains may be with the WHOIS service on CentOS as I'm getting
Code: Select all
[root@uptime libexec]# ./check_domain.php -d google.com
^C
[root@uptime libexec]# ./check_domain.php -d google.com.my
^C
[root@uptime ~]# whois google.com
[Querying whois.verisign-grs.com]
[Redirected to whois.markmonitor.com]
[Querying whois.markmonitor.com]
^C
I had to Ctrl+C to kill them as they just hang, when Nagios runs the command it will return CRITICAL timed out 60 seconds. Seems it would be more consistent to return UNKNOWN for cases like this, the script waits for the WHOIS output indefinitely, it should probably wait 30 seconds then return "UNKNOWN Error Running WHOIS". But looks like I also need to fix that WHOIS service on this server
.
Re: Domain Expiration check with global domain support
Posted: Thu Jan 21, 2016 3:01 pm
by CWSI
I actually left one of those "./check_domain.php -d google.com" checks running in the background and just got a warning from AWS that the server was running at 100% CPU utilization, so certainly some poor behavior in that script
Appears to be a bug in jwhois -
https://bugzilla.redhat.com/show_bug.cgi?id=469412, taking their time solving it though
.
Re: Domain Expiration check with global domain support
Posted: Thu Jan 21, 2016 3:18 pm
by rkennedy
CWSI wrote:Hey,
Thanks for the reply. Agreed, WHOIS seems to be troublesome and I guess my first query was whether anybody had found a workaround for this preferably without having to use the -s switch as it isn't too scalable. Looks like my other problem with more day-to-day domains may be with the WHOIS service on CentOS as I'm getting
Code: Select all
[root@uptime libexec]# ./check_domain.php -d google.com
^C
[root@uptime libexec]# ./check_domain.php -d google.com.my
^C
[root@uptime ~]# whois google.com
[Querying whois.verisign-grs.com]
[Redirected to whois.markmonitor.com]
[Querying whois.markmonitor.com]
^C
I had to Ctrl+C to kill them as they just hang, when Nagios runs the command it will return CRITICAL timed out 60 seconds. Seems it would be more consistent to return UNKNOWN for cases like this, the script waits for the WHOIS output indefinitely, it should probably wait 30 seconds then return "UNKNOWN Error Running WHOIS". But looks like I also need to fix that WHOIS service on this server
.
As the whois servers are regulated, the -s flag is really your only option with this plugin. While a lot of this data is scraped, the registrars do try and stop it. One thing to note to, is after x hits on a whois API - you may get blocked by that specific registrar for all of your requests. This goes along with the whois databases trying to prevent all of those checks.
Honestly, the hang could be from the API not responding, or an actual bug as you mentioned. I read through that bug report though, and it sounds like it definitely could correspond to this issue.
How often do you really need these checks run realistically? If you're looking for scale, I'd recommend setting your check_interval to 10080.