support@xlroutes.com
Login

Retrying Ruby Geocoder Requests after OVER_QUERY_LIMIT errors

Approximate time to read: 2 min


XLRoutes Dynamic IP’s are designed to give you access to the Google Maps Geocoding API on cloud platforms like Heroku or CloudControl. Our proxy network spreads out your API requests across a number of machines, adding more machines as necessary to handle everybody’s requests. This service means you should not see any OVER_QUERY_LIMIT errors until you reach your daily limit (2500 as of November 2013).

Unfortunately Google also limits requests on a per second basis so if you submit more than 4 requests a second you will temporarily receive an OVER_QUERY_LIMIT error. We mitigate this to a large extent by having a buffer of spare capacity but if you attempt to make hundreds of calls a second you may still see this error.

So what is the best way to handle this temporary error?

Solution

Try, try and try again! Unless you are a website with high loads doing geocoding based on incoming web requests you will most commonly hit this issue when doing some bulk geocoding. In Ruby this is commonly done in a rake task. This gist shows you how to use the Ruby Geocoder gem to bulk geocode a collection of objects, retrying if you temporarily hit the OVER_QUERY_LIMIT error. 

namespace :geocoder do
  desc "Geocode all objects without coordinates."
  task :all_with_retry => :environment do
    class_name = ENV['CLASS'] || ENV['class']
    sleep_timer = ENV['SLEEP'] || ENV['sleep']
    raise "Please specify a CLASS (model)" unless class_name
    klass = class_from_string(class_name)

    klass.not_geocoded.each do |obj|
      geocode_with_retry obj
      sleep(sleep_timer.to_f) unless sleep_timer.nil?
    end
  end
end

def geocode_with_retry(obj)
  tries ||= 3
  puts "Trying again, #{tries} tries left..." if tries < 3
  puts Time.now.strftime("%Y%m%d%H%M%S")+": Geocoding " + obj.to_s
  obj.geocode
  if obj.geocoded?
    obj.save
  else
    sleep(3.0/tries) #Back off in 1s, 2s, 3s intervals
    raise Geocoder::OverQueryLimitError
  end 
  rescue Geocoder::OverQueryLimitError => e
    retry unless (tries -= 1).zero?
end

##
# Get a class object from the string given in the shell environment.
# Similar to ActiveSupport's +constantize+ method.
#
def class_from_string(class_name)
  parts = class_name.split("::")
  constant = Object
  parts.each do |part|
    constant = constant.const_get(part)
  end
  constant
end

Caveats

The Ruby Geocoder doesn’t raise exceptions when calls to geocode fail so to determine if a retry if necessary you have to check whether it has been geocoded. This could result in false retries where you are trying to geocode invalid data.

⟵ PREVNEXT ⟶