ruby - 如何在使用 Ruby Mechanize 网络爬行时绕过网络错误

Question

我正在使用 Ruby mechanize 网络爬虫从流行的房地产网站中提取数据。我使用家庭住址作为关键字来抓取 Zillow、Redfin 等的公共(public)数据。
我基本上试图绕过任何 HTTP 和网络错误。以下救援功能似乎无法完成这项工作。

def scrape_single(key_word)
    #setup agent
    agent = Mechanize.new{ |agent|
        agent.user_agent_alias = 'Mac Safari'
    }
    agent.ignore_bad_chunking = true
    agent.verify_mode = OpenSSL::SSL::VERIFY_NONE 
    agent.request_headers = { "Accept-Encoding" => ""}
    agent.follow_meta_refresh = true
    agent.keep_alive = false

    #page setup
    begin
      agent.get(@@search_engine) do |page|
        @@search_result = page.form('f') do |search|
          search.q = key_word
        end.submit
      end 
    rescue Timeout::Error
      puts "Timeout"
      retry
    rescue Net::HTTPGatewayTimeOut => e
      if e.response_code == '504' || '502'
        e.skip
        sleep 5
      end
    rescue Net::HTTPBadGateway  => e
      if e.response_code == '504' || '502'
        e.skip
        sleep 5
      end
    rescue Net::HTTPNotFound => e
      if e.response_code == '404'
        e.skip
        sleep 5
      end
    rescue Net::HTTPFatalError => e
      if e.response_code == '503'
        e.skip
      end
    rescue Mechanize::ResponseCodeError => e
      if e.response_code == '404'
        e.skip
        sleep 5
      elsif e.response_code == '502'
        e.skip
        sleep 5
      else
        retry
      end
    rescue Errno::ETIMEDOUT
      retry
    end

    return @@search_result      # returns Mechanize::Page
  end 

以下是我为 MA 中带有地址的关键字收到的错误消息示例。

/home/ec2-user/.gem/ruby/2.1/gems/mechanize-2.7.5/lib/mechanize/http/agent.rb:323:in `fetch': 404 => Net::HTTPNotFound for https://www.redfin.com/MA/WASHINGTON/306-WERDEN-RD-Unknown/home/134059623 -- unhandled response (Mechanize::ResponseCodeError)

当您输入上述 URL 时，您看到的实际消息是:

Cannot GET /MA/WASHINGTON/306-WERDEN-RD-Unknown/home/134059623

我的目标是简单地忽略和跳过零星错误并转到下一个关键字。我无法真正在网上找到有效的解决方案，任何反馈将不胜感激。

Answer 1

如果我理解引发的错误是 Mechanize::响应代码错误这显然是一个 404 响应代码。但是在您的脚本中，您不会从 引发 404 response_code Mechanize::响应代码错误

all_response_code = ['403', '404', '502']

rescue Mechanize::ResponseCodeError => e
  if all_response_code.include? response_code 
    e.skip
    sleep 5
  else
    retry
  end

也许如果你为 404 response_code 添加一个条件，它就会成功

编辑
我稍微更改了代码以减少行数