selenium - 功能规范仅在 CircleCI 或 Codeship 持续集成服务上失败

Question

我的基本功能规范在本地通过得很好，但在 CircleCI 和 Codeship 上失败了。失败的测试:

require 'spec_helper'

describe 'Authentication' do

  describe "sign in" do
    it "is the landing page for non-authenticated users" do
      user = create(:user)
      visit root_path

      expect( page ).to have_content 'LOG IN' # On sign-in page
      fill_in 'Email', with: user.email
      fill_in "Password", with: user.password
      click_button 'Sign in'

      expect( current_path ).to eq user_path(user)
    end
  end

  describe 'registration' do
    it "allows new users to register" do
      visit root_path

      click_link 'Sign up'
      fill_in 'Email', with: 'myfake@email.com'
      fill_in 'Password', with: 'password'
      fill_in 'Password confirmation', with: 'password'
      fill_in 'First name', with: 'John'
      fill_in 'Last name', with: 'Doe'
      click_button "Sign up"

      expect( current_path ).to include '/users/'
      expect( page ).to have_content "Welcome! You have signed up successfully."
    end
  end
end

测试在设置页面期望的第一行(分别为 expect( page ).to have_content "LOG IN" 和 click_link "Sign up" )失败，错误提示页面 HTML 完全空白:

expected to find text "LOG IN" in ""

我在 CircleCI 上保存了屏幕截图，它们确实显示了一个完全空白的页面。

这就是有趣的地方。我尝试通过 running/watching the specs on Circle using a VNC 调试问题.当我 a) 套装 driver: :selenium对于测试， b) 添加 sleep 1或两个测试页面预期之前的测试，以及 c) 在使用 VNC SSH 连接到他们的服务器后手动运行测试，我可以看到测试在 Selenium 中运行(他们在 VNC 中打开浏览器)并且它们完美地通过。

然而，在 VNC 之外，测试在两个 CI 服务器中始终失败。有无吨 sleep s 和 driver: :selenium .任何想法可能导致常规 CircleCI/Codeship 服务器与其 VCN/我的本地测试环境之间存在这种差异？我与 CircleCI 的人取得了联系，但他们暂时被难住了。

If relevant, I'm running Ruby 2.2.0, Rails 4.2, Capybara 2.4.4, Capybara-Webkit 1.4.1, and Selenium-Webdriver 2.44.0

一些可能相关的文件:

spec_helper.rb

ENV["RAILS_ENV"] = "test"

require File.expand_path("../../config/environment", __FILE__)

require "rspec/rails"
require "shoulda/matchers"
require "webmock/rspec"
require 'vcr'

Dir[Rails.root.join("spec/support/**/*.rb")].each { |file| require file }

module Features
  include Warden::Test::Helpers
  Warden.test_mode!

  def sign_in(user)
    login_as(user, scope: :user)
  end
end

module Controllers
  # Pre-parse controller responses for easy access
  def response_body
    body = JSON.parse(response.body)
    body.is_a?(Hash) ? body.to_sh : body.map(&:to_sh)
  end
end

module Mock
  def disable_webmock(&block)
    WebMock.disable!
    yield
    WebMock.enable!
  end
end

RSpec.configure do |config|
  config.expect_with :rspec do |c|
    c.syntax = :expect
  end

  # Save a screenshot to CircleCI when a feature test fails
  config.after(:each, :type => :feature) do |example|
    if example.exception
      artifact = save_page
      puts "\"#{example.description}\" failed. Page saved to #{artifact}"
    end
  end

  config.include Features, type: :feature
  config.include Controllers, type: :controller
  config.include Mock
  config.include Formulaic::Dsl, type: :feature
  config.infer_spec_type_from_file_location!
  config.infer_base_class_for_anonymous_controllers = false
  config.order = "random"
  config.use_transactional_fixtures = false
end

RSpec::Matchers.define :hash_eq do |expected|
  match do |actual|
    actual.recursive_symbolize_keys == expected.recursive_symbolize_keys
  end
end

VCR.configure do |c|
  c.cassette_library_dir = 'spec/fixtures/vcr_cassettes'
  c.hook_into :webmock
  c.allow_http_connections_when_no_cassette = true
  c.configure_rspec_metadata!
  c.ignore_hosts '127.0.0.1', 'localhost:3000'
end

ActiveRecord::Migration.maintain_test_schema!
Capybara.javascript_driver = :webkit

if ENV['CIRCLE_ARTIFACTS']
  Capybara.save_and_open_page_path = ENV['CIRCLE_ARTIFACTS']
end

WebMock.disable_net_connect!(allow_localhost: true)

database_cleaner.rb

RSpec.configure do |config|
  config.use_transactional_fixtures = false

  config.before(:suite) do
    DatabaseCleaner.strategy = :truncation
  end

  config.before(:each) do
    DatabaseCleaner.start
  end

  config.after(:each) do
    DatabaseCleaner.clean
  end

end

Answer 1

我可以想到要尝试的几件事:

如果您想让您的规范 headless 运行，请尝试使用 poltergeist 而不是 capybara-webkit。 poltergeist 有几个优点，我在这里描述过:https://stackoverflow.com/a/24108439/634576

Selenium 可能只是一种干扰，但是，如果您想让它工作，请确保您的 CI 环境具有正确版本的 Firefox。 selenium-webdriver 的每个版本似乎都需要范围很窄的 Firefox 版本。 On CircleCI, you can configure circle.yml to install a specific version of Firefox.现在我们使用 selenium-webdriver 2.46.2 和 Firefox 39.0.3，在 circle.yml 中安装了以下内容:

dependencies:
  pre:
    - sudo apt-get update; sudo apt-get install firefox=39.0.3+build2-0ubuntu0.12.04.1

我不知道 Codeship。