ruby - CodeClimate 覆盖分析在 Travis CI 上是错误的

Question

我在新项目中遇到 codeclimate-test-reporter 和 Travis CI 的问题。 Travis CI 上的覆盖率分析与本地完全不同。具体来说，本地的 LOC 是 56(这是准确的)，但当我在 Travis CI 上构建项目时，它被列为 10,333。

代码和输出

本地输出(56/56 LOC)

Finished in 0.0749 seconds (files took 0.60563 seconds to load)
7 examples, 0 failures

Coverage report generated for RSpec to /Users/lee/GitHub/rescuetime/coverage. 
56 / 56 LOC (100.0%) covered.
Coverage = 100.0%. Sending report to https://codeclimate.com for branch 
data-analytics-api... done.

Travis CI 输出(4954/10333 LOC？!)

Coverage report generated for RSpec to /home/travis/build/leesharma/rescuetime/coverage. 
4954 / 10333 LOC (47.94%) covered.
Coverage = 47.94%. Sending report to https://codeclimate.com for branch 
data-analytics-api... done.

spec_helper.rb (与 codeclimate-test-reporter 相关的位)

require 'codeclimate-test-reporter'
require 'simplecov'
SimpleCov.formatter = SimpleCov::Formatter::MultiFormatter[
    SimpleCov::Formatter::HTMLFormatter,
    CodeClimate::TestReporter::Formatter
]
SimpleCov.start
...

VCR.configure do |config|
  ...
  config.ignore_hosts 'codeclimate.com'
end

Full project source code

我尝试过的

我已经确认代码完全相同，并且这些结果对于每个测试的 ruby 都是相同的(本地和 Travis CI，>=1.9.3 MRI)。我尝试将它限制为 Travis 的一个 ruby 版本，但结果是一样的。我曾尝试搜索类似的问题，但到目前为止，我只在他们的 repo 协议(protocol)中发现了一个开放/ Unresolved GitHub 问题。

老实说，我什至不确定接下来要尝试什么来调试它，因为我无法在本地重现它。知道接下来要尝试什么吗？

Answer 1

我向 CodeClimate 发送了一封电子邮件，他们的回复非常有帮助!

tl;dr

我调用的是 SimpleCov.start 而不是 CodeClimate::TestReporter.start，这意味着我的/vendor 和/spec 目录已包含在分析中。

我需要将 spec_helper.rb 的测试覆盖记者部分替换为:

# Test coverage reporters
require 'codeclimate-test-reporter'
CodeClimate::TestReporter.start
SimpleCov.formatter = SimpleCov::Formatter::MultiFormatter[
    SimpleCov::Formatter::HTMLFormatter,
    CodeClimate::TestReporter::Formatter
]

实际响应

Hey Lee-

Thanks for the detailed email. I did some digging on my end, and I believe I see what's happening here. This comes down to different exclusions being applied in different testing environments. Broadly speaking, the reason you're seeing such a high LOC in Travis is because coverage data for files in your /vendor and /spec folder ARE being included in the LOC count. I've explained below how to exclude these directories when Travis runs your builds and POSTs Code Climate coverage data.

To investigate this, I first looked over your spec_helper.rb file in GitHub, and I noticed one thing that was different from what Code Climate typically requires. Specifically, I saw that SimpleCov.start was being called instead of CodeClimate::TestReporter.start. That can cause a number of different problems, one of which is that the file exclusions we typically apply aren't triggered. That is, normally our reporter auto-excludes coverage data from all /vendor and /spec directories -- however, if you don't call CodeClimate::TestReporter.start then our exclusions won't kick in. In most cases, you won't want these directories included, since they don't represent code that you've written.

To correct this, I went ahead and forked your repository, and then added my fork to Travis. I then made some changes to the project's spec_helper.rb file, as seen here: https://github.com/jonathancadepowers/rescuetime/blob/master/spec/spec_helper.rb Note that I'm starting the Code Climate reporter (on Line 3) and that I'm no longer starting SimpleCov directly. Since Code Climate's Ruby test reporter is actually just a thin wrapper around SimpleCov, you don't need to directly start SimpleCov (the HTML report from SimpleCov should still get generated each time you run your tests).

After making my changes, I ran a build in Travis. Since your .travis.yml lists four different versions of Ruby, this resulted in four separate build jobs. For all of them, the line count dropped down to 30 (since our exclusions are now triggered) and you now have 100% coverage. See Line 152 here: https://travis-ci.org/jonathancadepowers/rescuetime/jobs/59266389#L152

My Travis build POSTed your Code Climate instance coverage data, so you now have 100% coverage there as well: https://codeclimate.com/github/leesharma/rescuetime

Of course, 56 doesn't equal 30. That said, I think I see why that's the case as well. I noticed from earlier payloads that Travis had sent us for your project that not only were /vendor files being included in the coverage payloads, but coverage data for files in your /spec folder were included as well. Digging in deeper, I saw that two specific /spec files were included: spec/unit/activities_spec.rb (11 total lines) and spec/unit/client_spec.rb (15 total lines). So: 56 - 11 - 15 = 30. That is, I would bet that locally /vendor is being excluded when you run your tests, but /spec is not. If you make the spec_helper.rb changes that I recommended above, both /vendor and /spec will be filtered out, taking you down to 30. So, 30 is probably the correct number here, not 56.

You should just be able to make the spec_helper.rb changes that I mentioned above to correct this issue. Of course, if there's anything else that I can help with, just let me know. Sorry again for the confusion here.

Thanks,

-Jonathan