scala - sbt 总是在 CI 中重新编译整个项目，即使有缓存？

Question

我正在努力通过这个基本工作流程将 SBT 用于 CI 过程:

编译测试

缓存 ~/.sbt和 ~/.ivy2/cache

缓存所有 target我的项目中的目录

在后续步骤中:

恢复 ~/.sbt和 ~/.ivy2/cache

恢复完整项目，包括以前生成的 target包含 .class 的目录文件和相同的源代码(应该是相同的结帐)

通过 sbt test 运行测试

100% 的时间， sbt test重新编译整个项目。我想了解或调试为什么会这样，因为自上次编译以来没有任何变化(好吧，什么都不应该改变，那么是什么让它相信某些东西？)

我目前正在使用带有 docker executor 的 circleci。这意味着有一个新的 docker 实例，来自同一个镜像，运行每个步骤，尽管我希望缓存可以解决这个问题。
.circleci/config.yml相关栏目(如果你不使用圆，这应该仍然是可以理解的；我已经注释了我能做的):

---
version: 2

jobs:
  # compile and cache compilation
  test-compile:
    working_directory: /home/circleci/myteam/myproj
    docker:
      - image: myorg/myimage:sbt-1.2.8
    steps:
      # the directory to be persisted (cached/restored) to the next step
      - attach_workspace:
          at: /home/circleci/myteam
      # git pull to /home/circleci/myteam/myproj
      - checkout
      - restore_cache:
          # look for a pre-existing set of ~/.ivy2/cache, ~/.sbt dirs 
          # from a prior build
          keys:
            - sbt-artifacts-{{ checksum "project/build.properties"}}-{{ checksum "build.sbt" }}-{{ checksum "project/Dependencies.scala" }}-{{ checksum "project/plugins.sbt" }}-{{ .Branch }}
      - restore_cache:
          # look for pre-existing set of 'target' dirs from a prior build
          keys:
            - build-{{ checksum "project/build.properties"}}-{{ checksum "build.sbt" }}-{{ checksum "project/Dependencies.scala" }}-{{ checksum "project/plugins.sbt" }}-{{ .Branch }}
      - run:
          # the compile step
          working_directory: /home/circleci/myteam/myproj
          command: sbt test:compile
      # per: https://www.scala-sbt.org/1.0/docs/Travis-CI-with-sbt.html
      # Cleanup the cached directories to avoid unnecessary cache updates
      - run:
          working_directory: /home/circleci
          command: |
            rm -rf /home/circleci/.ivy2/.sbt.ivy.lock
            find /home/circleci/.ivy2/cache -name "ivydata-*.properties" -print -delete
            find /home/circleci/.sbt -name "*.lock" -print -delete
      - save_cache:
          # cache ~/.ivy2/cache and ~/.sbt for subsequent builds
          key: sbt-artifacts-{{ checksum "project/build.properties"}}-{{ checksum "build.sbt" }}-{{ checksum "project/Dependencies.scala" }}-{{ checksum "project/plugins.sbt" }}-{{ .Branch }}-{{ .Revision }}
          paths:
            - /home/circleci/.ivy2/cache
            - /home/circleci/.sbt
      - save_cache:
          # cache the `target` dirs for subsequenet builds
          key: build-{{ checksum "project/build.properties"}}-{{ checksum "build.sbt" }}-{{ checksum "project/Dependencies.scala" }}-{{ checksum "project/plugins.sbt" }}-{{ .Branch }}-{{ .Revision }}
          paths:
            - /home/circleci/myteam/myproj/target
            - /home/circleci/myteam/myproj/project/target
            - /home/circleci/myteam/myproj/project/project/target
      # in circle, a 'workflow' undergoes several jobs, this first one 
      # is 'compile', the next will run the tests (see next 'job' section
      # 'test-run' below). 
      # 'persist to workspace' takes any files from this job and ensures 
      # they 'come with' the workspace to the next job in the workflow
      - persist_to_workspace:
          root: /home/circleci/myteam
          # bring the git checkout, including all target dirs
          paths:
            - myproj
      - persist_to_workspace:
          root: /home/circleci
          # bring the big stuff
          paths:
            - .ivy2/cache
            - .sbt

  # actually runs the tests compiled in the previous job
  test-run:
    environment:
      SBT_OPTS: -XX:+UseConcMarkSweepGC -XX:+UnlockDiagnosticVMOptions  -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -Duser.timezone=Etc/UTC -Duser.language=en -Duser.country=US
    docker:
      # run tests in the same image as before, but technically 
      # a different instance
      - image: myorg/myimage:sbt-1.2.8
    steps:
      # bring over all files 'persist_to_workspace' in the last job
      - attach_workspace:
          at: /home/circleci/myteam
      # restore ~/.sbt and ~/.ivy2/cache via `mv` from the workspace 
      # back to the home dir
      - run:
          working_directory: /home/circleci/myteam
          command: |
            [[ ! -d /home/circleci/.ivy2 ]] && mkdir /home/circleci/.ivy2

            for d in .ivy2/cache .sbt; do
              [[ -d "/home/circleci/$d" ]] && rm -rf "/home/circleci/$d"
              if [ -d "$d"  ]; then
                mv -v "$d" "/home/circleci/$d"
              else
                echo "$d does not exist" >&2
                ls -la . >&2
                exit 1
              fi
            done
      - run:
          # run the tests, already compiled
          # note: recompiles everything every time!
          working_directory: /home/circleci/myteam/myproj
          command: sbt test
          no_output_timeout: 3900s

workflows:
  version: 2
  build-and-test:
    jobs:
      - test-compile
      - test-run:
          requires:
            - test-compile

第二阶段的输出通常如下所示:

#!/bin/bash -eo pipefail
sbt test

[info] Loading settings for project myproj-build from native-packager.sbt,plugins.sbt ...
[info] Loading project definition from /home/circleci/myorg/myproj/project
[info] Updating ProjectRef(uri("file:/home/circleci/myorg/myproj/project/"), "myproj-build")...
[info] Done updating.
[warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
[info] Compiling 1 Scala source to /home/circleci/myorg/myproj/project/target/scala-2.12/sbt-1.0/classes ...
[info] Done compiling.
[info] Loading settings for project root from build.sbt ...
[info] Set current project to Piranha (in build file:/home/circleci/myorg/myproj/)
[info] Compiling 1026 Scala sources to /home/circleci/myorg/myproj/target/scala-2.12/classes ...

我能做些什么来确定为什么这是第二次重新编译所有源并缓解它？

我在 linux 容器中运行 sbt 1.2.8 和 scala 2.12.8。

更新

我还没有解决这个问题，但我想我会分享一个解决我最严重问题的方法。

主要问题:将“测试编译”与“测试运行”分开
次要问题:更快的构建，而不必在每次推送时重新编译所有内容

我对次要没有解决办法。对于初级:

我可以运行 scalatest runner从 CLI 通过 scala -cp ... org.scalatest.tools.Runner而不是通过 sbt test以避免任何重新编译的尝试。 runner 可以对目录 .class 进行操作文件。

变更概要:

更新 docker 容器以包含 scala cli 安装。 (不幸的是，我现在需要保持这些版本同步)

构建阶段:sbt test:compile 'inspect run' 'export test:fullClasspath' | tee >(grep -F '.jar' > ~test-classpath.txt)

编译但也记录一个可复制的类路径字符串，适合传入 scala -cp VALUE_HERE运行测试

测试阶段:scala -cp "$(cat test-classpath.txt)" org.scalatest.tools.Runner -R target/scala-2.12/test-classes/ -u target/test-reports -oD

使用编译后的 .class 通过 runner 运行 Scalatest target/scala-2.12/test-classes 中的文件，使用在编译阶段报告的类路径，并打印到标准输出以及报告目录

我不喜欢这个，它有一些问题，但我想我会分享这个解决方法。

Answer 1

如果您使用比 1.0.4 更新的 sbt 版本，则缓存对您不起作用，因为编译器将始终使所有内容无效。
此锌编译器问题已在此处报告:https://github.com/sbt/sbt/issues/4168

我的建议是降级 CI 的 sbt 版本。还要检查和验证 CI 是否正在更改 .sbt 或 .ivy2 文件时间戳。如果它们被更改，请通过压缩和解压缩它们来单独缓存它们。

我在 Bitbucket Pipelines CI 上遇到了同样的问题，并成功使其工作 here