scala - sbt-assembly:合并错误 - 重复数据删除

Question

我在使用 sbt assembly 时遇到了这些错误。

我正在使用 Spark，这似乎是这个问题的根源。

val Spark =  Seq(
  "org.apache.spark" %% "spark-core" % sparkVersion,
  "org.apache.spark" %% "spark-sql" % sparkVersion,
  "org.apache.spark" %% "spark-streaming" % sparkVersion
)

错误:

[error] 12 errors were encountered during merge
[trace] Stack trace suppressed: run last coreBackend/*:assembly for the full output.
[trace] Stack trace suppressed: run last core/*:assembly for the full output.
[trace] Stack trace suppressed: run last commons/*:assembly for the full output.
[error] (coreBackend/*:assembly) deduplicate: different file contents found in the following:
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.osgi/org.osgi.core/org.osgi.core-4.3.1.jar:OSGI-OPT/bnd.bnd
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.osgi/org.osgi.compendium/org.osgi.compendium-4.3.1.jar:OSGI-OPT/bnd.bnd
[error] deduplicate: different file contents found in the following:
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/bundles/com.google.guava/guava/guava-18.0.jar:com/google/common/base/Absent.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-network-common_2.11/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Absent.class
[error] deduplicate: different file contents found in the following:
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/bundles/com.google.guava/guava/guava-18.0.jar:com/google/common/base/Function.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-network-common_2.11/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Function.class
[error] deduplicate: different file contents found in the following:
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/bundles/com.google.guava/guava/guava-18.0.jar:com/google/common/base/Optional$1$1.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-network-common_2.11/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Optional$1$1.class
[error] deduplicate: different file contents found in the following:
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/bundles/com.google.guava/guava/guava-18.0.jar:com/google/common/base/Optional$1.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-network-common_2.11/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Optional$1.class
[error] deduplicate: different file contents found in the following:
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/bundles/com.google.guava/guava/guava-18.0.jar:com/google/common/base/Optional.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-network-common_2.11/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Optional.class
[error] deduplicate: different file contents found in the following:
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/bundles/com.google.guava/guava/guava-18.0.jar:com/google/common/base/Present.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-network-common_2.11/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Present.class
[error] deduplicate: different file contents found in the following:
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/bundles/com.google.guava/guava/guava-18.0.jar:com/google/common/base/Supplier.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-network-common_2.11/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Supplier.class
[error] deduplicate: different file contents found in the following:
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.hadoop/hadoop-yarn-common/hadoop-yarn-common-2.2.0.jar:org/apache/hadoop/yarn/factories/package-info.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.hadoop/hadoop-yarn-api/hadoop-yarn-api-2.2.0.jar:org/apache/hadoop/yarn/factories/package-info.class
[error] deduplicate: different file contents found in the following:
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.hadoop/hadoop-yarn-common/hadoop-yarn-common-2.2.0.jar:org/apache/hadoop/yarn/factory/providers/package-info.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.hadoop/hadoop-yarn-api/hadoop-yarn-api-2.2.0.jar:org/apache/hadoop/yarn/factory/providers/package-info.class
[error] deduplicate: different file contents found in the following:
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.hadoop/hadoop-yarn-common/hadoop-yarn-common-2.2.0.jar:org/apache/hadoop/yarn/util/package-info.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.hadoop/hadoop-yarn-api/hadoop-yarn-api-2.2.0.jar:org/apache/hadoop/yarn/util/package-info.class
[error] deduplicate: different file contents found in the following:
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-core_2.11/spark-core_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-launcher_2.11/spark-launcher_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.spark-project.spark/unused/unused-1.0.0.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-network-common_2.11/spark-network-common_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-network-shuffle_2.11/spark-network-shuffle_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-unsafe_2.11/spark-unsafe_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-sql_2.11/spark-sql_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-catalyst_2.11/spark-catalyst_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-streaming_2.11/spark-streaming_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] (core/*:assembly) deduplicate: different file contents found in the following:
[error] /Users/bryan/.ivy2/cache/org.osgi/org.osgi.core/jars/org.osgi.core-4.3.1.jar:OSGI-OPT/bnd.bnd
[error] /Users/bryan/.ivy2/cache/org.osgi/org.osgi.compendium/jars/org.osgi.compendium-4.3.1.jar:OSGI-OPT/bnd.bnd
[error] deduplicate: different file contents found in the following:
[error] /Users/bryan/.ivy2/cache/com.google.guava/guava/bundles/guava-18.0.jar:com/google/common/base/Absent.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Absent.class
[error] deduplicate: different file contents found in the following:
[error] /Users/bryan/.ivy2/cache/com.google.guava/guava/bundles/guava-18.0.jar:com/google/common/base/Function.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Function.class
[error] deduplicate: different file contents found in the following:
[error] /Users/bryan/.ivy2/cache/com.google.guava/guava/bundles/guava-18.0.jar:com/google/common/base/Optional$1$1.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Optional$1$1.class
[error] deduplicate: different file contents found in the following:
[error] /Users/bryan/.ivy2/cache/com.google.guava/guava/bundles/guava-18.0.jar:com/google/common/base/Optional$1.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Optional$1.class
[error] deduplicate: different file contents found in the following:
[error] /Users/bryan/.ivy2/cache/com.google.guava/guava/bundles/guava-18.0.jar:com/google/common/base/Optional.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Optional.class
[error] deduplicate: different file contents found in the following:
[error] /Users/bryan/.ivy2/cache/com.google.guava/guava/bundles/guava-18.0.jar:com/google/common/base/Present.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Present.class
[error] deduplicate: different file contents found in the following:
[error] /Users/bryan/.ivy2/cache/com.google.guava/guava/bundles/guava-18.0.jar:com/google/common/base/Supplier.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Supplier.class
[error] deduplicate: different file contents found in the following:
[error] /Users/bryan/.ivy2/cache/org.apache.hadoop/hadoop-yarn-common/jars/hadoop-yarn-common-2.2.0.jar:org/apache/hadoop/yarn/factories/package-info.class
[error] /Users/bryan/.ivy2/cache/org.apache.hadoop/hadoop-yarn-api/jars/hadoop-yarn-api-2.2.0.jar:org/apache/hadoop/yarn/factories/package-info.class
[error] deduplicate: different file contents found in the following:
[error] /Users/bryan/.ivy2/cache/org.apache.hadoop/hadoop-yarn-common/jars/hadoop-yarn-common-2.2.0.jar:org/apache/hadoop/yarn/factory/providers/package-info.class
[error] /Users/bryan/.ivy2/cache/org.apache.hadoop/hadoop-yarn-api/jars/hadoop-yarn-api-2.2.0.jar:org/apache/hadoop/yarn/factory/providers/package-info.class
[error] deduplicate: different file contents found in the following:
[error] /Users/bryan/.ivy2/cache/org.apache.hadoop/hadoop-yarn-common/jars/hadoop-yarn-common-2.2.0.jar:org/apache/hadoop/yarn/util/package-info.class
[error] /Users/bryan/.ivy2/cache/org.apache.hadoop/hadoop-yarn-api/jars/hadoop-yarn-api-2.2.0.jar:org/apache/hadoop/yarn/util/package-info.class
[error] deduplicate: different file contents found in the following:
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-core_2.11/jars/spark-core_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-launcher_2.11/jars/spark-launcher_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Users/bryan/.ivy2/cache/org.spark-project.spark/unused/jars/unused-1.0.0.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-network-shuffle_2.11/jars/spark-network-shuffle_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-unsafe_2.11/jars/spark-unsafe_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-sql_2.11/jars/spark-sql_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-catalyst_2.11/jars/spark-catalyst_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-streaming_2.11/jars/spark-streaming_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] (commons/*:assembly) deduplicate: different file contents found in the following:
[error] /Users/bryan/.ivy2/cache/org.osgi/org.osgi.core/jars/org.osgi.core-4.3.1.jar:OSGI-OPT/bnd.bnd
[error] /Users/bryan/.ivy2/cache/org.osgi/org.osgi.compendium/jars/org.osgi.compendium-4.3.1.jar:OSGI-OPT/bnd.bnd

我在这里尝试了所有推荐的解决方案，但没有成功。

sbt-assembly: deduplication found error

deduplicating commons-validator - sbt assembly

spark + sbt-assembly: "deduplicate: different file contents found in the following"

Answer 1

这不完全是问题的答案，但它是一种解决方法。

我希望这可以节省几百个工时。

使用sbt-native-packager而不是 sbt-assembly。

添加到plugins.sbt:

addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "1.0.0")

在你的build.sbt

enablePlugins(JavaAppPackaging)
enablePlugins(UniversalPlugin)

要为多个 Scala 版本构建文件，请使用 +

+ universal:packageBin

输出会告诉您文件的创建位置。

不幸的是生成的 jar 被压缩了。它不是一个脂肪 jar 。 (生成一个 fat jar 需要 sbt-assembly，它有同样的问题)

为了克服这个问题，我制作了一个简单的脚本(在 SBT 中)来解压缩生成的文件并将 jar 路径写入文件，这样我就可以轻松构建 Spark 提交脚本。

packageBin in TxtFormat := {

    val zippedJar = "core-backend-1.0.zip"
    val basePath = target.value / "universal"

    // Unzip to folder of JARs
    IO.unzip(basePath / zippedJar, basePath)

    val fileMappings = (mappings in Universal).value
    val sparkScriptOut = basePath / s"${packageName.value}.txt"

    // append all mappings to the list
    fileMappings foreach {
        case (file, name) => IO.append(sparkScriptOut, s"core-backend-1.0/$name${IO.Newline}")
    }
    sparkScriptOut
}

构建 zip 后，使用它来执行任务:

+ txtFormat:packageBin