scala - 运行 sbt 程序集时出错 : sbt deduplication error

Question

我正面临下面帖子中描述的确切问题，建议的答案没有帮助。 sbt-assembly: deduplication found error

[error] (*:assembly) deduplicate: different file contents found in the following:
[error] C:\Users\xxx\.ivy2\cache\org.eclipse.jetty.orbit\javax.transaction\orbits\javax.transaction-1.1.1.v201105210645.jar:META-INF/ECLIPSEF.RSA
[error] C:\Users\xxx\.ivy2\cache\org.eclipse.jetty.orbit\javax.servlet\orbits\javax.servlet-3.0.0.v201112011016.jar:META-INF/ECLIPSEF.RSA
[error] C:\Users\xxx\.ivy2\cache\org.eclipse.jetty.orbit\javax.mail.glassfish\orbits\javax.mail.glassfish-1.4.1.v201005082020.jar:META-INF/ECLIPSEF.RSA
[error] C:\Users\xxx\.ivy2\cache\org.eclipse.jetty.orbit\javax.activation\orbits\javax.activation-1.1.0.v201105071233.jar:META-INF/ECLIPSEF.RSA
[error] Total time: 14 s, completed Sep 9, 2014 5:21:01 PM

我的 build.sbt 文件包含

name := "Simple"

version := "0.1.0"

scalaVersion := "2.10.4"

libraryDependencies ++= Seq(
  "org.twitter4j" % "twitter4j-stream" % "3.0.3"
)

//libraryDependencies += "org.apache.spark" %% "spark-core" % "1.0.2"

libraryDependencies += "org.apache.spark" %% "spark-streaming" % "1.0.2"

libraryDependencies += "org.apache.spark" %% "spark-streaming-twitter" % "1.0.2"

libraryDependencies += "com.github.nscala-time" %% "nscala-time" % "0.4.2"

libraryDependencies ++= Seq(
    ("org.apache.spark"%%"spark-core"%"1.0.2").
    exclude("org.eclipse.jetty.orbit", "javax.servlet").
    exclude("org.eclipse.jetty.orbit", "javax.transaction").
    exclude("org.eclipse.jetty.orbit", "javax.mail").
    exclude("org.eclipse.jetty.orbit", "javax.activation").
    exclude("commons-beanutils", "commons-beanutils-core").
    exclude("commons-collections", "commons-collections").
    exclude("commons-collections", "commons-collections").
    exclude("com.esotericsoftware.minlog", "minlog")
)

resolvers += "Akka Repository" at "http://repo.akka.io/releases/"

    mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
    {
        case PathList("javax", "servlet", xs @ _*)         => MergeStrategy.first
        case PathList("javax", "transaction", xs @ _*)     => MergeStrategy.first
        case PathList("javax", "mail", xs @ _*)     => MergeStrategy.first
        case PathList("javax", "activation", xs @ _*)     => MergeStrategy.first
        case PathList(ps @ _*) if ps.last endsWith ".html" => MergeStrategy.first
        case "application.conf" => MergeStrategy.concat
        case "unwanted.txt"     => MergeStrategy.discard
        case x => old(x)
        }
    }

关于如何解决上述问题的任何指示？

Answer 1

如果您计划从 Spark 运行您的程序，那么我强烈建议添加所有 Spark 依赖项作为提供的，这样它们将被排除在组装任务之外。

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core"              % "1.0.2" % "provided",
  "org.apache.spark" %% "spark-streaming"         % "1.0.2" % "provided",
  "org.apache.spark" %% "spark-streaming-twitter" % "1.0.2" % "provided")

在另一种情况下，您需要从类路径中删除那些 jar 或向 mergeStrategy 添加适当的行，在您的情况下是

case PathList("META-INF", "ECLIPSEF.RSA") => MergeStrategy.first

如果您仍然希望处理 Spark 的依赖 hell ，sbt-dependency-graph插件应该有帮助。另请注意，其他 Spark 依赖项，如 spark-streaming 和 spark-streaming-twitter 可能也需要 exclude 指令。