scala - Scalding 和 Hadoop 与 sbt-assembly 的依赖关系问题

Question

我正在尝试使用 sbt 构建一个简单的 hadoop 作业，我正在尝试运行它以尝试在 Amazon EMR 上运行它。但是，当我运行 sbt assembly 时，出现以下错误:

[error] (*:assembly) deduplicate: different file contents found in the following:
[error] /Users/trenthauck/.ivy2/cache/org.mortbay.jetty/jsp-2.1/jars/jsp-2.1-6.1.14.jar:org/apache/jasper/compiler/Node$ChildInfo.class
[error] /Users/trenthauck/.ivy2/cache/tomcat/jasper-compiler/jars/jasper-compiler-5.5.12.jar:org/apache/jasper/compiler/Node$ChildInfo.class
[error] Total time: 10 s, completed Sep 14, 2013 4:49:24 PM

我试图按照这里的建议 https://groups.google.com/forum/#!topic/simple-build-tool/tzkq5TioIqM但是它没有用。

我的 build.sbt 看起来像:

import AssemblyKeys._

mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
  {
    case PathList("org", "apache", "jasper", xs @ _*) => MergeStrategy.last
    case x => old(x)
  }
}

assemblySettings

name := "Scaling Play"

version := "SNAPSHOT-0.1"

scalaVersion := "2.10.1"

libraryDependencies ++= Seq(
  "com.twitter" % "scalding-core_2.10" % "0.8.8",
  "com.twitter" % "scalding-args_2.10" % "0.8.8",
  "com.twitter" % "scalding-date_2.10" % "0.8.8",
  "org.apache.hadoop" % "hadoop-core" % "1.0.0"
  )

Answer 1

指令的顺序很重要。您更新程序集设置，以便稍后再次覆盖它。先定义assemblySettings，然后更新即可解决。

更新后的 build.sbt:

import AssemblyKeys._

assemblySettings

mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
  {
    case PathList("org", "apache", "jasper", xs @ _*) => MergeStrategy.last
    case x => old(x)
  }
}

…

之后你会发现有很多冲突的类和其他文件。在这种情况下，您将需要进行以下合并:

case PathList("org", "apache", xs @ _*) => MergeStrategy.last
case PathList("javax", "servlet", xs @ _*) => MergeStrategy.last
case PathList("com", "esotericsoftware", xs @ _*) => MergeStrategy.last
case PathList("project.clj") => MergeStrategy.last
case PathList("overview.html") => MergeStrategy.last
case x => old(x)

请注意，对类文件使用合并策略可能会产生问题，这是由该特定类的不兼容版本引起的。如果是这种情况，那么您的问题就更大了，因为依赖项彼此不兼容。然后您必须求助于删除依赖项并找到/制作兼容版本。