scala - 由于 Avro Mapred 依赖性，Spark 构建失败

Question

我有一个 scala spark 项目由于某些依赖 hell 而失败。这是我的 build.sbt:

scalaVersion := "2.13.3"

val SPARK_VERSION = "3.2.0"

libraryDependencies ++= Seq(
  "com.typesafe" % "config" % "1.3.1",
  "com.github.pathikrit" %% "better-files" % "3.9.1",
  "org.apache.commons" % "commons-compress" % "1.14",
  "commons-io" % "commons-io" % "2.6",
  "com.typesafe.scala-logging" %% "scala-logging" % "3.9.4",
  "ch.qos.logback" % "logback-classic" % "1.2.3" exclude ("org.slf4j", "*"),
  "org.plotly-scala" %% "plotly-render" % "0.8.1",
  "org.apache.spark" %% "spark-sql" % SPARK_VERSION,
  "org.apache.spark" %% "spark-mllib" % SPARK_VERSION,

  // Test dependencies
  "org.scalatest" %% "scalatest" % "3.2.10" % Test,
  "com.amazon.deequ" % "deequ" % "2.0.0-spark-3.1" % Test,
  "org.awaitility" % "awaitility" % "3.0.0" % Test,
  "org.apache.spark" %% "spark-core" % SPARK_VERSION % Test,
  "org.apache.spark" %% "spark-sql" % SPARK_VERSION % Test

这是构建失败:

[error] stack trace is suppressed; run 'last update' for the full output
[error] stack trace is suppressed; run 'last ssExtractDependencies' for the full output
[error] (update) lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] https://repo1.maven.org/maven2/org/apache/avro/avro-mapred/1.10.2/avro-mapred-1.10.2-hadoop2.jar: not found: https://repo1.maven.org/maven2/org/apache/avro/avro-mapred/1.10.2/avro-mapred-1.10.2-hadoop2.jar
[error] (ssExtractDependencies) lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] https://repo1.maven.org/maven2/org/apache/avro/avro-mapred/1.10.2/avro-mapred-1.10.2-hadoop2.jar: not found: https://repo1.maven.org/maven2/org/apache/avro/avro-mapred/1.10.2/avro-mapred-1.10.2-hadoop2.jar
[error] Total time: 5 s, completed Dec 19, 2021, 5:14:33 PM
[info] shutting down sbt server

这是因为我使用的是 Scala 2.13 吗？

Answer 1

我不得不做不可避免的事情并将其添加到我的 build.sbt 中:

ThisBuild / useCoursier := false