gpt4 book ai didi

scala - spark-core 依赖项中的冲突。它是如何工作的?

转载 作者:行者123 更新时间:2023-12-04 03:52:49 25 4
gpt4 key购买 nike

我的问题的入口点是一个异常(exception):

[info]   Cause: com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.8.4
[info] at com.fasterxml.jackson.module.scala.JacksonModule.setupModule(JacksonModule.scala:64)
[info] at com.fasterxml.jackson.module.scala.JacksonModule.setupModule$(JacksonModule.scala:51)
[info] at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:19)
[info] at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:745)
[info] at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82)
[info] at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)
[info] at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
[info] at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
[info] at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:83)
[info] at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:81)

发生在Spark升级后。我设法克服了这个问题,但我仍然不明白根本原因。看起来像是 Spark 依赖项的问题,请查看:

https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.12/2.4.7

它需要 com.fasterxml.jackson.core » jackson-databind 2.6.7.3。没问题。此外,它还需要: https://mvnrepository.com/artifact/org.json4s/json4s-jackson_2.12/3.5.3没问题。然而,https://mvnrepository.com/artifact/org.json4s/json4s-jackson_2.12/3.5.3需要 https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-databind/2.8.4现在,我们可以看到 spark-2.4.7jackson-databind 本身之间引入了依赖冲突。我的推理哪里错了?

最佳答案

正如您在父 pom 中看到的那样,com.fasterxml.jackson.core.* 被排除在 json4s-jackson 之外:

https://github.com/apache/spark/blob/v2.4.7/pom.xml#L755

所以,终究没有冲突

关于scala - spark-core 依赖项中的冲突。它是如何工作的?,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/64208323/

25 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com