apache-spark - 会发生什么 - NoSuchMethodError : com. datastax.driver.core.ResultSet.fetchMoreResults

Question

cassandra-connector-assembly-2.0.0 从 github 项目构建。

使用 Scala 2.11.8，cassandra-driver-core-3.1.0

sc.cassandraTable("mykeyspace", "mytable").select("something").where("key=?", key).mapPartitions(par => {
    par.map({ row => (row.getString("something"), 1 ) })
})
.reduceByKey(_ + _).collect().foreach(println)

同样的工作可以很好地读取较少的海量数据

java.lang.NoSuchMethodError: com.datastax.driver.core.ResultSet.fetchMoreResults()Lshade/com/datastax/spark/connector/google/common/util/concurrent/ListenableFuture;
    at com.datastax.spark.connector.rdd.reader.PrefetchingResultSetIterator.maybePrefetch(PrefetchingResultSetIterator.scala:26)
    at com.datastax.spark.connector.rdd.reader.PrefetchingResultSetIterator.next(PrefetchingResultSetIterator.scala:39)
    at com.datastax.spark.connector.rdd.reader.PrefetchingResultSetIterator.next(PrefetchingResultSetIterator.scala:17)
    at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
    at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
    at scala.collection.Iterator$$anon$12.next(Iterator.scala:444)
    at com.datastax.spark.connector.util.CountingIterator.next(CountingIterator.scala:16)
    at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
    at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:194)
    at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
    at org.apache.spark.scheduler.Task.run(Task.scala:85)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

任何人都可以建议或指出问题以及可能的解决方案吗？

Answer 1

有同样的问题

项目中有两个依赖项，它们都有 cassandra-driver-core 作为依赖项

spark-cassandra-connector_2.11-2.0.0-M3 &job-server-api_2.10-0.8.0-SNAPSHOT

spark-cassandra-connecter 预期 ResultSet.fetchMoreResults 具有不同的返回类型，因为它的 shading of guava

预计。 shade.com.datastax.spark.connector.google.common.util.concurrent.ListenableFuture

找到了。com.google.common.util.concurrent.ListenableFuture

切换到 unshaded version of cassandra-connector纠正问题