exception - 为什么我得到未读的 block 数据 - 非法状态异常？

Question

我只有以下内容:

JavaPairRDD<ImmutableBytesWritable, Result> dataRDD = jsc
            .newAPIHadoopRDD(
                    hbase_conf,
                    TableInputFormat.class,
                    org.apache.hadoop.hbase.io.ImmutableBytesWritable.class,
                    org.apache.hadoop.hbase.client.Result.class);

sparkConf.log().info("Count of data = "+String.valueOf(dataRDD.count()));

我得到这个异常(exception):

Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, server-name): java.lang.IllegalStateException: unread block data
    java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2394)

Answer 1

你可以在这里找到一些提示:https://issues.apache.org/jira/browse/SPARK-1867
一些人用 hadoop-common 库替换了 hadoop-client。您的情况似乎与我的情况类似，需要适当的 hbase jar。我加了 hbase/hbase-0.98.12/lib/*至 spark.executor.extraClasspath它消失了。这是设置此 jar 的另一种方法。 https://groups.google.com/forum/#!topic/spark-users/gXSfbjauAjo