hadoop - Cassandra pig 插入异常

Question

我正在使用 pig CassandraStroage() 将一个大数据集插入到 cassandra 中，运行 4 小时后，它崩溃并出现以下异常:

java.lang.NullPointerException
        at org.apache.cassandra.dht.RandomPartitioner.getToken(RandomPartitioner.java:134)
        at org.apache.cassandra.dht.RandomPartitioner.getToken(RandomPartitioner.java:36)
        at org.apache.cassandra.client.RingCache.getRange(RingCache.java:129)
        at org.apache.cassandra.hadoop.ColumnFamilyRecordWriter.write(ColumnFamilyRecordWriter.java:127)
        at org.apache.cassandra.hadoop.ColumnFamilyRecordWriter.write(ColumnFamilyRecordWriter.java:62)
        at org.apache.cassandra.hadoop.pig.CassandraStorage.putNext(Unknown Source)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOut
putFormat.java:138)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOut
putFormat.java:97)
        at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:498)
        at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:239)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:232)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:53)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)

知道为什么会这样吗？

Answer 1

虽然不是您的问题的原因，但值得注意的是，当尝试插入到指定分区键不存在的列族时，可能会发生此错误。

在这种情况下，它会在第一次命中 reducer 类时抛出异常。