hadoop - hadoop作业中的org.apache.hadoop.fs.BlockMissingException

Question

我在Hadoop工作中遇到以下异常。我无法找出导致此异常的原因。您能帮我解决这个错误吗？为什么会出现此错误？

异常堆栈跟踪:

org.apache.hadoop.fs.BlockMissingException: Could not obtain block: blk_6488469967470272993_1110 file=/media/ephemeral0/hadoop-root/5614861051_themeToProductsPipe_F832FB7243E64F41892F20744AF6A0B7/part-00054
    at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:3158)
    at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:2940)
    at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:3107)
    at java.io.DataInputStream.readFully(DataInputStream.java:195)
    at java.io.DataInputStream.readFully(DataInputStream.java:169)
    at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1576)
    at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1554)
    at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1543)
    at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1538)
    at org.apache.hadoop.mapred.SequenceFileRecordReader.(SequenceFileRecordReader.java:47)
    at org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:64)
    at cascading.tap.hadoop.io.MultiInputFormat$1.operate(MultiInputFormat.java:253)
    at cascading.tap.hadoop.io.MultiInputFormat$1.operate(MultiInputFormat.java:248)
    at cascading.util.Util.retry(Util.java:762)
    at cascading.tap.hadoop.io.MultiInputFormat.getRecordReader(MultiInputFormat.java:247)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:346)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:312)
    at org.apache.hadoop.mapred.Child$3.run(Child.java:205)

Answer 1

该错误意味着客户端无法按时访问包含该数据块的数据节点。这可能是由于网络问题，或者是由于数据节点太忙而无法响应或它们已关闭。

该异常不足以跟踪根本原因。