hadoop - SequenceFile.Writer的sync和syncFs是什么意思？

Question

环境:Hadoop 0.20.2-cdh3u5

我正在尝试使用使用 SequenceFile.Writer 的自定义工具将日志数据 (10G) 上传到 HDFS。

SequenceFile.Writer w = SequenceFile.createWriter(
                hdfs,
                conf,
                p,
                LongWritable.class,
                Text.class,
                4096,
                hdfs.getDefaultReplication(),
                hdfs.getDefaultBlockSize(),
                compressionType,
                codec,
                null,
                new Metadata());

在上传过程中，如果工具崩溃(没有显式调用close()方法)，已经上传的日志会丢失吗？

我应该及时调用sync()还是syncFs()，这两个方法是什么意思？

Answer 1

是的，可能吧。

sync() 创建一个同步点。正如 Tom White (Cloudera) 在《Hadoop - 权威指南》一书中所述

a sync point is a point in the stream which can used by to resynchronize with a record boundary if the reader is "lost" - for example after seeking to an arbitrary position on the stream.

现在 syncFS() 的实现非常简单:

   public void syncFs() throws IOException {
      if (out != null) {
        out.sync();                               // flush contents to file system
      }
    }

其中 out 是一个 FSDataOutputStream。同样，在同一本书中指出:

HDFS provides a method for forcing all buffers to be synchronized to the datanodes via the sync() method on FSDataOutputStream. After a successful call return from sync() HDFS garantees that the data written up to that point in the file is persisted and visible to all readers. In the event of a crash (of the client or HDFS), the data will not be lost.

但是一个脚注警告要查看 bug HDFS-200 , 因为上面提到的可见性并不总是总是受到尊重。