hadoop - 使用HDFS NFS网关时的输入/输出错误

Question

尝试使用已安装的HDFS NFS Gateway中的文件时出现“输入/输出错误”。尽管在Ambari中有set dfs.namenode.accesstime.precision=3600000，但仍然存在。例如，执行类似...

$ hdfs dfs -cat /hdfs/path/to/some/tsv/file | sed -e "s/$NULL_WITH_TAB/$TAB/g" | hadoop fs -put -f - /hdfs/path/to/some/tsv/file
$ echo -e "Lines containing null (expect zero): $(grep -c "\tnull\t" /nfs/hdfs/path/to/some/tsv/file)"

尝试从tsv中删除空值时，然后根据NFS位置检查该tsv中的空值会引发错误，但我在其他许多地方都看到了它(再次，已经有dfs.namenode.accesstime.precision = 3600000)。任何人都知道为什么会发生这种情况或调试建议吗？在这种情况下，谁能解释确切的“访问时间”？

Answer 1

从关于Apache hadoop mailing list的讨论中:

I think access time refers to the POSIX atime attribute for files, the “time of last access” as described here for instance (https://www.unixtutorial.org/atime-ctime-mtime-in-unix-filesystems). While HDFS keeps a correct modification time (mtime), which is important, easy and cheap, it only keeps a very low-resolution sense of last access time, which is less important, and expensive to monitor and record, as described here (https://issues.apache.org/jira/browse/HADOOP-1869) and here (https://superuser.com/questions/464290/why-is-cat-not-changing-the-access-time).

However, to have a conforming NFS api, you must present atime, and so the HDFS NFS implementation does. But first you have to configure it on. [...] many sites have been advised to turn it off entirely by setting it to zero, to improve HDFS overall performance. See for example here ( https://community.hortonworks.com/articles/43861/scaling-the-hdfs-namenode-part-4-avoiding-performa.html, section "Don’t let Reads become Writes”). So if your site has turned off atime in HDFS, you will need to turn it back on to fully enable NFS. Alternatively, you can maintain optimum efficiency by mounting NFS with the “noatime” option, as described in the document you reference.

[...] check under /var/log, eg with find /var/log -name ‘*nfs3*’ -print