hadoop - HDFS 用户的 Hive 查询失败

Question

如果我自己运行 Hive shell，我可以查询表。

但是如果我使用 sudo -u hdfs hive 运行 hive shell

然后我所有的查询都失败并显示错误消息

Application application_1447966350718_10654 failed 2 times due to AM Container for appattempt_1447966350718_10654_000002 exited with exitCode: -1000
For more detailed output, check application tracking page:http://vtdevana-cloudera11.dealer.ddc:8088/proxy/application_1447966350718_10654/Then, click on links to logs of each attempt.
Diagnostics: Application application_1447966350718_10654 initialization failed (exitCode=255) with output: main : command provided 0
main : user is nobody
main : requested yarn user is hdfs
Can't create directory /yarn/nm/usercache/hdfs/appcache/application_1447966350718_10654 - Permission denied
Did not create any app directories
Failing this attempt. Failing the application.

我应该怎么做才能使用 hdfs 用户查询 hive？

Answer 1

“/yarn/nm/usercache/hdfs/appcache/application_1447966350718_10654”是本地文件系统上的路径。

在你的yarn-site.xml中，有配置参数:yarn.nodemanager.local-dirs。

这个参数的说明是:

List of directories to store localized files in. An application's localized file directory will be found in: ${yarn.nodemanager.local-dirs}/usercache/${user}/appcache/application_${appid}. Individual containers' work directories, called container_${contid}, will be subdirectories of this.

因此，在您的情况下，此参数已设置为:

yarn.nodemanager.local-dirs=/yarn/nm/ 

因此，当您尝试以 hdfs 用户身份执行 Hive 查询时，它会尝试在您的本地文件系统中创建以下文件夹，但由于缺少权限，操作失败:

${yarn.nodemanager.local-dirs}/usercache/${user}/appcache/application_${appid} 

被翻译成

/yarn/nm/usercache/hdfs/appcache/application_1447966350718_10654

"${yarn.nodemanager.local-dirs}" is replaced by "/yarn/nm"
"${user}" is replaced by "hdfs"
"${appid}" is replaced by "1447966350718_10654"

要解决这个问题，需要给所有用户对顶层文件夹的写入权限:/yarn/nm/