hadoop - MapReduce 错误 UserGroupInformation - PriviledgedActionException

Question

我正在尝试在我的本地机器上运行以下 MapReduce 代码: https://github.com/Jeffyrao/warcbase/blob/extract-links/src/main/java/org/warcbase/data/ExtractLinks.java

但是，我遇到了这个异常:

[main] ERROR UserGroupInformation - PriviledgedActionException as:jeffy (auth:SIMPLE)   cause:java.io.IOException: java.util.concurrent.ExecutionException: java.io.IOException: Resource file:/Users/jeffy/Documents/Eclipse/warcbase/map_backup.txt is not publicly accessable and as such cannot be part of the public cache.
Exception in thread "main" java.io.IOException: java.util.concurrent.ExecutionException: java.io.IOException: Resource file:/Users/jeffy/Documents/Eclipse/warcbase/map_backup.txt is not publicly accessable and as such cannot be part of the public cache.
at org.apache.hadoop.mapred.LocalDistributedCacheManager.setup(LocalDistributedCacheManager.java:144)
at org.apache.hadoop.mapred.LocalJobRunner$Job.<init>(LocalJobRunner.java:155)
at org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:625)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:391)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1269)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1266)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:394)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1266)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1287)
at org.warcbase.data.ExtractLinks.run(ExtractLinks.java:254)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.warcbase.data.ExtractLinks.main(ExtractLinks.java:270)
Caused by: java.util.concurrent.ExecutionException: java.io.IOException: Resource file:/Users/jeffy/Documents/Eclipse/warcbase/map_backup.txt is not publicly accessable and as such cannot be part of the public cache.
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at org.apache.hadoop.mapred.LocalDistributedCacheManager.setup(LocalDistributedCacheManager.java:140)
... 14 more

我认为这个问题是因为我试图将文件添加到 DistributedCache(查看我的代码第 81-86 行和第 235 行)。欢迎任何建议。谢谢!

Answer 1

在本地环境中添加 DistributedCache 运行 Hadoop 2 作业时，我遇到了类似的问题。最后，我的问题的原因是 Hadoop 2 不仅验证路径本身具有公共(public)执行和读取访问权限，而且还验证其所有祖先目录都应具有执行权限。在这种情况下，如果 "/"或 "/Users"没有 755 权限，文件仍然无法添加到公共(public)缓存。

参见方法 static boolean ancestorsHaveExecutePermissions(FileSystem fs,
Path path, LoadingCache<Path,Future<FileStatus>> statCache)在 Hadoop 类 FSDownload.java

一种解决方案是授予对所有目录的权限(听起来不安全)。

更好的解决方案是确保所有要缓存的资源文件都在 /tmp 中默认具有 >755 权限的文件夹或任何其他文件夹。