java - 在hadoop中运行作业-错误安全性.UserGroupInformation

Question

我在hadoop中运行了一个作业(jar文件)，(我的jar文件还有一些其他jar文件，并且有一个zip文件。)但是我遇到了这个错误。如果有人可以帮助我，并说出此错误的原因，我将非常感谢。我认为此错误是由于正在运行的jar文件中存在一个zip文件，但是我如何手动将其解压缩？

     ERROR security.UserGroupInformation: PriviledgedActionException
     as:hduser cause:java.util.zip.ZipException: error reading zip file
    at java.util.zip.ZipFile.read(Native Method)
    at java.util.zip.ZipFile.access$1400(ZipFile.java:60)
    at java.util.zip.ZipFile$ZipFileInputStream.read(ZipFile.java:717)
    at java.util.zip.ZipFile$ZipFileInflaterInputStream.fill(ZipFile.java:419)
    at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
    at java.io.FilterInputStream.read(FilterInputStream.java:107)
    at org.apache.hadoop.util.RunJar.unJar(RunJar.java:55)
    at org.apache.hadoop.filecache.TrackerDistributedCacheManager.downloadCacheObject(TrackerDistributedCacheManager.java:438)
    at org.apache.hadoop.filecache.TrackerDistributedCacheManager.localizePublicCacheObject(TrackerDistributedCacheManager.java:477)
    at org.apache.hadoop.filecache.TrackerDistributedCacheManager.getLocalCache(TrackerDistributedCacheManager.java:192)
    at org.apache.hadoop.filecache.TaskDistributedCacheManager.setupCache(TaskDistributedCacheManager.java:182)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.<init>(LocalJobRunner.java:131)
    at org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:592)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:1013)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:550)
    at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:259)
    at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
    at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:160)

Answer 1

根据此日志:

as:hduser cause:java.util.zip.ZipException: error reading zip file
 at java.util.zip.ZipFile.read(Native Method)
 at java.util.zip.ZipFile.access$1400(ZipFile.java:60)

似乎您无权阅读该zip文件。

请尝试由您的用户读取该数据？ (sudo -u hduser head -l 1 /path/of/file.zip)，我想您也会看到权限错误。您应该首先对您的hadoop用户(hduser？)调用读取许可权

被SELinux阻止是另一个可能的问题，请检查日志:/var/log/audit/audit.log

如果您想在Linux文件系统中解压缩该zip文件，则可以运行

$解压缩/ path / of / file

如果这是hdfs路径，则可能需要先获取它(如果我可以操作的话)