hadoop - 出现错误nativeio.NativeIO:在Windows中通过Eclipse运行MapReduce时无法初始化NativeIO库

Question

我已经通过引用https://github.com/MuhammadBilalYar/Hadoop-On-Window/wiki/Step-by-step-Hadoop-2.8.0-installation-on-Window-10在Windows 7上配置了hadoop

运行map reduce程序(已成功编译)时，出现以下完全错误

18/10/03 20:04:21 INFO util.NativeCodeLoader: Loaded the native-hadoop library
18/10/03 20:04:21 ERROR nativeio.NativeIO: Unable to initialize NativeIO libraries
java.lang.NoClassDefFoundError: org/apache/hadoop/io/nativeio/NativeIO$POSIX$Stat
    at org.apache.hadoop.io.nativeio.NativeIO.initNative(Native Method)
    at org.apache.hadoop.io.nativeio.NativeIO.<clinit>(NativeIO.java:89)
    at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:655)
    at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:514)
    at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:349)
    at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:193)
    at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:942)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:550)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:580)
    at com.example.hadoop.textToparquet.TextParquetConverter.run(TextParquetConverter.java:41)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
    at com.example.hadoop.textToparquet.TextParquetConverter.main(TextParquetConverter.java:22)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.nativeio.NativeIO$POSIX$Stat
    at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 18 more
18/10/03 20:04:21 ERROR security.UserGroupInformation: PriviledgedActionException as:vxr83 cause:java.io.IOException: Failed to set permissions of path: \tmp\hadoop-vxr83\mapred\staging\vxr831620787823\.staging to 0700
Exception in thread "main" java.io.IOException: Failed to set permissions of path: \tmp\hadoop-vxr83\mapred\staging\vxr831620787823\.staging to 0700
    at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:691)
    at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:664)
    at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:514)
    at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:349)
    at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:193)
    at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:942)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:550)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:580)
    at com.example.hadoop.textToparquet.TextParquetConverter.run(TextParquetConverter.java:41)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
    at com.example.hadoop.textToparquet.TextParquetConverter.main(TextParquetConverter.java:22)

请帮忙

Answer 1

问题1:- java.io.IOException:无法设置以下路径的权限:\ tmp \ hadoop-vxr83 \ mapred \ staging \ vxr831620787823.staging到0700

解决方案:-

从以下URL下载patch-hadoop_7682-1.0.x-win，并将其放在C:\ hadoop-2.7.2 \ lib \下
https://github.com/congainc/patch-hadoop_7682-1.0.x-win

将此jar文件添加到Eclipse Java项目类路径

在您的程序中的

中，添加以下配置:

job.getConfiguration()。set(“fs.file.impl”，“com.conga.services.hadoop.patch.HADOOP_7682.WinLocalFileSystem”);

问题2:-错误nativeio.NativeIO:无法初始化NativeIO库

解决方案:-将vm arugment添加到Eclipse中的map reduce程序的运行配置中，作为-Djava.library.path = C:\ hadoop-2.7.2 \ lib \ native