java - 从 Windows 中的 Eclipse 中运行 MapReduce 作业时出错

Question

我在 Linux 机器上安装了伪分布式 Hadoop。我在 eclipse 中做了一些例子，它也安装在那台 Linux 机器上，并且运行良好。现在我想通过 eclipse(安装在 Windows 机器中)执行 MapReduce 作业并访问我的 Linux 机器中已经存在的 HDFS。我编写了以下驱动程序代码:

import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.FileInputFormat;
import org.apache.hadoop.mapred.FileOutputFormat;
import org.apache.hadoop.mapred.JobClient;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;

public class Windows_Driver extends Configured implements Tool{

	public static void main(String[] args) throws Exception {
		
		int exitcode = ToolRunner.run(new Windows_Driver(), args);
		System.exit(exitcode);
		
	}

	@Override
	public int run(String[] arg0) throws Exception {
		
		JobConf conf = new JobConf(Windows_Driver.class);

		conf.set("fs.defaultFS", "hdfs://<Ip address>:50070");

		FileInputFormat.setInputPaths(conf, new Path("sample"));
		FileOutputFormat.setOutputPath(conf, new Path("sam"));
		

		conf.setMapperClass(Win_Mapper.class);
		

		conf.setMapOutputKeyClass(Text.class);
		conf.setMapOutputValueClass(Text.class);

		conf.setOutputKeyClass(Text.class);
		conf.setOutputValueClass(Text.class);

		JobClient.runJob(conf);
		
		
		return 0;

	}

}

以及映射器代码:

import java.io.IOException;

import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.Mapper;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reporter;

public class Win_Mapper extends MapReduceBase implements Mapper<LongWritable, Text,Text, Text> {

	@Override
	public void map(LongWritable key, Text value, OutputCollector<Text, Text> o, Reporter arg3) throws IOException {
		
  
        ...
		
		o.collect(... , ...);
	}

}

当我运行此程序时，出现以下错误:

 SEVERE: PriviledgedActionException as:miracle cause:java.io.IOException: Failed to set permissions of path: \tmp\hadoop-miracle\mapred\staging\miracle1262421749\.staging to 0700
Exception in thread "main" java.io.IOException: Failed to set permissions of path: \tmp\hadoop-miracle\mapred\staging\miracle1262421749\.staging to 0700
	at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:691)
	at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:664)
	at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:514)
	at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:349)
	at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:193)
	at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)
	at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:942)
	at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
	at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
	at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:910)
	at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1353)
	at Windows_Driver.run(Windows_Driver.java:41)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
	at Windows_Driver.main(Windows_Driver.java:16)

如何纠正错误？如何从 Windows 远程访问我的 HDFS？

Answer 1

作业上的submit()方法创建一个内部Jobsubmitter实例，它将执行所有数据验证，包括输入路径、输出路径可用性、文件/目录创建权限和其他事情。在 MR 的不同阶段，它会创建临时目录，并在其中放置 temp.txt 文件。文件。临时目录取自具有属性 hadoop.tmp.dir 的 core-site.xml。您的系统的问题似乎是温度。目录为/tmp/并且运行 MR 作业的用户无权将其 rwx 状态更改为 700。请提供适当的权限并重新运行作业。