hadoop - 从Eclipse运行时，Hadoop映射减少程序错误-6ren

hadoop - 从Eclipse运行时，Hadoop映射减少程序错误

转载作者：行者123 更新时间：2023-12-02 21:59:28

我是hadoop map减少编程的新手。我尝试了wordcount的示例代码，并从Eclipse运行它。当我以本地模式从Eclipse运行它时，它也已经成功运行，当我构建一个jar并在hadoop上运行它时，它也成功运行了。但是，每当我尝试从Hadoop上的Eclipse运行它时，都会因该错误而失败。尽管访问hdfs位置没有问题，因为每次失败都会在hdfs中创建输出目录。我真的在这里没有头绪。请帮忙。

14/08/09 13:42:27 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
14/08/09 13:42:27 WARN mapred.JobClient: No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
14/08/09 13:42:27 INFO mapred.FileInputFormat: Total input paths to process : 1
14/08/09 13:42:28 WARN conf.Configuration: fs.default.name is deprecated. Instead, use fs.defaultFS
14/08/09 13:42:29 INFO mapred.JobClient: Running job: job_201408091257_0004
14/08/09 13:42:30 INFO mapred.JobClient:  map 0% reduce 0%
14/08/09 13:42:51 INFO mapred.JobClient: Task Id : attempt_201408091257_0004_m_000000_0, Status : FAILED
java.lang.RuntimeException: Error in configuring object
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:413)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
    at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja
14/08/09 13:42:53 INFO mapred.JobClient: Task Id : attempt_201408091257_0004_m_000001_0, Status : FAILED
java.lang.RuntimeException: Error in configuring object
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:413)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
    at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja
14/08/09 13:43:15 INFO mapred.JobClient: Task Id : attempt_201408091257_0004_m_000000_1, Status : FAILED
java.lang.RuntimeException: Error in configuring object
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:413)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
    at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja
14/08/09 13:43:15 INFO mapred.JobClient: Task Id : attempt_201408091257_0004_m_000001_1, Status : FAILED
java.lang.RuntimeException: Error in configuring object
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:413)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
    at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja
14/08/09 13:43:28 INFO mapred.JobClient: Task Id : attempt_201408091257_0004_m_000000_2, Status : FAILED
java.lang.RuntimeException: Error in configuring object
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:413)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
    at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja
14/08/09 13:43:29 INFO mapred.JobClient: Task Id : attempt_201408091257_0004_m_000001_2, Status : FAILED
java.lang.RuntimeException: Error in configuring object
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:413)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
    at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja
14/08/09 13:43:47 INFO mapred.JobClient: Job complete: job_201408091257_0004
14/08/09 13:43:47 INFO mapred.JobClient: Counters: 7
14/08/09 13:43:47 INFO mapred.JobClient:   Job Counters 
14/08/09 13:43:47 INFO mapred.JobClient:     Failed map tasks=1
14/08/09 13:43:47 INFO mapred.JobClient:     Launched map tasks=8
14/08/09 13:43:47 INFO mapred.JobClient:     Data-local map tasks=8
14/08/09 13:43:47 INFO mapred.JobClient:     Total time spent by all maps in occupied slots (ms)=143034
14/08/09 13:43:47 INFO mapred.JobClient:     Total time spent by all reduces in occupied slots (ms)=0
14/08/09 13:43:47 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
14/08/09 13:43:47 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
14/08/09 13:43:47 INFO mapred.JobClient: Job Failed: NA
Exception in thread "main" java.io.IOException: Job failed!
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1372)
    at test.WordCount.run(WordCount.java:46)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
    at test.WordCount.main(WordCount.java:51)

码

WordCount.java

package test;

import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.FileInputFormat;
import org.apache.hadoop.mapred.FileOutputFormat;
import org.apache.hadoop.mapred.JobClient;
import org.apache.hadoop.mapred.JobConf;

import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;

public class WordCount extends Configured implements Tool {

  @Override
  public int run(String[] args) throws Exception {

    if (args.length != 2) {
      System.out.printf(
          "Usage: %s [generic options] <input dir> <output dir>\n", getClass()
              .getSimpleName());
      ToolRunner.printGenericCommandUsage(System.out);
      return -1;
    }

    JobConf conf = new JobConf(getConf(), WordCount.class);
    conf.setJobName(this.getClass().getName());

    FileInputFormat.setInputPaths(conf, new Path(args[0]));
    FileOutputFormat.setOutputPath(conf, new Path(args[1]));

    conf.setMapperClass(WordMapper.class);
    conf.setReducerClass(SumReducer.class);

    conf.setMapOutputKeyClass(Text.class);
    conf.setMapOutputValueClass(IntWritable.class);

    conf.setOutputKeyClass(Text.class);
    conf.setOutputValueClass(IntWritable.class);

    conf.set("fs.defaultFS", "hdfs://localhost:8020");
    conf.set("mapred.job.tracker", "localhost:8021");

    conf.addResource(new Path("/etc/hadoop/conf/core-site.xml"));
    conf.addResource(new Path("/etc/hadoop/conf/hdfs-site.xml"));

    conf.setJarByClass(test.WordCount.class);

    JobClient.runJob(conf);
    return 0;
  }

  public static void main(String[] args) throws Exception {
    int exitCode = ToolRunner.run(new WordCount(), args);
    System.exit(exitCode);
  }
}

SumReducer.java

package test;
import java.io.IOException;
import java.util.Iterator;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.Reducer;
import org.apache.hadoop.mapred.Reporter;

public class SumReducer extends MapReduceBase implements
    Reducer<Text, IntWritable, Text, IntWritable> {

  @Override
  public void reduce(Text key, Iterator<IntWritable> values,
      OutputCollector<Text, IntWritable> output, Reporter reporter)
      throws IOException {

    int wordCount = 0;
    while (values.hasNext()) {
      IntWritable value = values.next();
      wordCount += value.get();
    }
    if (wordCount < 100)
    {
        output.collect(key, new IntWritable(wordCount));
    }
  }
}

WordMapper.java

package test;
import java.io.IOException;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.Mapper;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reporter;

public class WordMapper extends MapReduceBase implements
    Mapper<LongWritable, Text, Text, IntWritable> {

  @Override
  public void map(LongWritable key, Text value,
      OutputCollector<Text, IntWritable> output, Reporter reporter)
      throws IOException {
    String s = value.toString();
    for (String word : s.split("\\W+")) {
      if (word.length() > 0) {
        output.collect(new Text(word), new IntWritable(1));
      }
    }
  }
}

最佳答案

右键单击您的项目，构建路径->配置构建路径。选择“库”选项卡，单击“添加外部JAR”，然后将所有必需的JAR文件复制到hadoop目录中。我想，这将解决您的问题。

关于hadoop - 从Eclipse运行时，Hadoop映射减少程序错误，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/25248109/

文章推荐： sql - sql server中的列到行？

文章推荐： javascript - 从服务器路径检索图像

npm 安装不起作用 | npm 错误!路径 | npm 错误!代码 | npm 错误!错误 | npm 错误!系统调用 | npm 错误!恩恩特
我已经使用 vue-cli 两个星期了，直到今天一切正常。我在本地建立这个项目。 https://drive.google.com/open?id=0BwGw1zyyKjW7S3RYWXRaX24tQ
python - pytesseract 错误 Windows 错误 [错误 2]
您好，我正在尝试使用 python 库 pytesseract 从图像中提取文本。请找到代码: from PIL import Image from pytesseract import image_
C 错误 TLS 错误
我的错误 /usr/bin/ld: errno: TLS definition in /lib/libc.so.6 section .tbss mismatches non-TLS reference
r - 错误 `contrasts' 错误
我已经训练了一个模型，我正在尝试使用 predict函数但它返回以下错误。 Error in contrasts<-(*tmp*, value = contr.funs[1 + isOF[nn]])
postgresql - PowerBI 直接查询连接到 PostgreSQL 错误。 OLE 或 ODBC 错误 : [Expression. 错误] 我们无法将表达式折叠到数据源
根据Microsoft DataConnectors的信息我想通过 this ODBC driver 创建一个从 PowerBi 到 PostgreSQL 的连接器使用直接查询。我重用了 Micros
java - Android MediaPlayer 错误(在状态 1 中开始调用；错误 (-38, 0)；错误 (-38,0))
我已经为 SoundManagement 创建了一个包，其中有一个扩展 MediaPlayer 的类。我希望全局控制这个变量。这是我的代码: package soundmanagement; impo
heroku - PG::错误:错误:Heroku的内存不足
我在Heroku上部署了一个应用程序。我正在使用免费服务。我经常收到以下错误消息。 PG::Error: ERROR: out of memory 如果刷新浏览器，就可以了。但是随后，它又随机发生
.htaccess - .htaccess 错误，错误 500
我正在运行 LAMP 服务器，这个 .htaccess 给我一个 500 错误。其作用是过滤关键字并重定向到相应的域名。 Options +FollowSymLinks RewriteEngine
robocopy 错误，错误 32 (0x00000020)
我有两个驱动器 A 和 B。使用 python 脚本，我在“A”驱动器中创建一些文件，并运行 powerscript，该脚本以 1 秒的间隔将驱动器 A 中的所有文件复制到驱动器 B。我在 powe
postgresql 错误 - 错误 : input is out of range
下面的函数一直返回这个错误信息。我认为可能是 double_precision 字段类型导致了这种情况，我尝试使用 CAST，但要么不是这样，要么我没有做对...帮助？这是错误: ERROR: i
mysql - 错误 1064 MySQL 错误
这个问题已经有答案了: Syntax error due to using a reserved word as a table or column name in MySQL (1 个回答) 已关闭
mysql - mysql 错误(错误 1136)
我的数据库有这个小问题。我创建了一个表“articoli”，其中包含商品的品牌、型号和价格。每篇文章都由一个 id (ID_ARTICOLO)` 定义，它是一个自动递增字段。好吧，现在当我尝试插
c++ - 错误 C2228、错误 C2275
我是新来的。我目前正在 DeVry 在线学习中级 C++ 编程。我们正在使用 C++ Primer Plus 这本书，到目前为止我一直做得很好。我的老师最近向我们扔了一个曲线球。我目前的任务是这样的:
c++ - 错误 LNK2019 错误 C++
这个问题在这里已经有了答案: What is an undefined reference/unresolved external symbol error and how do I fix it?
html - 奇怪的 IE7 错误/错误
我的网站中有一段代码有问题；此错误仅发生在 Internet Explorer 7 中。我没有在这里发布我所有的 HTML/CSS 标记，而是发布了网站的一个版本 here . 如您所见，我在列中有
node.js - 错误!错误 : EPERM,
如果尝试在 USB 设备上构建 node.js 应用程序时在我的树莓派上使用 npm 时遇到一些问题。 package.json 看起来像这样: { "name" : "node-todo",
python - 无 Python 错误/错误？
在 Python 中，您有 None单例，在某些情况下表现得很奇怪: >>> a = None >>> type(a) >>> isinstance(a,None) Traceback (most
java - Android Studio 错误 - 错误 :java. util.concurrent.ExecutionException : com. android.tools.aapt2.Aapt2Exception:AAPT 错误
这是我的 build.gradle (Module:app) 文件: apply plugin: 'com.android.application' android { compileSdkV
android - 任务 ':app:compileDebugJavaWithJavac' 执行失败。错误 :(2055, 52) 错误 : ';' expected Error:(2055, 59) 错误:<标识符> 预期
我是 android 的新手，我的项目刚才编译和运行正常，但在我尝试实现抽屉导航后，它给了我这个错误 FAILURE: Build failed with an exception. What wen
PHP 7.2.25 错误!= 错误？
谁能解释一下？我想我正在做一些非常愚蠢的事情，并且急切地等待着启蒙。我得到这个输出: phpversion() == 7.2.25-1+0~20191128.32+debian8~1.gbp108

行者123

个人简介

我是一名优秀的程序员,十分优秀！

作者热门文章

滴滴打车优惠券免费领取

全站热门文章

首页

博学

6Ren·AI

商城

hadoop - 从Eclipse运行时，Hadoop映射减少程序错误