- html - 出于某种原因,IE8 对我的 Sass 文件中继承的 html5 CSS 不友好?
- JMeter 在响应断言中使用 span 标签的问题
- html - 在 :hover and :active? 上具有不同效果的 CSS 动画
- html - 相对于居中的 html 内容固定的 CSS 重复背景?
我正在使用Hadoop Map-Reduce API,并执行一个简单的任务来计算arounf 20MB大型输入文件中指定的一年中的最高模板度。
一切运行正常,即Mapper任务运行正常,Reducer任务运行正常,输出文件也生成良好。
但是问题在于,我在Hadoop UI页面上看不到任何内容,在“作业”选项卡,“作业进度”甚至“作业历史”中都看不到。
这是我的wordcound java文件:
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
public class WordCount extends Configured implements Tool{
/**
* Main function which calls the run method and passes the args using ToolRunner
* @param args Two arguments input and output file paths
* @throws Exception
*/
public static void main(String[] args) throws Exception{
int exitCode = ToolRunner.run(new WordCount(), args);
System.exit(exitCode);
}
/**
* Run method which schedules the Hadoop Job
* @param args Arguments passed in main function
*/
public int run(String[] args) throws Exception {
if (args.length != 2) {
System.err.printf("Usage: %s needs two arguments <input> <output> files\n",
getClass().getSimpleName());
return -1;
}
//Initialize the Hadoop job and set the jar as well as the name of the Job
Job job = new Job();
job.setJarByClass(WordCount.class);
job.setJobName("WordCounter");
//Add input and output file paths to job based on the arguments passed
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.setOutputFormatClass(TextOutputFormat.class);
//Set the MapClass and ReduceClass in the job
job.setMapperClass(MapClass.class);
job.setReducerClass(ReduceClass.class);
//Wait for the job to complete and print if the job was successful or not
int returnValue = job.waitForCompletion(true) ? 0:1;
if(job.isSuccessful()) {
System.out.println("Job was successful");
} else if(!job.isSuccessful()) {
System.out.println("Job was not successful");
}
return returnValue;
}
}
hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/Users/bng/Documnents/hDir/hdfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/Users/bng/Documnents/hDir/hdfs/data</value >
</property>
</configuration>
core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost/</value>
</property>
<property>
<name>dfs.http.address</name>
<value>50070</value>
</property>
</configuration>
mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
yarn-site.xml
<?xml version="1.0"?>
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>localhost</value>
</property>
</configuration>
0 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
202 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - session.id is deprecated. Instead, use dfs.metrics.session-id
203 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with processName=JobTracker, sessionId=
402 [main] WARN org.apache.hadoop.mapreduce.JobSubmitter - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
408 [main] WARN org.apache.hadoop.mapreduce.JobSubmitter - No job jar file set. User classes may not be found. See Job or Job#setJar(String).
419 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
471 [main] INFO org.apache.hadoop.mapreduce.JobSubmitter - number of splits:1
593 [main] INFO org.apache.hadoop.mapreduce.JobSubmitter - Submitting tokens for job: job_local1149743576_0001
764 [main] INFO org.apache.hadoop.mapreduce.Job - The url to track the job: http://localhost:8080/
766 [Thread-10] INFO org.apache.hadoop.mapred.LocalJobRunner - OutputCommitter set in config null
766 [main] INFO org.apache.hadoop.mapreduce.Job - Running job: job_local1149743576_0001
774 [Thread-10] INFO org.apache.hadoop.mapred.LocalJobRunner - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
815 [Thread-10] INFO org.apache.hadoop.mapred.LocalJobRunner - Waiting for map tasks
816 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.LocalJobRunner - Starting task: attempt_local1149743576_0001_m_000000_0
859 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.yarn.util.ProcfsBasedProcessTree - ProcfsBasedProcessTree currently is supported only on Linux.
860 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.Task - Using ResourceCalculatorProcessTree : null
865 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.MapTask - Processing split: file:/Users/bng/Downloads/hadoop-2.6.4/files/input.txt:0+366
995 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.MapTask - (EQUATOR) 0 kvi 26214396(104857584)
995 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.MapTask - mapreduce.task.io.sort.mb: 100
995 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.MapTask - soft limit at 83886080
998 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.MapTask - bufstart = 0; bufvoid = 104857600
998 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.MapTask - kvstart = 26214396; length = 6553600
1003 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.MapTask - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
1010 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.LocalJobRunner -
1011 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.MapTask - Starting flush of map output
1011 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.MapTask - Spilling map output
1011 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.MapTask - bufstart = 0; bufend = 594; bufvoid = 104857600
1011 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.MapTask - kvstart = 26214396(104857584); kvend = 26214172(104856688); length = 225/6553600
1020 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.MapTask - Finished spill 0
1024 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.Task - Task:attempt_local1149743576_0001_m_000000_0 is done. And is in the process of committing
1032 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.LocalJobRunner - map
1032 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.Task - Task 'attempt_local1149743576_0001_m_000000_0' done.
1033 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.LocalJobRunner - Finishing task: attempt_local1149743576_0001_m_000000_0
1033 [Thread-10] INFO org.apache.hadoop.mapred.LocalJobRunner - map task executor complete.
1035 [Thread-10] INFO org.apache.hadoop.mapred.LocalJobRunner - Waiting for reduce tasks
1035 [pool-3-thread-1] INFO org.apache.hadoop.mapred.LocalJobRunner - Starting task: attempt_local1149743576_0001_r_000000_0
1041 [pool-3-thread-1] INFO org.apache.hadoop.yarn.util.ProcfsBasedProcessTree - ProcfsBasedProcessTree currently is supported only on Linux.
1041 [pool-3-thread-1] INFO org.apache.hadoop.mapred.Task - Using ResourceCalculatorProcessTree : null
1044 [pool-3-thread-1] INFO org.apache.hadoop.mapred.ReduceTask - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@6a57da8b
1058 [pool-3-thread-1] INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl - MergerManager: memoryLimit=1336252800, maxSingleShuffleLimit=334063200, mergeThreshold=881926912, ioSortFactor=10, memToMemMergeOutputsThreshold=10
1060 [EventFetcher for fetching Map Completion Events] INFO org.apache.hadoop.mapreduce.task.reduce.EventFetcher - attempt_local1149743576_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
1092 [localfetcher#1] INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher - localfetcher#1 about to shuffle output of map attempt_local1149743576_0001_m_000000_0 decomp: 710 len: 714 to MEMORY
1108 [localfetcher#1] INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput - Read 710 bytes from map-output for attempt_local1149743576_0001_m_000000_0
1141 [localfetcher#1] INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl - closeInMemoryFile -> map-output of size: 710, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->710
1142 [EventFetcher for fetching Map Completion Events] INFO org.apache.hadoop.mapreduce.task.reduce.EventFetcher - EventFetcher is interrupted.. Returning
1143 [pool-3-thread-1] INFO org.apache.hadoop.mapred.LocalJobRunner - 1 / 1 copied.
1143 [pool-3-thread-1] INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
1160 [pool-3-thread-1] INFO org.apache.hadoop.mapred.Merger - Merging 1 sorted segments
1160 [pool-3-thread-1] INFO org.apache.hadoop.mapred.Merger - Down to the last merge-pass, with 1 segments left of total size: 702 bytes
1162 [pool-3-thread-1] INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl - Merged 1 segments, 710 bytes to disk to satisfy reduce memory limit
1163 [pool-3-thread-1] INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl - Merging 1 files, 714 bytes from disk
1165 [pool-3-thread-1] INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl - Merging 0 segments, 0 bytes from memory into reduce
1165 [pool-3-thread-1] INFO org.apache.hadoop.mapred.Merger - Merging 1 sorted segments
1166 [pool-3-thread-1] INFO org.apache.hadoop.mapred.Merger - Down to the last merge-pass, with 1 segments left of total size: 702 bytes
1167 [pool-3-thread-1] INFO org.apache.hadoop.mapred.LocalJobRunner - 1 / 1 copied.
1186 [pool-3-thread-1] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
1193 [pool-3-thread-1] INFO org.apache.hadoop.mapred.Task - Task:attempt_local1149743576_0001_r_000000_0 is done. And is in the process of committing
1195 [pool-3-thread-1] INFO org.apache.hadoop.mapred.LocalJobRunner - 1 / 1 copied.
1195 [pool-3-thread-1] INFO org.apache.hadoop.mapred.Task - Task attempt_local1149743576_0001_r_000000_0 is allowed to commit now
1196 [pool-3-thread-1] INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter - Saved output of task 'attempt_local1149743576_0001_r_000000_0' to file:/Users/bng/Downloads/hadoop-2.6.4/files/output/_temporary/0/task_local1149743576_0001_r_000000
1197 [pool-3-thread-1] INFO org.apache.hadoop.mapred.LocalJobRunner - reduce > reduce
1197 [pool-3-thread-1] INFO org.apache.hadoop.mapred.Task - Task 'attempt_local1149743576_0001_r_000000_0' done.
1197 [pool-3-thread-1] INFO org.apache.hadoop.mapred.LocalJobRunner - Finishing task: attempt_local1149743576_0001_r_000000_0
1197 [Thread-10] INFO org.apache.hadoop.mapred.LocalJobRunner - reduce task executor complete.
1772 [main] INFO org.apache.hadoop.mapreduce.Job - Job job_local1149743576_0001 running in uber mode : false
1774 [main] INFO org.apache.hadoop.mapreduce.Job - map 100% reduce 100%
1775 [main] INFO org.apache.hadoop.mapreduce.Job - Job job_local1149743576_0001 completed successfully
1784 [main] INFO org.apache.hadoop.mapreduce.Job - Counters: 30
File System Counters
FILE: Number of bytes read=2542
FILE: Number of bytes written=495530
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=1
Map output records=57
Map output bytes=594
Map output materialized bytes=714
Input split bytes=119
Combine input records=0
Combine output records=0
Reduce input groups=47
Reduce shuffle bytes=714
Reduce input records=57
Reduce output records=47
Spilled Records=114
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=10
Total committed heap usage (bytes)=468713472
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=366
File Output Format Counters
Bytes Written=430
Job was successful
最佳答案
问题是通过STS / eclipse运行时的作业配置。
因此,在运行方法中添加了作业配置,并配置了 yarn 资源管理器和defaultFS,如下所示:
public int run(String[] args) throws Exception {
if (args.length != 2) {
System.err.printf("Usage: %s needs two arguments <input> <output> files\n",
getClass().getSimpleName());
return -1;
}
Configuration configuration = getConf();
configuration.set("fs.defaultFS", "hdfs://172.20.12.168");
configuration.set("mapreduce.jobtracker.address", "localhost:54311");
configuration.set("mapreduce.framework.name", "yarn");
configuration.set("yarn.resourcemanager.address", "127.0.0.1:8032");
//Initialize the Hadoop job and set the jar as well as the name of the Job
Job job = new Job();
job.setJarByClass(WordCount.class);
job.setJobName("WordCounter");
//Add input and output file paths to job based on the arguments passed
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.setOutputFormatClass(TextOutputFormat.class);
//Set the MapClass and ReduceClass in the job
job.setMapperClass(MapClass.class);
job.setReducerClass(ReduceClass.class);
//Wait for the job to complete and print if the job was successful or not
int returnValue = job.waitForCompletion(true) ? 0:1;
if(job.isSuccessful()) {
System.out.println("Job was successful");
} else if(!job.isSuccessful()) {
System.out.println("Job was not successful");
}
return returnValue;
}
关于hadoop - Hadoop UI未显示作业选项卡,作业进度和作业历史记录,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/44045121/
我希望使用 API 根据处理 Q 的大小更改运行的 Web 作业实例的数量,我知道我可以在门户中设置规则,但最短聚合时间为 60 分钟,并且我如果我们突然遇到大量工作,不希望系统在扩展之前等待 60
假设我有一个 spark 应用程序并且有两个操作导致两个 spark 作业。 //spark Application //Spark Job1 .... erro
大家好! 作为我对Java的自学的一部分,我正在尝试完成可用的Java初学者分配之一here(非常古老的东西-2001) 问题是我不知道如何应对这个挑战:(我将不胜感激任何建议,因为该解决方案不再可用
我一直在使用 HADOOP 1.2.1 服务器,并在那里执行许多 pig 作业。最近,我考虑将我的 Hadoop 服务器更改为 HADOOP 2.2.0。所以我在 HADOOP 2.2.0 中尝试了一
好的,我修复了静态错误。现在我只是想找出为什么每个对象都得到相同的条目(即相同的名字、年龄、体重等)。这是代码: package classlab3b; import classlab3B.BodyM
我的家庭作业中的一个问题需要一些帮助,我已经尝试了大约一个小时,但无法运行。 列出购买商品数量超过每位顾客平均商品数量的顾客 表格如下: Customer(Cnum, CustomerName, Ad
Kubernetes Jobs重复创建 Pod,直到指定数量的容器成功终止。作业通常与更高级别的CronJob机制一起使用,该机制会按循环计划自动启动新作业。 定期使用 Jobs 和 CronJobs
我有以下工作类(我已经删除了实际的工作代码): @On("0 0 1 * * ?") public class DailyJob extends Job { @Override pub
假设您将 cron 作业配置为每分钟运行一次以做某事。如果实际任务运行时间超过一分钟会发生什么? cron 会创建另一个作业实例/线程吗?还是 cron 会等待并确保上一次运行完成? 谢谢! 最佳答案
我们正在使用 TeamCity 7 并想知道是否可以仅在前一个步骤失败时才运行步骤?我们在构建步骤配置中的选项让您可以选择仅在所有步骤都成功时执行,即使步骤失败,或者始终运行它。 有没有办法仅在前一个
我在 oracle 中编写作业以执行存储过程,但是当时机成熟时,它不会无缘无故地发生任何事情。 是否有某种日志可以让我查看是否发生了错误或其他事情? 我使用 dbms_job 包来创建作业 恩克斯。
我正在用 Java 创建一个用于文件共享的 p2p 应用程序。每个对等节点都将在我的机器上的不同端口上运行并监听请求。但我遇到的问题是,当创建 PeerNode 实例时,我的代码会进入无限循环。以下是
我正在尝试创建一个队列,但当我运行 php artisanqueue:work 时它不起作用,我在终端中得到的只是 [2017-11-30 19:56:27] Processing: App\Jobs
我正在使用PHP库phpseclib0.2.2将SSH自动化到我的一台服务器中。我将其设置为每5分钟运行一次的cron任务。 在设置完它并确保其运行等情况下注销后,我看到了以下内容: $ logout
有没有办法获取多分支管道作业扫描收集到的所有分支的名称? 我想设置一个依赖于现有构建作业的夜间构建,因此需要检查多分支作业是否包含某些特定分支。另一种方法是检查现有作业。 最佳答案 我通过使用 Jen
我在编程方面还很陌生,我不太确定如何完成分配给我的学校作业。 Write a function void print_min(unsigned char a, short b,int c),which
我的作业有问题,需要帮助! 问题 1: 完成下面的 Java 方法,以便 raiseToPower(x,n) 将数字 x 提高到整数 n 次方(即计算值 xn )。请记住 x-n = 1/xn,x0
我正在做一项家庭作业,该作业有四个文本字段和一个文本区域,以及一个将文本字段和文本区域保存到文本文件的按钮,每行一个元素。然后,应出现一个对话框通知用户文件已保存。当对话框关闭时,它应该清空文本字段和
我需要运行一个名为ArrayHolder的java程序,它将运行两个线程。 ArrayHolder 将有一个 Array。 ThreadSeven 会用 7 覆盖该 Array 的每个元素,并用 1
在我的程序中,应该读取学生姓名、ID 号和 GPA,将其分配给指定的学生,然后打印出来。一切都编译正常,但出现错误 Error: Could not find or load main class L
我是一名优秀的程序员,十分优秀!