gpt4 book ai didi

hadoop - Giraph 作业始终以本地模式运行

转载 作者:可可西里 更新时间:2023-11-01 16:42:41 25 4
gpt4 key购买 nike

我在 Hadoop 2.6.0 上运行 Giraph 1.1.0。mapredsite.xml 看起来像这样

<configuration>

<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
<description>The runtime framework for executing MapReduce jobs. Can be one of
local, classic or yarn.</description>
</property>

<property>
<name>mapreduce.map.memory.mb</name>
<value>4096</value>
<name>mapreduce.reduce.memory.mb</name>
<value>8192</value>
</property>
<property>
<name>mapreduce.map.java.opts</name>
<value>-Xmx3072m</value>
<name>mapreduce.reduce.java.opts</name>
<value>-Xmx6144m</value>
</property>
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<value>4</value>
</property>
<property>
<name>mapred.map.tasks</name>
<value>4</value>
</property>
</configuration>

giraph-site.xml 看起来像这样

<configuration>
<property>
<name>giraph.SplitMasterWorker</name>
<value>true</value>
</property>
<property>
<name>giraph.logLevel</name>
<value>error</value>
</property>
</configuration>

我不想在本地模式下运行作业。我还将环境变量 MAPRED_HOME 设置为 HADOOP_HOME。这是运行程序的命令。

hadoop jar myjar.jar hu.elte.inf.mbalassi.msc.giraph.betweenness.BetweennessComputation /user/$USER/inputbc/inputgraph.txt /user/$USER/outputBC 1.0 1

当我运行这段代码来计算图中顶点的介数中心性时,出现以下异常

Exception in thread "main" java.lang.IllegalArgumentException: checkLocalJobRunnerConfiguration: When using LocalJobRunner, you cannot run in split master / worker mode since there is only 1 task at a time!
at org.apache.giraph.job.GiraphJob.checkLocalJobRunnerConfiguration(GiraphJob.java:168)
at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:236)
at hu.elte.inf.mbalassi.msc.giraph.betweenness.BetweennessComputation.runMain(BetweennessComputation.java:214)
at hu.elte.inf.mbalassi.msc.giraph.betweenness.BetweennessComputation.main(BetweennessComputation.java:218)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

我应该怎么做才能确保作业不在本地模式下运行?

最佳答案

几天前我遇到了这个问题。幸运的是我通过这样做解决了它。

修改配置文件mapred-site.xml,确保'mapreduce.framework.name'属性的值为'yarn',如果有则添加'mapreduce.jobtracker.address'属性值为'yarn'不是。

mapred-site.xml 看起来像这样:

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobtracker.address</name>
<value>yarn</value>
</property>
</configuration>

修改mapred-site.xml后重启hadoop,然后运行你的程序,设置'-w'后的值大于1,'giraph.SplitMasterWorker'的值为'true'。可能会奏效。

至于问题的原因,我只是引用某人的说法:这些属性是为单节点执行而设计的,必须是在节点集群中执行操作时更改。在这种情况下,jobtracker 必须指向其中一台将执行任务的机器NodeManager 守护进程(Hadoop 从站)。至于框架,应该是更改为“ yarn ”。

关于hadoop - Giraph 作业始终以本地模式运行,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/39287757/

25 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com