configuration - 更改 Hadoop 从节点上的默认配置？

Question

目前，我正在尝试通过命令行参数传递一些值，然后使用实现工具的 GenericOptionsParser 解析它。

我从主节点运行这样的东西:

bin/hadoop jar MYJAR.jar MYJOB -D mapred.reduce.tasks=13

但这只适用于法师!!有什么方法可以使它也适用于奴隶吗？

我使用 Hadoop 0.20.203。

感谢任何帮助。

Answer 1

但这只适用于法师!!有什么方法可以使它也适用于奴隶吗？

根据"Hadoop : The Definitive Guide" .在客户端设置一些属性是没有用的。您需要在配置文件中进行相同的设置。请注意，您还可以在配置文件中创建新属性，并使用配置对象在代码中读取它们。

Be aware that some properties have no effect when set in the client configuration. Forexample, if in your job submission you set mapred.tasktracker.map.tasks.maximum withthe expectation that it would change the number of task slots for the tasktrackers running your job, then you would be disappointed, since this property only is only honoredif set in the tasktracker’s mapred-site.html file. In general, you can tell the componentwhere a property should be set by its name, so the fact that mapred.task.tracker.map.tasks.maximum starts with mapred.tasktracker gives you a clue that it can be set only for the tasktracker daemon. This is not a hard and fast rule, however, so in some cases you may need to resort to trial and error, or even reading the source.

您还可以使用 HADOOP_*_OPTS 配置 Hadoop 变量的环境在 conf/hadoop-env.sh 文件中。

同样，根据 "Hadoop : The Definitive Guide" .

Do not confuse setting Hadoop properties using the -D property=value option to GenericOptionsParser (and ToolRunner) with setting JVM system properties using the -Dproperty=value option to the java command. The syntax for JVM system properties does not allow any whitespace between the D and the property name, whereas GenericOptionsParser requires them to be separated by whitespace.

JVM system properties are retrieved from the java.lang.System class, whereas Hadoop properties are accessible only from a Configuration object.