hadoop - 客户端计算机上的mapred-site.xml必须与hadoop集群中的mapred-site.xml相同吗？

Question

与hadoop api configuration on the client machine有关。

如果我们尝试将客户端计算机上的mapred-site.xml保持尽可能少=仅指定mapred.job.tracker，则hadoop.jar内部的mapred-default会接管并放置不需要的属性，例如mapred.tasktracker.map.tasks.maximum = 2。然后，这些值与任务一起提交，并覆盖集群配置中的那些值:(

这里正确的方法是什么？您是否将文件从群集复制到客户端计算机？

Answer 1

the mapred-default from inside the hadoop.jar takes over, and puts unwanted properties, e.g. mapred.tasktracker.map.tasks.maximum=2. Then these values are submitted with ttask, and override those in the cluster config :(

我假设您引用的是job.xml文件中设置的属性。由于在启动时TaskTracker守护程序会读取 mapred.tasktracker.map.tasks.maximum属性，因此在客户端设置诸如 mapred.tasktracker.map.tasks.maximum之类的某些属性应该没有效果。尽管在job.xml中指定了 mapred.tasktracker.map.tasks.maximum，但它不是特定于作业的。

您如何验证属性已被覆盖？转到JobTracker页面(http:// jotracker:50030 / jobtracker.jsp)并验证是否已为TaskTracker覆盖了特定属性。

what's the right approach here? do you replicate the files from your cluster into your client machine?

只是为了避免混淆，我将在客户端和节点上有单独的文件，并在其中具有最低限度的配置属性，而让其他属性采用默认值。

根据 Hadoop : The Definitive Guide

Be aware that some properties have no effect when set in the client configuration. For example, if in your job submission you set mapred.tasktracker.map.tasks.maximum with the expectation that it would change the number of task slots for the tasktrackers running your job, then you would be disappointed, since this property only is only honored if set in the tasktracker’s mapred-site.html file. In general, you can tell the component where a property should be set by its name, so the fact that mapred.task tracker.map.tasks.maximum starts with mapred.tasktracker gives you a clue that it can be set only for the tasktracker daemon. This is not a hard and fast rule, however, so in some cases you may need to resort to trial and error, or even reading the source.