hadoop - 如何在hadoop mapreduce中提交带有配置文件目录的作业？

Question

我有一个问题:我想提交一个带有配置文件目录的作业，并且该目录不在hadoop dfs中，hadoop是否可以执行此操作？我想提交目录，就像提交job的jar文件一样，以便hadoop可以将job和配置目录分发到所有datanode。我发现使用以下命令时-files选项似乎不起作用:bin / hadoop jar job.jar输入输出-files conf_dir。看来-files选项仅在hadoop流中有效

Answer 1

I want to submit a job with a directory of configure files and the directory is not in the hadoop dfs, does hadoop has some option to do this?

使用 --config选项指定配置文件的目录。

I want to submit the directory like submit the jar file of job so that hadoop can distribute the job and the configuration directory to all datanode.

在提交作业之前，配置文件应该存在于所有节点中，以启动守护程序，因此在提交作业时分发它们是没有意义的。要发送任何其他参数，请使用 -D option。如果sidedata很大，则使用 DistributedCache分发数据并在mapper和reducer中读取它。

I found that the -files option seems do not work when I use this command： bin/hadoop jar job.jar input output -files conf_dir. It seems that the -files option only works in hadoop streaming.

-files选项采用逗号分隔的 列表(任何文件(不是目录))，并将这些文件分发到所有节点。框架不读取那些文件，映射和reduce任务中的代码必须显式读取文件。

另外，根据 Hadoop - The Definitive Guide

Be aware that some properties have no effect when set in the client configuration. For example, if in your job submission you set mapred.tasktracker.map.tasks.maximum with the expectation that it would change the number of task slots for the tasktrackers run- ning your job, then you would be disappointed, since this property only is only honored if set in the tasktracker’s mapred-site.html file. In general, you can tell the component where a property should be set by its name, so the fact that mapred.task tracker.map.tasks.maximum starts with mapred.tasktracker gives you a clue that it can be set only for the tasktracker daemon. This is not a hard and fast rule, however, so in some cases you may need to resort to trial and error, or even reading the source.