apache-flink - 为什么在申请任务管理器时 '-n' 或 '-yn' 之类的参数不起作用

Question

运行follow命令向yarn提交作业，但不期望分配任务管理器的数量。集群有足够的资源<220vcores, 720G mem>

/data/clusterserver/flink-1.7.2/bin/flink run -m yarn-cluster -yn 4 -ys 2  /data/clusterserver/flink-1.7.2/examples/batch/WordCount.jar --input hdfs:///tmp/wordcount.txt --output hdfs:///tmp/count.result

2019-04-10 15:04:57,397 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2019-04-10 15:04:57,397 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2019-04-10 15:04:57,405 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - The argument yn is deprecated in will be ignored.
2019-04-10 15:04:57,405 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - The argument yn is deprecated in will be ignored.
2019-04-10 15:04:57,554 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Cluster specification: ClusterSpecification{masterMemoryMB=2048, taskManagerMemoryMB=2048, numberTaskManagers=4, slotsPerTaskManager=2}
2019-04-10 15:04:58,017 WARN  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - The configuration directory ('/data/clusterserver/flink-1.7.2/conf') contains both LOG4J and Logback configuration files. Please delete or rename one of them.
2019-04-10 15:05:01,419 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Submitting application master application_1554348641734_0105
2019-04-10 15:05:01,446 INFO  org.apache.hadoop.yarn.client.api.impl.YarnClientImpl         - Submitted application application_1554348641734_0105
2019-04-10 15:05:01,447 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Waiting for the cluster to be allocated
2019-04-10 15:05:01,450 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deploying cluster, current state ACCEPTED
2019-04-10 15:05:05,747 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - YARN application has been deployed successfully.
Starting execution of program

从INFO日志可以看出，发送给yarn的参数是
{masterMemoryMB=2048, taskManagerMemoryMB=2048, numberTaskManagers=4, slotsPerTaskManager=2}但是，flink WebUI 中只有一个 taskManager。

使用
yarn-session.sh与 -n也遇到了同样的问题，即从命令行控制 taskManagers 的数量不起作用。

flink 版本 1.7.2

hadoop 版本 2.8.5(容量调度程序)

java: 1.8.0_91

Answer 1

截至 Flink 1.5 ，Flink 支持全资源弹性。这意味着 Flink 能够动态启动和停止 TaskManagers在 Yarn 上运行时。结果是您不再需要指定容器数量/TaskManagers对于 Yarn 上的 Flink 集群。这就是为什么 Flink 不再尊重 -n/-yn选项。

相反，Flink 会考虑作业的并行性来计算执行作业所需的槽数，然后启动尽可能多的 TaskManagers需要满足插槽要求。

您可以找到有关部署更改的更多信息 here和 here .