resources - Apache 弗林克 : Limit number of CPUs in a TaskManager

Question

首先我在独立模式下运行!

我一直试图找到任何配置，但我还没有找到任何相关信息。

在 Spark 中有一些配置可以让你限制在每个从属中使用的 CPU 数量:

• SPARK_WORKER_CORES(工作人员配置)
• spark.executor.cores(集群配置)

但在 Flink 中，您可以设置要使用的最大内存和任务槽的数量(这只是划分内存)，如前所述 in the official documentation :

• taskmanager.numberOfTaskSlots: The number of parallel operator or user function instances that a single TaskManager can run (DEFAULT: 1). If this value is larger than 1, a single TaskManager takes multiple instances of a function or operator. That way, the TaskManager can utilize multiple CPU cores, but at the same time, the available memory is divided between the different operator or function instances. This value is typically proportional to the number of physical CPU cores that the TaskManager’s machine has (e.g., equal to the number of cores, or half the number of cores).

和here more focused关于我的问题:

Each task slot represents a fixed subset of resources of the TaskManager. A TaskManager with three slots, for example, will dedicate 1/3 of its managed memory to each slot. Slotting the resources means that a subtask will not compete with subtasks from other jobs for managed memory, but instead has a certain amount of reserved managed memory. Note that no CPU isolation happens here; currently slots only separate the managed memory of tasks.

谢谢!!

Answer 1

我也在找同样的问题。据我了解，没有任何配置可以设置每个插槽的 CPU 数量。设置插槽数将在插槽之间分配内存，从而减少每个插槽的内存。我最好的猜测是将插槽数设置为 1，并为在容器(可能是 docker)中运行的任务管理器进程提供可用的 CPU。您可以通过增加任务管理器的数量来实现相同的并行性。