dask-distributed - 如何将任务分配给 Dask.Distributed 中的特定工作人员

Question

我对使用 Dask Distributed 作为任务执行器很感兴趣。
在 Celery 中，可以将任务分配给特定的 worker 。如何使用 Dask Distributed？

Answer 1

有2个选项:

Specify workers by name或主机或 IP(但仅限正面声明):

dask-worker scheduler_address:8786 --name worker_1

然后选择之一:

client.map(func, sequence, workers='worker_1')
client.map(func, sequence, workers=['192.168.1.100', '192.168.1.100:8989', 'alice', 'alice:8989'])
client.submit(f, x, workers='127.0.0.1')
client.submit(f, x, workers='127.0.0.1:55852')
client.submit(f, x, workers=['192.168.1.101', '192.168.1.100'])
future = client.compute(z, workers={z: '127.0.0.1',
                                x: '192.168.0.1:9999'})
future = client.compute(z, workers={(x, y): ['192.168.1.100', '192.168.1.101:9999']})

使用 Resources概念。您可以为工作人员指定可用资源，例如:

dask-worker scheduler:8786 --resources "CAN_PROCESS_QUEUE_ALICE=2"

并指定所需的资源，如

client.submit(aggregate, processed, resources={'CAN_PROCESS_QUEUE_ALICE': 1})

或者

z = some_dask_object.map_parititons(func)
z.compute(resources={tuple(y.__dask_keys__()): {'CAN_PROCESS_QUEUE_ALICE': 1})