python - Luigi Python 中的资源是什么？

Question

在 Web 界面和 https://github.com/spotify/luigi/blob/master/luigi/task.py 中我可以看到一个任务可以有“资源”。 Task 类中还有一个名为 process_resources() 的占位符函数，它只返回作为资源的空字典。

这个神话般的资源是什么东西？

Answer 1

我还没有测试过这个，但它看起来像是一个任意值，调度程序可以使用它来确定是否根据配置中的值限制作业。来自docs :

This section can contain arbitrary keys. Each of these specifies the amount of a global resource that the scheduler can allow workers to use. The scheduler will prevent running jobs with resources specified from exceeding the counts in this section. Unspecified resources are assumed to have limit 1. Example resources section for a configuration with 2 hive resources and 1 mysql resource:

[resources]
hive: 2
mysql: 1

Note that it was not necessary to specify the 1 for mysql here, but it is good practice to do so when you have a fixed set of resources.