architecture - 设计一个通用的作业调度程序

Question

关闭。这个问题需要多问 focused 。目前不接受答案。

---

想要改进此问题吗？更新问题，使其仅关注一个问题 editing this post .

已关闭 4 年前。

Improve this question

我正在尝试设计一个通用的作业调度程序，以扩展我的架构知识和在面试中思考系统设计问题的能力。到目前为止，我的想法如下。您能否指出我应该在哪些方面努力以全面地解决此类问题？

我已经在线阅读了很多资源，但需要一些具体的指导来继续前进。

Design a generic job scheduler for company X (which is one of the big technology firms of today).

Use cases

Create/Read/Update/Delete Jobs

Investigate jobs that have been run in the past (type of job, time spent, details)

Constraints

How many jobs will be run on the system per sec?

= # jobs/hour due to users + # jobs/hour due to machines

= 1m * 0.5 /day/24/3600 + 1m/50*20/24/3600

~= 12 jobs/sec

How much data will the system need to store?

Reasoning: I am only storing the job execution details, the actual work (script execution) is done > on other machines and some of the data collected is end time, success/failure status,etc. These are > all likely just text, maybe with graphing for illustration purpose. I will be storing the data of > > all jobs executed on in the system via the job scheduler (i.e. over the past 10 years)

= (Size of page where job details are set up + size of data collected about job ) * # of jobs * 365 > days * 10 years = 1 MB * 900 000 * 365 * 10

~= 3600 000 000 MB

= 3600 000 GB

=3600 TB =3.6 PB

Abstract Design

Based on the information above, we do not need to have too many machines to hold the data. I would break up the design into the following:

Application layer: serves the requests, shows UI details.

Data storage layer: Acts like a big hash table: Stores the mappings of key-value (key would be the jobs organized by dateTime they were run, while the values would show details of these jobs). This is to enable easy search of the historical and/or scheduled jobs.

The bottlenecks:

Traffic : 12 jobs/sec is not too challenging. If this spikes, we can use a load balancer to distribute the jobs to different servers for execution.

Data: At 3.6 TB, we need a hash table that can be easily queried for fast access to the jobs which have been executed in the application.

Scaling the abstract design

The nature of this job scheduler is that it each job possesses one of a few states: Pending, Failed,Success, Terminated. No business logic Returns little data.

For handling the traffic we can have an application server that handles 12 requests/sec and a backup in case this one fails. In future, we can use load balancer to reduce the number of requests going to each server (assuming >1 server are in production) Advantage of this would be to reduce number of requests/server, increase availability (in case one server fails, and handle spike-y traffic well).

For data storage, to store 3.6 TB of data we will need a few machines to hold it in database. We can use a noSQL db or SQL db. Given how the latter has more widespreaduse and community support which would help in troubleshooting matters and is used by large firms at the moment, I would choose mySQL db.

As the data grows, I would adopt the following strategies to handle it:

1) Create unique index on the hash

2) Scale mySQL db vertically by adding more memory

3) Partition the data by sharding

4) Use a master-slave replication strategy with master-master replication to ensure redundancy of data

Conclusion

Hence, this would be my design of the components of a job scheduler.

Answer 1

大多数大型作业调度程序都会考虑文档中未涵盖的方面。

一些关键问题是:(排名不分先后)

• 取消 - 您经常想要终止一项长时间运行的作业，或阻止一项作业运行。
• 优先级 - 您通常希望高优先级作业优先于低优先级作业运行。但是，以一种低优先级作业不会在生成大量作业的系统中永远等待的方式实现这一点是“不平凡的”
• 资源 - 某些作业可能只能在具有某些资源的系统上进行调度。例如。有些需要大量内存、快速本地磁盘或快速网络访问。有效地分配这些是很棘手的。
• 依赖关系 - 某些作业可能只有在其他作业完成后才能运行，因此无法在给定时间之前进行安排。
• 截止日期 - 有些工作需要在给定时间内完成。 (或者至少在给定时间开始。)
• 权限 - 某些用户可能只能向特定资源组、特定属性或特定数量的作业等提交作业。
• 配额 - 某些系统为用户提供指定的系统时间，运行作业会从中减去。这可能会对您示例中的数字产生重大影响。
• 暂停 - 某些系统允许对作业进行检查点和暂停，然后再恢复。

我确定还有更多 - 尝试查看 slurm 上的文档或grid-engine获取更多想法。

需要考虑的其他事项:

1. 您的抽象设计可能需要更多细节来支持这些高级概念。
2. 您不需要频繁访问 3.6TB 数据中的大部分 - 将其分为最新数据和旧数据，如果您允许较慢地访问旧数据(并命中磁盘)。
3. 您可能有不同类别的用户，至少是“管理员”和“用户”。这对于应用程序的结构意味着什么。
4. 真正的作业调度应用程序每秒能够处理更多请求 - slurm 建议持续 33 个/秒，且突发量更高，但我的理解是，它可能会明显高于此值。
5. 通常需要通过网页以外的界面提交作业或查询作业状态 - 这对您的应用程序结构意味着什么。 (我要么为核心引擎使用更简单的提交 API，并让 Web UI 作为其愚蠢的翻译器，并且所有其他方法都使用相同的 API，要么使用带有简单 Web 前端的 REST API))
6. 如何检测服务器故障？两台服务器足以可靠地确定这一点吗？为此，通常使用基于仲裁的措施，或者对第三台服务器进行连接测试。如果发生故障的服务器重新上线，您该怎么办？