java - 是否有允许在集群应用程序中订购事件的 API？

Question

考虑到以下事实，是否存在现有的开源 Java API(可能作为某些更大产品的一部分)可以实现一种算法，从而在集群环境中实现事件的可重现排序:

1) There are N sources of events, each with a unique ID.
2) Each event produced has an ID/timestamp, which, together with
   its source ID, makes it uniquely identifiable.
3) The ids can be used to sort the events.
4) There are M application servers receiving those events.
   M is normally 3.
5) The events can arrive at any one or more of the application
   servers, in no specific order.
6) The events are processed in batches.
7) The servers have to agree for each batch on the list of events
   to process.
8) The event each have earliest and latest batch ID in which they
   must be processed.
9) They must not be processed earlier, and are "failed" if they
   cannot be processed before the deadline.
10) The batches are based on the real clock time. For example,
    one batch per second.
11) The events of a batch are processed when 2 of the 3 servers
    agree on the list of events to process for that batch (quorum).
12) The "third" server then has to wait until it possesses all the
    required events before it can process that batch too.
13) Once an event was processed or failed, the source has to be
    informed.
14) [EDIT] Events from one source must be processed (or failed) in
    the order of their ID/timestamp, but there is no causality
    between different sources.

不太正式地说，我有那些接收事件的服务器。它们以相同的初始状态开始，并且应该通过商定以何种顺序处理哪个事件来保持同步。对我来说幸运的是，事件不会尽快处理，而是“稍后”处理，这样我就有一些时间让服务器在截止日期之前达成一致。但我不确定这是否真的会对算法产生任何真正的影响。如果所有服务器都同意所有批处理，那么它们将始终保持同步，因此在查询时呈现一致的 View 。

虽然我对 Java API 最满意，但如果我可以从 Java 调用它，我会接受其他东西。而如果没有开源的API，但是有一个清晰的算法，我也会以此为答案，尝试自己实现。

Answer 1

看看这个问题和你的后续行动，可能“没有”一个 API 可以满足你的要求。今天你可以看看 Kafka(来自 LinkedIn)

• Apache Kafka

以及“日志”实体的一般概念，人们喜欢称之为“大数据”:

• The Log: What every software engineer should know about real-time data's unifying abstraction

实际上，对于你的问题，我将从有关“日志”的博客开始。用我的话来说，它的工作方式——而且 Kafka 并不是唯一进行日志处理的包——工作原理如下:

• 而不是基于队列的消息传递/发布-订阅
• Kafka 使用消息“日志”
• 订阅者(或端点)可以使用日志
• 日志保证“有序”；它处理千兆数据，速度很快
  • 仔细检查保证，通常需要权衡可靠性
• 您刚刚阅读了日志，我认为默认情况下读取具有破坏性。
• 如果有订阅者组，则每个人都可以在日志条目消失之前“阅读”。

日志的基本处理(计算)过程是一个 Map-Reduce-Filter 模型，因此您可以快速读取所有内容；只保留你想要的东西；处理它(减少)产生结果。

缺点似乎是你需要集群和其他东西才能让它真正发光。由于提到了不同的服务器或站点，我认为我们仍在正轨上。我发现启动并运行 Apache 下载很麻烦，因为它倾向于假设非 Windows 环境(ho hum)。

另一个“快速”选项是

• Apache Apollo

这需要您进行连接不同服务器的管道工作。由于要求包括...

servers that receive events. They start with the same initial state, and should keep in sync by agreeing on which event to process in which order. Luckily for me, the events are not to be processed ASAP, but "in a bit", so that I have some time to get the servers to agree before the deadline

我建议查看"Getting Started" example or tutorial with Kafka然后查看类似的 ZooKeeper 组织的消息/日志软件。祝你好运并享受!