Graphite Graph - 我们更新图表的速度有多快？

Question

我们正在尝试将 Graphite 用于(近)实时图形网络系统。然而，我们似乎无法以超过 1 秒的更新率加速 Graphite 。最终我们希望有 100 毫秒的更新

通过阅读常见问题解答，它听起来像 Graphite 很快 - 但这要么非常具有误导性，要么我不了解如何加速 Graphite

耳语的计时信息似乎使用 UNIX 时间戳

How scalable is Graphite?

From a CPU perspective, Graphite scales horizontally on both the frontend and the backend, meaning you can simply add more machines to the mix to get more throughput. It is also fault tolerant in the sense that losing a backend machine will cause a minimal amount of data loss (whatever that machine had cached in memory) and will not disrupt the system if you have sufficient capacity remaining to handle the load.

From an I/O perspective, under load Graphite performs lots of tiny I/O operations on lots of different files very rapidly. This is because each distinct metric sent to Graphite is stored in its own database file, similar to how many tools (drraw, Cacti, Centreon, etc) built on top of RRD work. In fact, Graphite originally did use RRD for storage until fundamental limitations arose that required a new storage engine.

High volume (a few thousand distinct metrics updating minutely) pretty much requires a good RAID array. Graphite's backend caches incoming data if the disks cannot keep up with the large number of small write operations that occur (each data point is only a few bytes, but most disks cannot do more than a few thousand I/O operations per second, even if they are tiny). When this occurs, Graphite's database engine, whisper, allows carbon to write multiple data points at once, thus increasing overall throughput only at the cost of keeping excess data cached in memory until it can be written.

How real-time are the graphs?

Very. Even under heavy load, where the number of metrics coming in each time interval is much greater than the rate at which your storage system can perform I/O operations and lots of data points are being cached in the storage pipeline (see previous question for explanation), Graphite still draws real-time graphs. The trick is that when the Graphite webapp receives a request to draw a graph, it simultaneously retrieves data off the disk as well as from the pre-storage cache (which may be distributed if you have multiple backend servers) and combines the two sources of data to create a real-time graph.

此外，它们仅显示秒数，此处不显示小数点:
http://graphite.readthedocs.org/en/latest/config-carbon.html
和 from and until must be a time specification conforming to the AT-STYLE time specification described这里: http://oss.oetiker.ch/rrdtool/doc/rrdfetch.en.html .
http://graphite.wikidot.com/url-api-reference

那是什么？ Graphite 快吗？或者它只是快速处理大型数据集 - 我们正在寻找一个简单易用的数据包数据的网络接收器来直观地显示 - Graphite 似乎是一个很好的解决方案，但现在我们已经配置并运行了它，我猜我们只是浪费了一个很多时间

谢谢!

Answer 1

Graphite 将在您的 storage-schemas.conf 中按照最精细的定义精度存储最多一个数据点(接收到的额外数据点将被删除)。 最精确的可能是 1 秒。 例如保留 = 1s :6h,1min:7d,10min:5y

为了实现您的目标，您需要在 Graphite 前面放置一个聚合器。聚合器将接收所有指标并聚合数据，刷新到 Graphite 存储以匹配存储模式。聚合器将对指标执行计算(平均值、总和、平均值等)并将其发送出去。例如在最后一秒内，您平均需要 14 毫秒来处理请求，或者在过去 10 秒内，请求总数为 4234。

因此，虽然您不能以比 1 秒更精细的粒度进行报告，但您可以使用聚合器来捕获 1 秒时间间隔内发生的事情的总和和平均值并报告。

两个常见的选择是 StatsD和 Graphite provided in carbon aggregator.

** StatsD，IMO 是要走的路。它是一个网络守护进程，您单独运行并通过 UDP 发送给它。也就是说，您可以使用 carbon-aggregator.py 执行相同的操作(例如 UDP)。