cassandra - HIVE/HDFS 用于大规模实时存储传感器数据？

Question

我正在评估具有以下要求的传感器数据收集系统，

1. 100 万个端点每分钟发送 100 字节的数据(作为时间序列)。

2. 基本上数百万次对存储的小写入。

   This data is write-once, so basically it never gets updated.
   

3. 访问要求
   一个。需要定期访问用户的完整数据(不那么频繁)
   湾。需要定期(更频繁地)访问用户的部分数据。例如，我需要在过去一小时/天/周/月收集的传感器数据用于分析/报告。

已开始将 Hive/HDFS 视为一种选择。有人可以评论 Hive 在这种用例中的适用性吗？我担心虽然分布式存储需求会起作用，但它似乎更适合数据仓库应用程序而不是实时数据收集/存储。

HBase/Cassandra 在这种情况下更有意义吗？

Answer 1

我认为 HBase 对您来说是一个不错的选择。事实上，HBase 中已经有一个开源/源代码实现，可以解决您可能想要使用的类似问题。看看openTSB这是解决类似问题的开源实现。以下是他们的简介的简短摘录:

OpenTSDB is a distributed, scalable Time Series Database (TSDB) written on top of HBase. OpenTSDB was written to address a common need: store, index and serve metrics collected from computer systems (network gear, operating systems, applications) at a large scale, and make this data easily accessible and graphable. Thanks to HBase's scalability, OpenTSDB allows you to collect many thousands of metrics from thousands of hosts and applications, at a high rate (every few seconds). OpenTSDB will never delete or downsample data and can easily store billions of data points. As a matter of fact, StumbleUpon uses it to keep track of hundred of thousands of time series and collects over 600 million data points per day in their main production datacenter.