- html - 出于某种原因,IE8 对我的 Sass 文件中继承的 html5 CSS 不友好?
- JMeter 在响应断言中使用 span 标签的问题
- html - 在 :hover and :active? 上具有不同效果的 CSS 动画
- html - 相对于居中的 html 内容固定的 CSS 重复背景?
我想将流分析记录分批聚合 n 条记录,每条记录满足以下条件:
我还没有找到使用窗口函数来实现此目的的方法(因为它们是基于时间的,而不是基于计数的)。此外,像 CollectTOP
这样的函数也不起作用,因为它是针对每个记录而不是每个批处理进行评估的。
最佳答案
正如评论中提到的,Azure 流分析没有事件计数的概念。
为了能够对每批特定大小的事件进行重新分组,第一步是对它们进行排名 ( CollectTop )。与任何流处理器一样,ASA 需要 time window 来定义该等级。
这不符合要求,也在评论中进行了澄清。抱歉,ASA 似乎不是这里的解决方案。
如果您可以容忍时间窗口 - 意味着容忍延迟(记录在窗口末尾输出)并容忍不完整的批处理(大多数时间窗口不会有 n 个事件的倍数) - 那么这可以通过多步查询部分实现。
在输入中使用此 input file 以及此查询:
WITH Ranking AS (
SELECT
System.Timestamp() as ts,
COLLECTTOP(1000000) OVER (ORDER BY EntryTime, TollId) as c
FROM [entry] AS e TIMESTAMP BY EntryTime
GROUP BY TumblingWindow(minute,5)
),
Unfolding AS (
SELECT
System.Timestamp() AS windowEnd,
FLOOR((d.ArrayValue.rank-1) /3.0) as batch3,
d.ArrayValue.rank,
d.ArrayValue.value.*
FROM Ranking AS r
CROSS APPLY GetElements(r.c) AS d
),
Batching AS (
SELECT
System.Timestamp() as ts,
Collect() as batch
FROM Unfolding
GROUP BY System.Timestamp, batch3
)
SELECT * FROM Batching
我们将得到:
{"ts":"2014-09-10T12:05:00.0000000Z","batch":[{"windowEnd":"2014-09-10T12:05:00.0000000Z","batch3":0.0,"rank":1,"EntryTime":"2014-09-10T12:01:00.0000000Z","TollId":1,"LicensePlate":"JNB 7001","State":"NY","Make":"Honda","Model":"CRV","VehicleType":1,"VehicleWeight":0,"Toll":7,"Tag":null},{"windowEnd":"2014-09-10T12:05:00.0000000Z","batch3":0.0,"rank":2,"EntryTime":"2014-09-10T12:02:00.0000000Z","TollId":1,"LicensePlate":"YXZ 1001","State":"NY","Make":"Toyota","Model":"Camry","VehicleType":1,"VehicleWeight":0,"Toll":4,"Tag":123456789},{"windowEnd":"2014-09-10T12:05:00.0000000Z","batch3":0.0,"rank":3,"EntryTime":"2014-09-10T12:02:00.0000000Z","TollId":3,"LicensePlate":"ABC 1004","State":"CT","Make":"Ford","Model":"Taurus","VehicleType":1,"VehicleWeight":0,"Toll":5,"Tag":456789123}]}
{"ts":"2014-09-10T12:05:00.0000000Z","batch":[{"windowEnd":"2014-09-10T12:05:00.0000000Z","batch3":1.0,"rank":4,"EntryTime":"2014-09-10T12:03:00.0000000Z","TollId":1,"LicensePlate":"BNJ 1007","State":"NY","Make":"Honda","Model":"CRV","VehicleType":1,"VehicleWeight":0,"Toll":5,"Tag":789123456},{"windowEnd":"2014-09-10T12:05:00.0000000Z","batch3":1.0,"rank":5,"EntryTime":"2014-09-10T12:03:00.0000000Z","TollId":2,"LicensePlate":"XYZ 1003","State":"CT","Make":"Toyota","Model":"Corolla","VehicleType":1,"VehicleWeight":0,"Toll":4,"Tag":null},{"windowEnd":"2014-09-10T12:05:00.0000000Z","batch3":1.0,"rank":6,"EntryTime":"2014-09-10T12:05:00.0000000Z","TollId":2,"LicensePlate":"CDE 1007","State":"NJ","Make":"Toyota","Model":"4x4","VehicleType":1,"VehicleWeight":0,"Toll":6,"Tag":321987654}]}
{"ts":"2014-09-10T12:10:00.0000000Z","batch":[{"windowEnd":"2014-09-10T12:10:00.0000000Z","batch3":0.0,"rank":1,"EntryTime":"2014-09-10T12:06:00.0000000Z","TollId":2,"LicensePlate":"BAC 1005","State":"NY","Make":"Toyota","Model":"Camry","VehicleType":1,"VehicleWeight":0,"Toll":5.5,"Tag":567891234},{"windowEnd":"2014-09-10T12:10:00.0000000Z","batch3":0.0,"rank":2,"EntryTime":"2014-09-10T12:07:00.0000000Z","TollId":1,"LicensePlate":"ZYX 1002","State":"NY","Make":"Honda","Model":"Accord","VehicleType":1,"VehicleWeight":0,"Toll":6,"Tag":234567891},{"windowEnd":"2014-09-10T12:10:00.0000000Z","batch3":0.0,"rank":3,"EntryTime":"2014-09-10T12:07:00.0000000Z","TollId":2,"LicensePlate":"ZXY 1001","State":"PA","Make":"Toyota","Model":"Camry","VehicleType":1,"VehicleWeight":0,"Toll":4,"Tag":987654321}]}
{"ts":"2014-09-10T12:10:00.0000000Z","batch":[{"windowEnd":"2014-09-10T12:10:00.0000000Z","batch3":1.0,"rank":4,"EntryTime":"2014-09-10T12:08:00.0000000Z","TollId":3,"LicensePlate":"CBA 1008","State":"PA","Make":"Ford","Model":"Mustang","VehicleType":1,"VehicleWeight":0,"Toll":4.5,"Tag":891234567},{"windowEnd":"2014-09-10T12:10:00.0000000Z","batch3":1.0,"rank":5,"EntryTime":"2014-09-10T12:09:00.0000000Z","TollId":2,"LicensePlate":"CDB 1003","State":"PA","Make":"Volvo","Model":"C30","VehicleType":1,"VehicleWeight":0,"Toll":5,"Tag":765432198},{"windowEnd":"2014-09-10T12:10:00.0000000Z","batch3":1.0,"rank":5,"EntryTime":"2014-09-10T12:09:00.0000000Z","TollId":2,"LicensePlate":"DCB 1004","State":"NY","Make":"Volvo","Model":"S80","VehicleType":1,"VehicleWeight":0,"Toll":5.5,"Tag":654321987}]}
{"ts":"2014-09-10T12:10:00.0000000Z","batch":[{"windowEnd":"2014-09-10T12:10:00.0000000Z","batch3":2.0,"rank":7,"EntryTime":"2014-09-10T12:09:00.0000000Z","TollId":3,"LicensePlate":"YZX 1009","State":"NY","Make":"Volvo","Model":"V70","VehicleType":1,"VehicleWeight":0,"Toll":4.5,"Tag":912345678},{"windowEnd":"2014-09-10T12:10:00.0000000Z","batch3":2.0,"rank":8,"EntryTime":"2014-09-10T12:10:00.0000000Z","TollId":1,"LicensePlate":"CBD 1005","State":"NY","Make":"Toyota","Model":"Camry","VehicleType":1,"VehicleWeight":0,"Toll":4,"Tag":543219876},{"windowEnd":"2014-09-10T12:10:00.0000000Z","batch3":2.0,"rank":9,"EntryTime":"2014-09-10T12:10:00.0000000Z","TollId":3,"LicensePlate":"BCD 1002","State":"NY","Make":"Toyota","Model":"Rav4","VehicleType":1,"VehicleWeight":0,"Toll":5.5,"Tag":876543219}]}
{"ts":"2014-09-10T12:15:00.0000000Z","batch":[{"windowEnd":"2014-09-10T12:15:00.0000000Z","batch3":0.0,"rank":1,"EntryTime":"2014-09-10T12:11:00.0000000Z","TollId":1,"LicensePlate":"NJB 1006","State":"CT","Make":"Ford","Model":"Focus","VehicleType":1,"VehicleWeight":0,"Toll":4.5,"Tag":678912345},{"windowEnd":"2014-09-10T12:15:00.0000000Z","batch3":0.0,"rank":2,"EntryTime":"2014-09-10T12:12:00.0000000Z","TollId":3,"LicensePlate":"PAC 1209","State":"NJ","Make":"Chevy","Model":"Malibu","VehicleType":1,"VehicleWeight":0,"Toll":6,"Tag":219876543},{"windowEnd":"2014-09-10T12:15:00.0000000Z","batch3":0.0,"rank":3,"EntryTime":"2014-09-10T12:15:00.0000000Z","TollId":2,"LicensePlate":"BAC 1005","State":"PA","Make":"Peterbilt","Model":"389","VehicleType":2,"VehicleWeight":2.675,"Toll":15.5,"Tag":567891234}]}
{"ts":"2014-09-10T12:15:00.0000000Z","batch":[{"windowEnd":"2014-09-10T12:15:00.0000000Z","batch3":1.0,"rank":4,"EntryTime":"2014-09-10T12:15:00.0000000Z","TollId":3,"LicensePlate":"EDC 3109","State":"NJ","Make":"Ford","Model":"Focus","VehicleType":1,"VehicleWeight":0,"Toll":4,"Tag":198765432}]}
{"ts":"2014-09-10T12:20:00.0000000Z","batch":[{"windowEnd":"2014-09-10T12:20:00.0000000Z","batch3":0.0,"rank":1,"EntryTime":"2014-09-10T12:18:00.0000000Z","TollId":2,"LicensePlate":"DEC 1008","State":"NY","Make":"Toyota","Model":"Corolla","VehicleType":1,"VehicleWeight":0,"Toll":4,"Tag":null},{"windowEnd":"2014-09-10T12:20:00.0000000Z","batch3":0.0,"rank":2,"EntryTime":"2014-09-10T12:20:00.0000000Z","TollId":1,"LicensePlate":"DBC 1006","State":"NY","Make":"Honda","Model":"Civic","VehicleType":1,"VehicleWeight":0,"Toll":5,"Tag":432198765},{"windowEnd":"2014-09-10T12:20:00.0000000Z","batch3":0.0,"rank":3,"EntryTime":"2014-09-10T12:20:00.0000000Z","TollId":2,"LicensePlate":"APC 2019","State":"NJ","Make":"Honda","Model":"Civic","VehicleType":1,"VehicleWeight":0,"Toll":4,"Tag":345678912}]}
{"ts":"2014-09-10T12:25:00.0000000Z","batch":[{"windowEnd":"2014-09-10T12:25:00.0000000Z","batch3":0.0,"rank":1,"EntryTime":"2014-09-10T12:22:00.0000000Z","TollId":1,"LicensePlate":"EDC 1019","State":"NJ","Make":"Honda","Model":"Accord","VehicleType":1,"VehicleWeight":0,"Toll":4,"Tag":null}]}
这保证了每个批处理不会有超过 N=3 的事件。但我们在 5 分钟窗口内强制执行这一保证。当一个时间窗口内的记录较少时,我们将得到一个少于N条记录的批处理。
关于azure - 聚合多批记录,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/70393302/
我有一个 Cassandra 集群,里面有 4 个表和数据。 我想使用聚合函数(sum,max ...)发出请求,但我在这里读到这是不可能的: http://www.datastax.com/docu
我有以下两张表 Table: items ID | TITLE 249 | One 250 | Two 251 | Three 我投票给这些: Table: votes VID | IID | u
这个问题在这里已经有了答案: Update MongoDB field using value of another field (12 个答案) 关闭 3 年前。 我想根据另一个“源”集合的文档中
我的收藏包含以下文件。我想使用聚合来计算里面有多少客户,但我遇到了一些问题。我可以获得总行数,但不能获得总(唯一)客户。 [{ _id: "n001", channel: "Kalip
我有下表 Id Letter 1001 A 1001 H 1001 H 1001 H 1001 B 1001 H 1001 H 1001
得到一列的表 ABC。 “创建”的日期列。所以样本值就像; created 2009-06-18 13:56:00 2009-06-18 12:56:00 2009-06-17 14:02:0
我有一个带有数组字段的集合: {[ name:String buyPrice:Int sellPrice:Int ]} 我试图找到最低和最高买入/卖出价格。在某些条目中,买入或卖出价格为零
我有以下问题: 在我的 mongo db 中,我有以下结构: { "instanceId": "12", "eventId": "0-1b", "activityType":
下面给出的是我要在其上触发聚合查询的 Elasticsearch 文档。 { "id": 1, "attributes": [ { "fieldId": 1,
我正在使用 Django 的 aggregate query expression总计一些值。最终值是一个除法表达式,有时可能以零作为分母。如果是这种情况,我需要一种方法来逃避,以便它只返回 0。 我
我正在学习核心数据,特别是聚合。 当前我想要做的事情:计算表中在某些条件上具有逆关系的多对关系的记录数。 目前我正在这样做: NSExpression *ex = [NSExpression expr
我需要有关 Delphi 中的 ClientDatasets 的一些帮助。 我想要实现的是一个显示客户的网格,其中一列显示每个客户的订单数量。我将 ClientDataset 放在表单上并从 Delp
我的集合有 10M 个文档,并且有一个名为 movieId 的字段;该文档具有以下结构: { "_id" : ObjectId("589bed43e3d78e89bfd9b779"), "us
这个问题已经有答案了: What is the difference between association, aggregation and composition? (21 个回答) 已关闭 9
我在 elasticsearch 中有一些类似于这些示例的文档: { "id": ">", "list": [ "a", "b", "c" ] } { "id"
我正在做一些聚合。但是结果完全不是我所期望的,似乎它们没有聚合索引中与我的查询匹配的所有文档,在这种情况下 - 它有什么好处? 例如,首先我做这个查询: {"index":"datalayer","t
假设我在 ES 中有这些数据。 | KEY | value | |:-----------|------------:| | A |
可能在我的文档中,我有一个被分析的文本字段。我只是在ElasticSearch AggregationAPI中迷路了。我需要2种不同情况的支持: 情况A)结果是带有计数标记(条款)的篮子下降。 情况B
我正在为网上商店构建多面过滤功能,如下所示: Filter on Brand: [ ] LG (10) [ ] Apple (5) [ ] HTC (3) Filter on OS: [ ] Andr
我有一个父/子关系并且正在搜索 child 。 是否可以在父属性上创建聚合? 例如parent 是 POST,children 是 COMMENT。如果父项具有“类别”属性,是否可以搜索 COMMEN
我是一名优秀的程序员,十分优秀!