gpt4 book ai didi

amazon-kinesis - Kinesis 分区键始终位于同一个分片中

转载 作者:行者123 更新时间:2023-12-04 21:56:32 26 4
gpt4 key购买 nike

我有一个带有 2 个分片的 kinesis 流,如下所示:

{
"StreamDescription": {
"StreamStatus": "ACTIVE",
"StreamName": "my-stream",
"Shards": [
{
"ShardId": "shardId-000000000001",
"HashKeyRange": {
"EndingHashKey": "17014118346046923173168730371587",
"StartingHashKey": "0"
},
{
"ShardId": "shardId-000000000002",
"HashKeyRange": {
"EndingHashKey": "340282366920938463463374607431768211455",
"StartingHashKey": "17014118346046923173168730371588"
},
]
}
}

发送方设置一个分区,通常是一个 UUID。它总是落在上面的 shard-002 中,这使得系统没有负载平衡,因此不可扩展。

作为旁注,kinesis 使用 md5sum 分配一条记录,然后将其发送到包含其范围内结果散列的分片。事实上,当我在我使用的 UUId 上测试它时,它们确实总是落在同一个分片中。
echo -n 80f6302fca1e48e590b09af84f3150d3 | md5sum
4527063413b015ade5c01d88595eec11

17014118346046923173168730371588 < 4527063413b015ade5c01d88595eec11 < 340282366920938463463374607431768211455

关于如何解决这个问题的任何想法?

最佳答案

首先看这个问答:How to decide total number of partition keys in AWS kinesis stream?

关于你的情况;您有 2 个分片,但它们的哈希键范围不相等。

分片 1 的分区键数包含:

17014118346046923173168730371587 - 0 = 17014118346046923173168730371587

分片 2 的分区键数包含:

340282366920938463463374607431768211455 - 17014118346046923173168730371587 = 34028234990682015174368201517436837

这两者有很大的不同;

17014118346046923173168730371587:17 x 10^30

340282349906820117416451434263037839868:34 x 10^37

如果分片 1 介于“0 - 170141183460469231731687303715884105727”之间,分片 2 介于“1701411834604692317316873037157864637373637373646373637363736373637373730373637373737373737373737360373737373605727”之间,那就太棒了。

您可能使用过台式机或其他精度较低的计算器。试试更好的计算器。看下面的例子;

package com.cagricelebi.kinesis.core.utils;

import java.math.BigInteger;

public class MyCalc {

public static void main(String[] args) {
try {

String num1 = "340282366920938463463374607431768211455";
String num2 = "-17014118346046923173168730371587";

String diff = bigCalc(num1, num2, "1", "1");
System.out.println("result1 : " + diff); // 340282349906820117416451434263037839868

String optimumHalf = bigCalc(num1, "0", "1", "2");
System.out.println("result2 : " + optimumHalf); // 170141183460469231731687303715884105727

} catch (Exception e) {
e.printStackTrace();
}
}

/**
* Basic calculator.
* First adds up first two elements, than multiplies the summation.
* The result is the division of the multilication to divisor.
*
* @param bigInt A
* @param bigInt2 B
* @param multiplicator C
* @param divisor D
* @return ((A+B)*C)/D
*/
private static String bigCalc(String bigInt, String bigInt2, String multiplicator, String divisor) {
BigInteger summation = new BigInteger(bigInt).add(new BigInteger(bigInt2));
BigInteger multiplication = summation.multiply(new BigInteger(multiplicator));
BigInteger division = multiplication.divide(new BigInteger(divisor));
return division.toString();
}

}

关于amazon-kinesis - Kinesis 分区键始终位于同一个分片中,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/33633464/

26 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com