amazon-web-services - 亚马逊 Kinesis : Caught exception while sync'ing Kinesis shards and leases

Question

我正在尝试制作 Snowplow在 AWS 上工作。当我尝试在实例上运行流丰富服务时，我收到此异常:

[main] INFO com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Syncing Kinesis shard info
[main] ERROR com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardSyncTask - Caught exception while sync'ing Kinesis shards and leases
[cw-metrics-publisher] WARN com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable - Could not publish 4 datums to CloudWatch

我不认为错误是由于 Cloud Watch 造成的:

Caught exception while sync'ing Kinesis shards and leases

Answer 1

正如上面的评论中所提到的，当您缺乏 Kinesis Client Library (KCL) 所需的 AWS 资源的权限时，这个错误就会出现。 .这可以是 DynamoDB、CloudWatch 或 Kinesis。对于 Snowplow 的 Stream Enrich 组件，您需要以下权限:

输入kinesis流的读取权限(收集器好)

输出 kinesis 流的写入权限(丰富好的和丰富的坏)

列出对 kinesis 流的权限

对 DynamoDB 状态表的读/写/创建权限(表名是流中的“appName”值丰富 application.conf)

PutMetricData 到 Cloudwatch

满足这些需求的 IAM 策略的模板版本如下:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kinesis:DescribeStream",
        "kinesis:GetShardIterator",
        "kinesis:GetRecords",
        "kinesis:ListShards"
      ],
      "Resource": [
        "${collector_stream_out_good}"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
          "kinesis:ListStreams"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "kinesis:DescribeStream",
        "kinesis:PutRecord",
        "kinesis:PutRecords"
      ],
      "Resource": [
        "${enricher_stream_out_good}",
        "${enricher_stream_out_bad}"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:CreateTable",
        "dynamodb:DescribeTable",
        "dynamodb:Scan",
        "dynamodb:GetItem",
        "dynamodb:PutItem",
        "dynamodb:UpdateItem",
        "dynamodb:DeleteItem"
      ],
      "Resource": [
        "${enricher_state_table}"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "cloudwatch:PutMetricData"
      ],
      "Resource": "*"
    }
  ]
}

我写了一个 blog post这涵盖了 Stream Enrich 和其他 Snowplow 组件所需的 IAM 权限，因为有关确切所需权限的文档在 Snowplow 文档中很少/不存在。

希望有帮助!