个人中心
gpt4 book ai didi

amazon-web-services - AWS Kinesis Firehose无法将数据索引到AWS Elasticsearch中

转载 作者：行者123 更新时间：2023-12-02 23:42:58 24 4
gpt4 key购买 nike

我正在尝试将数据从Amazon Kinesis Data Firehose发送到Amazon Elasticsearch Service，但是它记录了一条错误消息503 Service Unavailable。但是，我可以到达Elasticsearch端点(https://vpc-XXX.<region>.es.amazonaws.com)并对其进行查询。我还经历了How can I prevent HTTP 503 Service Unavailable errors in Amazon Elasticsearch Service?，可以确认我的设置有足够的资源。
这是我在保存失败日志的S3备份存储桶中遇到的错误:

{
    "attemptsMade": 8,
    "arrivalTimestamp": 1599748282943,
    "errorCode": "ES.ServiceException",
    "errorMessage": "Error received from Elasticsearch cluster. <html><body><h1>503 Service Unavailable</h1>\nNo server is available to handle this request.\n</body></html>",
    "attemptEndingTimestamp": 1599748643460,
    "rawData": "eyJ0aWNrZXJfc3ltYm9sIjoiQUxZIiwic2VjdG9yIjoiRU5FUkdZIiwiY2hhbmdlIjotNi4zNSwicHJpY2UiOjg4LjgzfQ==",
    "subsequenceNumber": 0,
    "esDocumentId": "49610662085822146490768158474738345331794592496281976834.0",
    "esIndexName": "prometheus-2020-09",
    "esTypeName": ""
},
任何人都有如何解决此问题的想法，并将数据索引到Elasticsearch中吗？

最佳答案

原来，我的问题是选择了错误的安全组。

我正在使用与附加到Elasticsearch实例相同的安全组(我将其命名为elasticsearch-${domain_name})(该实例允许TCP从firehose_es安全组进/出端口443)。我应该选择firehose_es安全组。
根据注释的要求，这是firehose_es SG的Terraform配置。

resource "aws_security_group" "firehose_es" {
  name        = "firehose_es"
  description = "Firehose to send logs to Elasticsearch"
  vpc_id      = module.networking.aws_vpc_id
}

resource "aws_security_group_rule" "firehose_es_https_ingress" {
  type              = "ingress"
  from_port         = 443
  to_port           = 443
  protocol          = "tcp"
  security_group_id = aws_security_group.firehose_es.id
  cidr_blocks       = ["10.0.0.0/8"]
}

resource "aws_security_group_rule" "firehose_es_https_egress" {
  type                     = "egress"
  from_port                = 443
  to_port                  = 443
  protocol                 = "tcp"
  security_group_id        = aws_security_group.firehose_es.id
  source_security_group_id = aws_security_group.elasticsearch.id
}

在问这个问题(这也许是为什么有些人问这个问题的原因)之前，我已解决的另一件事是使用正确的角色并将正确的策略附加到该角色。这是我的角色(作为Terraform配置) 
// https://docs.aws.amazon.com/firehose/latest/dev/controlling-access.html
data "aws_iam_policy_document" "firehose_es_policy_specific" {
  statement {
    actions = [
      "s3:AbortMultipartUpload",
      "s3:GetBucketLocation",
      "s3:GetObject",
      "s3:ListBucket",
      "s3:ListBucketMultipartUploads",
      "s3:PutObject"
    ]
    resources = [
      aws_s3_bucket.firehose.arn,
      "${aws_s3_bucket.firehose.arn}/*"
    ]
  }

  statement {
    actions = [
      "es:DescribeElasticsearchDomain",
      "es:DescribeElasticsearchDomains",
      "es:DescribeElasticsearchDomainConfig",
      "es:ESHttpPost",
      "es:ESHttpPut"
    ]

    resources = [
      var.elasticsearch_domain_arn,
      "${var.elasticsearch_domain_arn}/*",
    ]
  }

  statement {
    actions = [
      "es:ESHttpGet"
    ]

    resources = [
      "${var.elasticsearch_domain_arn}/_all/_settings",
      "${var.elasticsearch_domain_arn}/_cluster/stats",
      "${var.elasticsearch_domain_arn}/${var.name_prefix}${var.name}_${var.app}*/_mapping/type-name",
      "${var.elasticsearch_domain_arn}/_nodes",
      "${var.elasticsearch_domain_arn}/_nodes/stats",
      "${var.elasticsearch_domain_arn}/_nodes/*/stats",
      "${var.elasticsearch_domain_arn}/_stats",
      "${var.elasticsearch_domain_arn}/${var.name_prefix}${var.name}_${var.app}*/_stats"
    ]
  }

  statement {
    actions = [
      "ec2:DescribeVpcs",
      "ec2:DescribeVpcAttribute",
      "ec2:DescribeSubnets",
      "ec2:DescribeSecurityGroups",
      "ec2:DescribeNetworkInterfaces",
      "ec2:CreateNetworkInterface",
      "ec2:CreateNetworkInterfacePermission",
      "ec2:DeleteNetworkInterface",
    ]

    resources = [
      "*"
    ]
  }
}

resource "aws_kinesis_firehose_delivery_stream" "ecs" {
  name        = "${var.name_prefix}${var.name}_${var.app}"
  destination = "elasticsearch"

  s3_configuration {
    role_arn           = aws_iam_role.firehose_es.arn
    bucket_arn         = aws_s3_bucket.firehose.arn
    buffer_interval    = 60
    compression_format = "GZIP"
  }

  elasticsearch_configuration {
    domain_arn = var.elasticsearch_domain_arn
    role_arn   = aws_iam_role.firehose_es.arn

    # If Firehose cannot deliver to Elasticsearch, logs are sent to S3
    s3_backup_mode = "FailedDocumentsOnly"

    buffering_interval = 60
    buffering_size     = 5

    index_name            = "${var.name_prefix}${var.name}_${var.app}"
    index_rotation_period = "OneMonth"

    vpc_config {
      subnet_ids         = var.elasticsearch_subnet_ids
      security_group_ids = [var.firehose_security_group_id]
      role_arn           = aws_iam_role.firehose_es.arn
    }
  }
}

再次阅读 Controlling Access with Amazon Kinesis Data Firehose文章后，我能够弄清自己的错误。

关于amazon-web-services - AWS Kinesis Firehose无法将数据索引到AWS Elasticsearch中，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/63839946/

24 4 0
文章推荐： elasticsearch - 按数组中的出现次数进行计数和排序
文章推荐： powershell - 如何选择某些哈希表值，但在Powershell脚本中内联更改其他值
文章推荐： powershell - Exchange:脚本和命令行中的Get-Mailbox输出不同
文章推荐： android - 混合不同格式的Android音频文件
行者123
个人简介

我是一名优秀的程序员,十分优秀！

滴滴打车优惠券免费领取
滴滴打车优惠券
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com