docker - 使用带有端口映射的 docker-compose 进行本地 Kafka 测试

Question

我正在尝试使用 docker-compose 启动并运行黑盒服务测试的基础设施。我想在 Jenkins 上为多个服务执行此操作，因此有必要绑定(bind)到特定于构建的端口(例如 50012 而不是 9092)，因为构建应该能够并行运行。

问题是，生成消息失败。

这是我的docker-compose.yml:

---
version: "3.4"

services:

  zookeeper:
    image: "confluentinc/cp-zookeeper:latest"
    ports:
      - "2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    extra_hosts:
      - "moby:127.0.0.1"

  kafka:
    image: "confluentinc/cp-kafka:latest"
    environment:
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_PROTOCOL_NAME: INSIDE
      KAFKA_ADVERTISED_PROTOCOL_NAME: OUTSIDE
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
      KAFKA_LISTENERS: "OUTSIDE://localhost:50012,INSIDE://kafka:9092"
      KAFKA_ADVERTISED_LISTENERS: "OUTSIDE://localhost:50012,INSIDE://kafka:9092"
      KAFKA_BROKER_ID: 1
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
    ports:
      - "50012:9092"
    extra_hosts:
      - "moby:127.0.0.1"
    depends_on:
      - zookeeper

我没有看到任何错误，但这个错误(我认为可以忽略):

ERROR Could not submit metrics to Kafka topic __confluent.support.metrics: Failed to construct kafka producer (io.confluent.support.metrics.BaseMetricsReporter)

当我尝试像这样使用 Kafka 控制台生产者发送消息时

echo 'my-message' | kafka-console-producer.sh --broker-list localhost:50012 --topic test

我得到这个异常并且消息没有被发送:

[2018-07-25 10:56:31,283] ERROR Error when sending message to topic test with key: null, value: 10 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)

org.apache.kafka.common.errors.TimeoutException:60000 毫秒后无法更新元数据。

---

更新:感谢 Robin 的正确回答，我们能够使用以下 docker-compose.yml 运行它:

---
version: "3.4"

services:

zookeeper:
    image: "confluentinc/cp-zookeeper:latest"
    environment:
    ZOOKEEPER_CLIENT_PORT: 2181
    ZOOKEEPER_TICK_TIME: 2000
    extra_hosts:
      - "moby:127.0.0.1"

kafka:
    image: "confluentinc/cp-kafka:latest"
    environment:
    KAFKA_BROKER_ID: 1
    KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
    KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT"
    KAFKA_INTER_BROKER_LISTENER_NAME: "INSIDE"
    KAFKA_ADVERTISED_LISTENERS: "INSIDE://kafka:29092,OUTSIDE://localhost:50012"
    KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
    ports:
      - "50012:50012"
    extra_hosts:
      - "moby:127.0.0.1"
    depends_on:
      - zookeeper

Answer 1

你的配置有问题

      KAFKA_ADVERTISED_LISTENERS: "OUTSIDE://localhost:50012,INSIDE://kafka:9092"
[…]
    ports:
      - "50012:9092"

KAFKA_ADVERTISED_LISTENERS 就是代理广告的地址。因此，一方面您将其配置为 localhost:50012，但另一方面您从 Docker 配置中将 9092 公开为外部端口。

如果你只是简单地使用

 ports:
      - "50012:50012"

那么它应该可以正常工作。

看看这个 docker-compose有关在外部和内部公开 Kafka 的工作配置的示例。