spring-boot - 通过使用 Docker-Compose 和 Spring Boot 运行的 IP 地址访问远程主机中的 Kafka

Question

我有这个 docker-compose.yml，我在其中运行 Zookeeper、Kafka、Kafka Connect 和 KafDrop，问题是，当我在本地运行时，我可以从我的 Spring Boot 应用程序连接以使用一些主题消息。

我需要的是在 Linux 机器上运行相同的配置，并能够以相同的方式从 Spring Boot 应用程序进行连接。

当在 Linux 机器上远程运行它时，一切似乎都运行正常，但是当我尝试从 Spring Boot 应用程序连接时，我收到一些错误，表明连接出现问题。

我会尝试一步一步地解释，看看是否有人可以对此给出“启示”:

docker-compose.yml:

version: '3'

services:

  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    networks: 
      - broker-kafka
    ports:
      - 2181:2181
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  kafka:
    image: confluentinc/cp-kafka:latest
    networks: 
      - broker-kafka
    restart: unless-stopped
    depends_on:
      - zookeeper
    ports:
      - 9092:9092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENERS: 
         INTERNAL://kafka:29092,
         EXTERNAL://localhost:9092
      KAFKA_ADVERTISED_LISTENERS: 
         INTERNAL://kafka:29092,
         EXTERNAL://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 
         INTERNAL:PLAINTEXT,
         EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_LOG_RETENTION_HOURS: 12
    
  connect:
    image: cdc:latest
    networks: 
      - broker-kafka
    depends_on:
      - zookeeper
      - kafka
    ports:
      - 8083:8083
    environment:
      CONNECT_BOOTSTRAP_SERVERS: kafka:29092
      CONNECT_REST_PORT: 8083
      CONNECT_GROUP_ID: connect-1
      CONNECT_CONFIG_STORAGE_TOPIC: connect-1-config
      CONNECT_OFFSET_STORAGE_TOPIC: connect-1-offsets
      CONNECT_STATUS_STORAGE_TOPIC: connect-1-status
      CONNECT_KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter
      CONNECT_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
      CONNECT_OFFSET.STORAGE.REPLICATION.FACTOR: 1
      CONNECT_CONFIG.STORAGE.REPLICATION.FACTOR: 1
      CONNECT_OFFSET.STORAGE.PARTITIONS: 1
      CONNECT_STATUS.STORAGE.REPLICATION.FACTOR: 1
      CONNECT_STATUS.STORAGE.PARTITIONS: 1
      CONNECT_REST_ADVERTISED_HOST_NAME: localhost
      
  kafdrop:
    image: obsidiandynamics/kafdrop:latest
    networks: 
      - broker-kafka
    depends_on:
      - kafka
    ports:
      - 19000:9000
    environment:
      KAFKA_BROKERCONNECT: kafka:29092
      
networks: 
  broker-kafka:
    driver: bridge

我需要的是向我的网络公开此 IP 机器，以供我的 Spring Boot 应用程序访问。假设这台 Linux 机器的 IP 为 10.12.54.99。如何使 Kafka 可以通过以下方式访问:10.12.54.99:9090？

这是我的 application.properties:

spring.kafka.bootstrap-servers=10.12.54.99:9092

spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.apache.kafka.common.serialization.StringSerializer

spring.kafka.consumer.enable-auto-commit=false
spring.kafka.consumer.auto-commit-interval=100
spring.kafka.consumer.max-poll-records=10
spring.kafka.consumer.key-deserializer=org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
spring.kafka.consumer.value-deserializer=org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
spring.kafka.consumer.group-id=connect-sql-server
spring.kafka.consumer.auto-offset-reset=earliest

spring.kafka.listener.ack-mode=manual-immediate
spring.kafka.listener.poll-timeout=3000
spring.kafka.listener.concurrency=3

spring.kafka.properties.spring.deserializer.key.delegate.class=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.properties.spring.deserializer.value.delegate.class=org.apache.kafka.common.serialization.StringDeserializer

这是一个唯一的特定于消费者的应用程序(这里没有使用生产者)。

当我运行应用程序时:

2020-12-07 10:59:40.361  WARN 58716 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-connect-sql-server-1, groupId=connect-sql-server] Connection to node -1 (/10.12.54.99:9092) could not be established. Broker may not be available.
2020-12-07 10:59:40.362  WARN 58716 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-connect-sql-server-1, groupId=connect-sql-server] Bootstrap broker 10.12.54.99:9092 (id: -1 rack: null) disconnected

所有的防火墙端口都在 Linux 防火墙机器中启用。

任何启发将不胜感激。

Answer 1

您需要绑定(bind)您的服务器的公网IP，以便能够远程访问代理。但是，如果您不想对 ip 进行硬编码，则可以使用 .env 文件。

执行以下操作:

1. 创建 config.env 文件。

2. 在 config.env 中添加此行并添加您的主机 ip，如下所示:

   DOCKER_HOST_IP=111.111.11.111

3. 更新您的 docker-compose:

version: '3'

services:

  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    networks: 
      - broker-kafka
    ports:
      - ${DOCKER_HOST_IP:-127.0.0.1}:2181:2181
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  kafka:
    image: confluentinc/cp-kafka:latest
    networks: 
      - broker-kafka
    restart: unless-stopped
    depends_on:
      - zookeeper
    ports:
      - ${DOCKER_HOST_IP:-127.0.0.1}:9092:9092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENERS: 
         INTERNAL://kafka:29092,
         EXTERNAL://localhost:9092
      KAFKA_ADVERTISED_LISTENERS: 
         INTERNAL://kafka:29092,
         EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9092:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 
         INTERNAL:PLAINTEXT,
         EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_LOG_RETENTION_HOURS: 12
    
  connect:
    image: cdc:latest
    networks: 
      - broker-kafka
    depends_on:
      - zookeeper
      - kafka
    ports:
      - 8083:8083
    environment:
      CONNECT_BOOTSTRAP_SERVERS: kafka:29092
      CONNECT_REST_PORT: 8083
      CONNECT_GROUP_ID: connect-1
      CONNECT_CONFIG_STORAGE_TOPIC: connect-1-config
      CONNECT_OFFSET_STORAGE_TOPIC: connect-1-offsets
      CONNECT_STATUS_STORAGE_TOPIC: connect-1-status
      CONNECT_KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter
      CONNECT_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
      CONNECT_OFFSET.STORAGE.REPLICATION.FACTOR: 1
      CONNECT_CONFIG.STORAGE.REPLICATION.FACTOR: 1
      CONNECT_OFFSET.STORAGE.PARTITIONS: 1
      CONNECT_STATUS.STORAGE.REPLICATION.FACTOR: 1
      CONNECT_STATUS.STORAGE.PARTITIONS: 1
      CONNECT_REST_ADVERTISED_HOST_NAME: localhost
      
  kafdrop:
    image: obsidiandynamics/kafdrop:latest
    networks: 
      - broker-kafka
    depends_on:
      - kafka
    ports:
      - 19000:9000
    environment:
      KAFKA_BROKERCONNECT: kafka:29092
      
networks: 
  broker-kafka:
    driver: bridge

如果未找到 DOCKER_HOST_IP，它将绑定(bind)到 127.0.0.1。

1. 运行以下命令:

sudo docker-compose -f path-to-docker-compose.yml --env-file path-to-config.env up -d --force-recreate