docker - 如何调试为什么Telegraf不将数据从MQTT发送到influxdb

Question

我正在尝试如下设置TIG堆栈:

传感器数据-> MQTT(Eclipse-Mosquitto)-> Telegraf-> InfluxDB-> Grafana

我尚未设置传感器数据的源，但是在其当前状态下，所有内容都可以编译并运行而没有明显的错误，并且似乎都已连接，但是发布到MQTT主题的数据并没有最终出现在我的Influxdb数据库中。

$ sudo docker-compose up
Creating network "tig-samson-feature-setup-docker-compose_default" with the default driver
Creating mosquitto ... done
Creating influxdb  ... done
Creating grafana   ... done
Creating telegraf  ... done
Attaching to influxdb, mosquitto, grafana, telegraf
mosquitto    | 1583407265: mosquitto version 1.6.9 starting
mosquitto    | 1583407265: Config loaded from /mosquitto/config/mosquitto.conf.
mosquitto    | 1583407265: Opening ipv4 listen socket on port 1883.
mosquitto    | 1583407265: Opening ipv6 listen socket on port 1883.
mosquitto    | 1583407266: New connection from 192.168.32.1 on port 1883.
mosquitto    | 1583407266: New client connected from 192.168.32.1 as mosqsub|5779-jackbridge (p1, c1, k60).
grafana      | t=2020-03-05T11:21:05+0000 lvl=info msg="Starting Grafana" logger=server version=6.6.2 commit=3fa63cfc34 branch=HEAD compiled=2020-02-20T12:03:49+0000
grafana      | t=2020-03-05T11:21:05+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini
grafana      | t=2020-03-05T11:21:05+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini
grafana      | t=2020-03-05T11:21:05+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.data=/var/lib/grafana"

...continues indefinitely.. can add more detail if it's helpfuL!

以下是我的一些代码以及我尝试过的一些事情

在ubuntu VM上运行的文件架构(在Mac上的virtualbox中运行)

docker-compose.yml
telegraf.conf
influxdb (folder)
-----data (folder)
-----------data (locked folder auto generated)
-----------meta (locked folder auto generated)
-----------wal (locked folder auto generated)
data 
----mosquitto (empty folder)

docker 组成

# # docker-compose.yml
version: '3'

services: 
  influxdb:
    container_name: influxdb
    image: influxdb
    ports:
        - "8086:8086"
    environment:
      - INFLUXDB_DB=sensors
      - INFLUXDB_ADMIN_USER=admin
      - INFLUXDB_ADMIN_PASSWORD=telegraf
    volumes:
      - ./influxdb/data:/var/lib/influxdb
    restart: always

  telegraf: 
    container_name: telegraf
    image: telegraf
    restart: always
    depends_on: 
      - mosquitto
      - influxdb
    volumes:
    - ./telegraf.conf:/etc/telegraf/telegraf.conf:ro   


  grafana: 
    container_name: grafana
    image: grafana/grafana
    links:
      - influxdb
    hostname: grafana
    ports:
        - "3000:3000"

  mosquitto:
    container_name: mosquitto
    image: eclipse-mosquitto
    ports:
      - "1883:1883"
      - "9001:9001"
    restart: always
    volumes:
      - ./mosquitto.conf:/mqtt/config:ro
#      - /data/mosquitto:mosquitto/data

(我用docker-compose up启动程序)

telegraf.conf

 [[outputs.file]]
#   ## Files to write to, "stdout" is a specially handled file.
   files = ["stdout"]

# # Read metrics from MQTT topic(s)
[[inputs.mqtt_consumer]]
  servers = ["tcp://localhost:1883"]
  topics = [
    "sensors"
  ]
  data_format = "influx"
#   ## MQTT broker URLs to be used. The format should be scheme://host:port,
#   ## schema can be tcp, ssl, or ws.
#   servers = ["tcp://127.0.0.1:1883"]


[[outputs.influxdb]]
urls = ["http://influxdb:8086"]
database = "sensors"
skip_database_creation = true
username = "telegraf"
password = "telegraf"

我可以访问telegraf docker容器并ping influxdb，因此似乎已连接

root@fc1d9af8f0c6:/# curl -sl -I http://influxdb:8086/ping
HTTP/1.1 204 No Content
Content-Type: application/json
Request-Id: 12b4923a-5ec7-11ea-8006-0242c0a81002
X-Influxdb-Build: OSS
X-Influxdb-Version: 1.7.10
X-Request-Id: 12b4923a-5ec7-11ea-8006-0242c0a81002
Date: Thu, 05 Mar 2020 09:52:49 GMT

我可以在其他终端上发布和订阅MQTT:

$ mosquitto_pub -t 'sensors' -m 'strength, site=room1 value=HELLO tony'

这是在另一个订阅传感器的终端中接收的

$ mosquitto_sub -t 'sensors' -v # note this is executed before mosquitto_pub
sensors strength, site=room1 value=HELLO tony

当我运行mosquitto_sub时，运行docker-compose的终端没有任何 Activity 。

当我运行mosquitto_pub时，客户端被检测为正在连接，但是publish事件似乎未注册

mosquitto    | 1583407850: New connection from 192.168.32.1 on port 1883.
mosquitto    | 1583407850: New client connected from 192.168.32.1 as mosqpub|7039-jackbridge (p1, c1, k60).
mosquitto    | 1583407850: Client mosqpub|7039-jackbridge disconnected.

Telegraf似乎正在向Influxdb发送指标。在telegraf [[outputs.file]]中，我正在发送的文件= ['stdout']，您在上面。目前尚不清楚指标是否已到达influxdb，因为我无法找到它们。

端子输出样本

telegraf     | disk,device=sda1,fstype=ext4,host=foobar,mode=rw,path=/etc/resolv.conf inodes_used=189723i,total=10499674112i,free=3154145280i,used=6791987200i,used_percent=68.2877210177679,inodes_total=655360i,inodes_free=465637i 1583406223000000000
telegraf     | cpu,cpu=cpu-total,host=foobar usage_steal=0,usage_system=7.14285714285916,usage_softirq=0.8281573498966311,usage_nice=0,usage_iowait=0.10351966873709728,usage_irq=0,usage_guest=0,usage_guest_nice=0,usage_user=56.41821946171202,usage_idle=35.507246376802996 1583406223000000000
telegraf     | cpu,cpu=cpu0,host=foobar usage_user=56.41821946171202,usage_irq=0,usage_steal=0,usage_system=7.14285714285916,usage_idle=35.507246376802996,usage_nice=0,usage_iowait=0.10351966873709728,usage_softirq=0.8281573498966311,usage_guest=0,usage_guest_nice=0 1583406223000000000
telegraf     | processes,host=foobar sleeping=2i,paging=0i,idle=0i,dead=0i,total_threads=11i,blocked=0i,zombies=0i,stopped=0i,running=0i,total=2i,unknown=0i 1583406223000000000
telegraf     | 2020-03-05T11:03:53Z D! [outputs.file] Wrote batch of 26 metrics in 1.676587ms
telegraf     | 2020-03-05T11:03:53Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
telegraf     | 2020-03-05T11:03:53Z D! [outputs.influxdb] Wrote batch of 26 metrics in 8.885827ms
telegraf     | 2020-03-05T11:03:53Z D! [outputs.influxdb] Buffer fullness: 23 / 10000 metrics

在Influxdb中，创建了我的传感器数据库

compose$ sudo docker exec -it influxdb influx
[sudo] password for jackbridger:
Connected to http://localhost:8086 version 1.7.10
InfluxDB shell version: 1.7.10
> show databases
name: databases
name
----
sensors
_internal
> use _internal


> show stats
name: runtime
Alloc    Frees    HeapAlloc HeapIdle HeapInUse HeapObjects HeapReleased HeapSys  Lookups Mallocs  NumGC NumGoroutine PauseTotalNs Sys      TotalAlloc
-----    -----    --------- -------- --------- ----------- ------------ -------  ------- -------  ----- ------------ ------------ ---      ----------
16426120 26535115 16426120  44695552 21823488  225331      28131328     66519040 0       26760446 277   26           6132353      72483064 1337594448

name: queryExecutor
queriesActive queriesExecuted queriesFinished queryDurationNs recoveredPanics
------------- --------------- --------------- --------------- ---------------
1             3               2               3462237         0

name: database
tags: database=_internal
numMeasurements numSeries
--------------- ---------
12              467


...continues for a while.. can add more detail if it's helpfuL!

我尝试设置蚊子的音量，但无法正常工作-我认为这可能是一个问题

任何人都可以提供关于如何调试它的任何指导，以弄清楚Telegraf为何不将添加到MQTT的消息发送到Influxdb中的传感器数据库？

谢谢!

Answer 1

以下内容由一位 friend 非常有帮助地提供，它解决了该问题

# # Read metrics from MQTT topic(s)
[[inputs.mqtt_consumer]]
  servers = ["tcp://localhost:1883"]
  topics = [
    "sensors"
  ]
  data_format = "influx"
#   ## MQTT broker URLs to be used. The format should be scheme://host:port,
#   ## schema can be tcp, ssl, or ws.
#   servers = ["tcp://127.0.0.1:1883"]

localhost:1883会将主机解析为telegraf容器。如果要引用在同一docker compose设置中运行的另一个容器，则可以将服务名称用作主机名，因此在这种情况下，请尝试tcp:// mosquitto:1883。在这里阅读: https://docs.docker.com/compose/networking/

现在，每个容器都可以查找主机名web或db，并取回相应容器的IP地址。例如，网络的应用程序代码可以连接到URL postgres:// db:5432并开始使用Postgres数据库。