docker - 如何为 Swisscom elasticsearch 公共(public)云 (CloudFoundry) 配置 Kibana

Question

注意:此问题针对Swisscom提供的Elasticsearch服务

问题:(又名:tl;dr)

需要什么配置才能让官方 Kibana docker 容器连接到 Swisscom Elasticsearch 服务？

背景:

直到大约一年前 Swisscom public cloud在单一服务产品中提供完整的 ELK 堆栈(Elasticsearch、Logstash、Kibana)。当此服务停止时，Swisscom 仅提供 Elasticsearch 服务取而代之，并要求客户通过提供的 CloudFoundry build_packs(Kibana、Logstash)设置自己的 Kibana 和 Logstash 解决方案。此处讨论了迁移建议:https://ict.swisscom.ch/2018/04/building-the-elk-stack-on-our-new-elasticsearch/

最近，在 Swisscom 基于 CloudFoundry 的 PaaS 产品上运行应用程序的底层操作系统(称为“堆栈”)已经升级。前面提到的 build_packs 现已过时，并且已被 Swisscom 声明为弃用。现在的建议是转向通用 Docker container provided by Elastic如此处所述:https://github.com/swisscom/kibana-buildpack/issues/3

我尝试了什么:

CloudFoundry 通常可以很好地与 Docker 容器配合使用，并且整个事情应该像为 docker 容器提供一些有效配置一样简单明了。我当前的 Kibana manifest.yml 看起来像这样，但 Kibana 应用程序最终无法连接:

---
applications:
- name: kibana-test-example
  docker:
    image: docker.elastic.co/kibana/kibana:6.1.4
  memory: 4G
  disk_quota: 5G
  services:
    - elasticsearch-test-service
  env:
    SERVER_NAME: kibana-test
    ELASTICSEARCH_URL: https://abcdefghijk.elasticsearch.lyra-836.appcloud.swisscom.com
    ELASTICSEARCH_USERNAME: username_provided_by_elasticsearch_service
    ELASTICSEARCH_PASSWORD: password_provided_by_elasticsearch_service
    XPACK_MONITORING_ENABLED: true

附加信息:

Swisscom 提供的 Elasticsearch Service 目前运行在 6.1.3 版本上。据我所知，它安装了 x-pack。

你遇到了什么错误？

我尝试了一下配置，发现了不同的错误，其中大部分似乎与针对 Elasticsearch 服务的身份验证失败有关。

这是一些示例性的初始日志输出(不过说真的，您需要一个正在运行的 Kibana 才能读取它...)

   2019-05-10T08:08:34.43+0200 [CELL/0] OUT Cell eda692ed-f4c3-4a5e-86aa-c0d1641b029f successfully created container for instance 385e5b7f-1570-46cd-532a-c5b4
   2019-05-10T08:08:48.60+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:08:48Z","tags":["info","optimize"],"pid":6,"message":"Optimizing and caching bundles for graph, monitoring, apm, kibana, stateSessionStorageRedirect, timelion, login, logout, dashboardViewer and status_page. This may take a few minutes"}
   2019-05-10T08:15:07.68+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:07Z","tags":["info","optimize"],"pid":6,"message":"Optimization of bundles for graph, monitoring, apm, kibana, stateSessionStorageRedirect, timelion, login, logout, dashboardViewer and status_page complete in 379.08 seconds"}
   2019-05-10T08:15:07.77+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:07Z","tags":["status","plugin:kibana@6.1.4","info"],"pid":6,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
   2019-05-10T08:15:07.82+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:07Z","tags":["status","plugin:elasticsearch@6.1.4","info"],"pid":6,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
   2019-05-10T08:15:07.86+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:07Z","tags":["status","plugin:xpack_main@6.1.4","info"],"pid":6,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
   2019-05-10T08:15:07.86+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:07Z","tags":["status","plugin:graph@6.1.4","info"],"pid":6,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
   2019-05-10T08:15:07.88+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:07Z","tags":["status","plugin:monitoring@6.1.4","info"],"pid":6,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
   2019-05-10T08:15:07.89+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:07Z","tags":["status","plugin:xpack_main@6.1.4","error"],"pid":6,"state":"red","message":"Status changed from yellow to red - Authentication Exception","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
   2019-05-10T08:15:07.89+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:07Z","tags":["status","plugin:graph@6.1.4","error"],"pid":6,"state":"red","message":"Status changed from yellow to red - Authentication Exception","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
   2019-05-10T08:15:07.89+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:07Z","tags":["status","plugin:elasticsearch@6.1.4","error"],"pid":6,"state":"red","message":"Status changed from yellow to red - Authentication Exception","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
   2019-05-10T08:15:11.39+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:11Z","tags":["reporting","warning"],"pid":6,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent pending reports from failing on restart, please set xpack.reporting.encryptionKey in kibana.yml"}
   2019-05-10T08:15:11.39+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:11Z","tags":["status","plugin:reporting@6.1.4","error"],"pid":6,"state":"red","message":"Status changed from uninitialized to red - Authentication Exception","prevState":"uninitialized","prevMsg":"uninitialized"}

实际相关的错误信息似乎是这样的:

2019-05-10T08:15:11.66+0200 [APP/PROC/WEB/0] OUT {"type":"log","@timestamp":"2019-05-10T06:15:11Z","tags":["license","warning","xpack"],"pid":6,"message":"License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. [security_exception] unable to authenticate user [ABCDEFGHIJKLMNOPQRST] for REST request [/_xpack], with { header={ WWW-Authenticate=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } } :: {\"path\":\"/_xpack\",\"statusCode\":401,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"unable to authenticate user [ABCDEFGHIJKLMNOPQRST] for REST request [/_xpack]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}}],\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"unable to authenticate user [ABCDEFGHIJKLMNOPQRST] for REST request [/_xpack]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}},\\\"status\\\":401}\",\"wwwAuthenticateDirective\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}"}

我已尝试按照其他地方的建议设置 XPACK_SECURITY_ENABLED: false 以及设置实际的 SERVER_HOST，这似乎让事情变得更糟。

我非常感谢有人使用现有的 Kibana docker 镜像连接到 Swisscom 提供的 Elasticsearch 服务的工作示例。

Answer 1

会不会是你混淆了用户名和密码？当我检查我的服务 key 密码出现在用户名之前时，这可能会导致您这边出现复制粘贴错误:

cf service-key myece mykey|grep kibana_system
 "kibana_system_password": "aKvOpMVrXGCJ4PJht",
 "kibana_system_username": "aksTxVNyLU4JWiQOE6V",

我尝试使用您的 manifest.yml 推送 Kibana，它在我的案例中完美运行。

Swisscom 还更新了有关如何将 Kibana 和 Logstash 与 Docker 结合使用的文档:

https://docs.developer.swisscom.com/service-offerings/kibana-docker.html https://docs.developer.swisscom.com/service-offerings/logstash-docker.html