kubernetes - 在 istio 后面访问支持 nginx 的服务时，对等方重置连接

Question

修改了helm chart alerta 以在启用了 istio 的 GKE 集群上启动它。

alerta pod 和它的 sidecar 已经创建好了

▶ k get pods | grep alerta
alerta-758bc87dcf-tp5nv                        2/2     Running   0          22m

当我尝试访问我的虚拟服务指向的 url 时

出现以下错误

upstream connect error or disconnect/reset before headers. reset reason: connection termination

▶ k get vs alerta-virtual-service -o yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  annotations:
    helm.fluxcd.io/antecedent: mynamespace:helmrelease/alerta
  creationTimestamp: "2020-04-23T14:45:04Z"
  generation: 1
  name: alerta-virtual-service
  namespace: mynamespace
  resourceVersion: "46844125"
  selfLink: /apis/networking.istio.io/v1alpha3/namespaces/mynamespace/virtualservices/alerta-virtual-service
  uid: 2a3caa13-3900-4da1-a3a1-9f07322b52b0
spec:
  gateways:
  - mynamespace/istio-ingress-gateway
  hosts:
  - alerta.myurl.com
  http:
  - appendHeaders:
      x-request-start: t=%START_TIME(%s.%3f)%
    match:
    - uri:
        prefix: /
    route:
    - destination:
        host: alerta
        port:
          number: 80
    timeout: 60s

这是服务

▶ k get svc alerta -o yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    helm.fluxcd.io/antecedent: mynamespace:helmrelease/alerta
  creationTimestamp: "2020-04-23T14:45:04Z"
  labels:
    app: alerta
    chart: alerta-0.1.0
    heritage: Tiller
    release: alerta
  name: alerta
  namespace: mynamespace
  resourceVersion: "46844120"
  selfLink: /api/v1/namespaces/mynamespace/services/alerta
  uid: 4d4a3c73-ee42-49e3-a4cb-8c51536a0508
spec:
  clusterIP: 10.8.58.228
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
  selector:
    app: alerta
    release: alerta
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

但是，当我通过另一个 pod 执行到集群中并尝试到达 alerta svc 端点时:

/ # curl -IL http://alerta
curl: (56) Recv failure: Connection reset by peer
/ # nc -zv -w 3 alerta 80
alerta (10.8.58.228:80) open

虽然很明显端口是开放的

有什么建议吗？

这两个代理的链接会不会产生问题？ envoy 背后的 nginx？

容器日志看起来很正常

2020-04-23 15:34:40,272 DEBG 'nginx' stdout output: 
ip=\- [\23/Apr/2020:15:34:40 +0000] "\GET / HTTP/1.1" \200 \994 "\-" "\kube-probe/1.15+"
/web | /index.html | > GET / HTTP/1.1

编辑:这是一个冗长的 curl，其中明确设置了主机 header

/ # curl -v -H "host: alerta.myurl.com" http://alerta:80
* Rebuilt URL to: http://alerta:80/
*   Trying 10.8.58.228...
* TCP_NODELAY set
* Connected to alerta (10.8.58.228) port 80 (#0)
> GET / HTTP/1.1
> host: alerta.myurl.com
> User-Agent: curl/7.57.0
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer

app/pod使用的nginx配置文件如下FWIW

worker_processes 4;
pid /tmp/nginx.pid;

daemon off;
error_log /dev/stderr info;

events {
        worker_connections 1024;
}

http {
        client_body_temp_path /tmp/client_body;
        fastcgi_temp_path /tmp/fastcgi_temp;
        proxy_temp_path /tmp/proxy_temp;
        scgi_temp_path /tmp/scgi_temp;
        uwsgi_temp_path /tmp/uwsgi_temp;

        include /etc/nginx/mime.types;

        gzip on;
        gzip_disable "msie6";

        log_format main 'ip=\$http_x_real_ip [\$time_local] '
        '"\$request" \$status \$body_bytes_sent "\$http_referer" '
        '"\$http_user_agent"' ;

        log_format scripts '$document_root | $uri | > $request';

        default_type application/octet-stream;

        server {
                listen 8080 default_server;


                access_log /dev/stdout main;
                access_log /dev/stdout scripts;

                location ~ /api {
                        include /etc/nginx/uwsgi_params;
                        uwsgi_pass unix:/tmp/uwsgi.sock;

                        proxy_set_header Host $host:$server_port;
                        proxy_set_header X-Real-IP $remote_addr;
                        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                }

                root /web;
                index index.html;
                location / {
                        try_files $uri $uri/ /index.html;
                }
        }
}

编辑 2:尝试获取 istio 身份验证策略

✔                                                                                                             18h55m  ⍉
▶ kubectl get peerauthentication.security.istio.io
No resources found.
✔                                                                                                              18h55m
▶ kubectl get peerauthentication.security.istio.io/default -o yaml
Error from server (NotFound): peerauthentications.security.istio.io "default" not found

编辑 3:当从 istio 代理容器中对服务执行 curl 时

▶ k exec -it alerta-758bc87dcf-jzjgj -c istio-proxy bash
istio-proxy@alerta-758bc87dcf-jzjgj:/$ curl -v http://alerta:80
* Rebuilt URL to: http://alerta:80/
*   Trying 10.8.58.228...
* Connected to alerta (10.8.58.228) port 80 (#0)
> GET / HTTP/1.1
> Host: alerta
> User-Agent: curl/7.47.0
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer

Answer 1

我用 istio 1.5.2 创建了新的 gke 集群，事实上如果你检查 mtls，没有找到资源

kubectl get peerauthentication --all-namespaces

No resources found.

kubectl get peerauthentication.security.istio.io/default

Error from server (NotFound): peerauthentications.security.istio.io "default" not found

所以我试着做这个example这清楚地表明，当您使用 global.mtls.enabled=true 安装时，istio 处于严格的 tls 模式。

如果您按照提到的方式添加 pod、命名空间 here每个请求应该是 200，但不是

sleep.foo to httpbin.foo: 200
sleep.foo to httpbin.bar: 200
sleep.foo to httpbin.legacy: 200
sleep.bar to httpbin.foo: 200
sleep.bar to httpbin.bar: 200
sleep.bar to httpbin.legacy: 200
sleep.legacy to httpbin.foo: 000
command terminated with exit code 56
sleep.legacy to httpbin.bar: 000
command terminated with exit code 56
sleep.legacy to httpbin.legacy: 200

因此，如果您使用上面的 yaml 将 mtls 从严格更改为宽松

apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "default"
  namespace: "istio-system"
spec:
  mtls:
    mode: PERMISSIVE

现在可以了

sleep.foo to httpbin.foo: 200
sleep.foo to httpbin.bar: 200
sleep.foo to httpbin.legacy: 200
sleep.bar to httpbin.foo: 200
sleep.bar to httpbin.bar: 200
sleep.bar to httpbin.legacy: 200
sleep.legacy to httpbin.foo: 200
sleep.legacy to httpbin.bar: 200
sleep.legacy to httpbin.legacy: 200

---

此外 github issue您提供的错误。

---

关于问题

why the pod fails to mtls authenticate with itself, when curling from inside it

有一个github关于这个的问题。

---

另外看看这个istio docs .