Kubernetes + Metallb + traefik : how to get real client ip?

Question

traefik.toml:

defaultEntryPoints = ["http", "https"]

[entryPoints]
  [entryPoints.http]
  address = ":80"
    [entryPoints.http.forwardedHeaders]
      trustedIPs = ["0.0.0.0/0"]
    [entryPoints.http.redirect]
      entryPoint = "https"
  [entryPoints.https]
  address = ":443"
    [entryPoints.https.tls]
    [entryPoints.https.forwardedHeaders]
      trustedIPs = ["0.0.0.0/0"]
[api]

traefik 服务:

kind: Service
apiVersion: v1
metadata:
  name: traefik-ingress-service
  namespace: kube-system
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
    - protocol: TCP
      port: 80
      name: http
    - protocol: TCP
      port: 443
      name: https
  type: LoadBalancer

然后:

kubectl run source-ip-app --image=k8s.gcr.io/echoserver:1.4
deployment "source-ip-app" created

kubectl expose deployment source-ip-app --name=clusterip --port=80 --target-port=8080
service "clusterip" exposed

kubectl get svc clusterip
NAME        TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
clusterip   ClusterIP   10.5.55.102   <none>        80/TCP    2h

为 clusterip 创建入口:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: clusterip-ingress
  namespace: default
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  rules:
  - host: clusterip.staging
    http:
      paths:
      - backend:
          serviceName: clusterip
          servicePort: 80

clusterip.staging ip:192.168.0.69

从 IP 为 192.168.0.100 的其他电脑:

wget -qO - clusterip.staging

并获得结果:

CLIENT VALUES:
client_address=10.5.65.74
command=GET
real path=/
query=nil
request_version=1.1
request_uri=http://clusterip.staging:8080/

SERVER VALUES:
server_version=nginx: 1.10.0 - lua: 10001

HEADERS RECEIVED:
accept=text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
accept-encoding=gzip, deflate, br
accept-language=ru-RU,ru;q=0.8,en-US;q=0.5,en;q=0.3
cache-control=max-age=0
host=clusterip.staging
upgrade-insecure-requests=1
x-forwarded-for=10.5.64.0
x-forwarded-host=clusterip.staging
x-forwarded-port=443
x-forwarded-proto=https
x-forwarded-server=traefik-ingress-controller-755cc56458-t8q9k
x-real-ip=10.5.64.0
BODY:
-no body in request-

kubectl get svc --all-namespaces

NAMESPACE     NAME                      TYPE           CLUSTER-IP    EXTERNAL-IP    PORT(S)                                                 AGE
default       clusterip                 NodePort       10.5.55.102   <none>         80:31169/TCP                                            19h
default       kubernetes                ClusterIP      10.5.0.1      <none>         443/TCP                                                 22d
kube-system   kube-dns                  ClusterIP      10.5.0.3      <none>         53/UDP,53/TCP                                           22d
kube-system   kubernetes-dashboard      ClusterIP      10.5.5.51     <none>         443/TCP                                                 22d
kube-system   traefik-ingress-service   LoadBalancer   10.5.2.37     192.168.0.69   80:32745/TCP,443:30219/TCP                              1d
kube-system   traefik-web-ui            NodePort       10.5.60.5     <none>         80:30487/TCP                                            7d

如何在我的安装中获取真实IP(192.168.0.100)？为什么选择 x-real-ip 10.5.64.0？ 我在文档中找不到答案。

Answer 1

当kube-proxy使用iptables模式时，它使用NAT将数据发送到payload工作的节点，你会丢失原来的SourceIP 在这种情况下的地址。

据我了解，您在 Traefik Ingress Service 后面使用 Matallb (因为它的类型是 LoadBalancer)。这意味着从客户端到后端的流量是这样的:

客户端 -> Metallb -> Traefik LB -> Traefik 服务 -> 后端 Pod。

Traefik 正常工作并添加 header x-*，包括包含虚假内容的 x-forwarded-for 和 x-real-ip地址，这就是原因:

来自Metallb documentation :

MetalLB understands the service’s externalTrafficPolicy option and implements different announcements modes depending on the policy and announcement protocol you select.

• Layer2

  This policy results in uniform traffic distribution across all pods in the service. However, kube-proxy will obscure the source IP address of the connection when it does load-balancing, so your pod logs will show that external traffic appears to be coming from the cluster’s leader node.

• BGP

  • “Cluster” traffic policy

    With the default Cluster traffic policy, every node in your cluster will attract traffic for the service IP. On each node, the traffic is subjected to a second layer of load-balancing (provided by kube-proxy), which directs the traffic to individual pods.

    ......

    The other downside of the “Cluster” policy is that kube-proxy will obscure the source IP address of the connection when it does its load-balancing, so your pod logs will show that external traffic appears to be coming from your cluster’s nodes.

  • “Local” traffic policy

    With the Local traffic policy, nodes will only attract traffic if they are running one or more of the service’s pods locally. The BGP routers will load-balance incoming traffic only across those nodes that are currently hosting the service. On each node, the traffic is forwarded only to local pods by kube-proxy, there is no “horizontal” traffic flow between nodes.

    This policy provides the most efficient flow of traffic to your service. Furthermore, because kube-proxy doesn’t need to send traffic between cluster nodes, your pods can see the real source IP address of incoming connections.

最后，获取真实源IP地址的唯一方法是使用TrafficPolicy的“Local”模式。

如果你设置了它，你就会得到你想要的。