google-kubernetes-engine - GCP - 具有网络端点组 NEG 的区域 GKE 集群(具有 HTTP LoadBalancer 和 Cloud Armor)

Question

感谢 Gabriel Hodoroaga 和他的 tutorial我们在 GCP 中配置了这个流程:

Internet > HTTP Load Balancer > Network Endpoint Groups > GKE in one zone > ingress-nginx

但是我们需要将 GKE 从区域切换到区域。所以我重建了这个配置，但很多人认为我是通过 gcloud 命令手动完成的。我相信有一些更好的解决方案，因为它有很大的缺点:

• 它仅适用于初始部署。如果带有 ingress-nginx 的 pod 稍后移动到不同的区域(重启后)，连接将中断，并且必须手动将后端与正确的 neg 重新关联。
• 我们需要将它应用到我们不使用 gcloud 命令但一切都通过 github 操作和 terraform 设置的环境中。

初始部署后运行良好:但是在重新启动 ingress-nginx pod 之后，它会移动到不同的区域并且后端保持与旧区域的连接:

我们的配置在以下教程中描述:

https://github.com/robinpecha/gcp-regionalgke-httplb-negs-ingressnginx/blob/main/lb-negs-nging-reg.sh.md

---

GCP - HTTP 负载均衡器 > NEGS > 区域 GKE 集群 > INGRESS-NGINX

基于 tutorial的 Gabriel Hodoroaga .

变量

至少替换 YOURDOMAIN。

CLUSTER_NAME="lb-negs-nging-reg"
REGION="europe-west2"
YOURDOMAIN="put-your-domain.here"
echo $CLUSTER_NAME ; echo $REGION ; echo $YOURDOMAIN

创建集群

gcloud container clusters create $CLUSTER_NAME --region $REGION --machine-type "e2-medium" --enable-ip-alias --num-nodes=2 

添加 helm ingress-nginx

helm repo update
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

安装ingress-nginx

为 ingress-nginx 创建一个文件 values.regional.yaml:

cat << EOF > values.regional.yaml
controller:
  service:
    type: ClusterIP
    annotations:
      cloud.google.com/neg: '{"exposed_ports": {"80":{"name": "ingress-nginx-80-neg"}}}'
EOF

并安装它:

helm install -f values.regional.yaml ingress-nginx ingress-nginx/ingress-nginx

安装虚拟网络服务器

准备配置:

cat << EOF > dummy-app-lightweb.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: lightweb
spec:
  selector:
    matchLabels:
      app: dummy
  replicas: 3
  template:
    metadata:
      labels:
        app: dummy
    spec:
      containers:
      - name: lightweb
        image: alastairhm/alpine-lighttpd-php
        ports:
        - name: http
          containerPort: 80
        lifecycle:
          postStart:
            exec:
              command: ["/bin/sh", "-c", 'wget https://raw.githubusercontent.com/robinpecha/hello-world/main/php-header/index.php -P /var/www/']
---
apiVersion: v1
kind: Service
metadata:
  name: dummy-service
spec:
  type: NodePort
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: dummy
EOF

应用这个配置:

kubectl apply -f dummy-app-lightweb.yaml 

现在您可以检查您的虚拟网络服务器是否正常工作:

kubectl get pods
#  NAME                                        READY   STATUS    RESTARTS   AGE
#  ingress-nginx-controller-???????????-????   1/1     Running   0          5m8s
#  lightweb-???????????-????                   1/1     Running   0          4m35s
#  lightweb-???????????-????                   1/1     Running   0          4m35s
#  lightweb-???????????-????                   1/1     Running   0          4m35s

kubectl port-forward lightweb-???????????-???? 8080:80
#  Forwarding from 127.0.0.1:8080 -> 80
#  Forwarding from [::1]:8080 -> 80

Check in your browser http://localhost:8080

Ctrl+C

创建入口对象

准备配置。不要忘记将 $YOURDOMAIN 的 dns 记录指向本教程末尾显示的 ip。或者简单地编辑您的本地主机文件以获取假域:

cat << EOF > dummy-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: dummy-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: "$YOURDOMAIN"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: dummy-service
            port:
              number: 80
EOF

并应用它:

kubectl apply -f dummy-ingress.yaml 

找到入口的网络标签和区域

NETWORK_TAGS=$(gcloud compute instances list --filter="name=( $(kubectl get pod -l app.kubernetes.io/name=ingress-nginx -o jsonpath='{.items[0].spec.nodeName}') )" --format="value(tags.items[0])") ; echo $NETWORK_TAGS 

NODEZONE=$(gcloud compute instances list --filter="name=( $(kubectl get pod -l app.kubernetes.io/name=ingress-nginx -o jsonpath='{.items[0].spec.nodeName}') )" --format="value(zone)"); echo $NODEZONE 

配置防火墙

gcloud compute firewall-rules create $CLUSTER_NAME-lb-fw --allow tcp:80 --source-ranges 130.211.0.0/22,35.191.0.0/16 --target-tags $NETWORK_TAGS 

添加健康检查配置

gcloud compute health-checks create http app-service-80-health-check --request-path /healthz --port 80 --check-interval 60 --unhealthy-threshold 3 --healthy-threshold 1 --timeout 5 

添加后端服务

gcloud compute backend-services create $CLUSTER_NAME-lb-backend --health-checks app-service-80-health-check --port-name http --global --enable-cdn --connection-draining-timeout 300 

将我们的 NEG 附加到后端服务

gcloud compute backend-services add-backend $CLUSTER_NAME-lb-backend --network-endpoint-group=ingress-nginx-80-neg --network-endpoint-group-zone=$NODEZONE --balancing-mode=RATE --capacity-scaler=1.0 --max-rate-per-endpoint=1.0 --global 

设置前端

gcloud compute url-maps create $CLUSTER_NAME-url-map --default-service $CLUSTER_NAME-lb-backend 
gcloud compute target-http-proxies create $CLUSTER_NAME-http-proxy --url-map $CLUSTER_NAME-url-map 
gcloud compute forwarding-rules create $CLUSTER_NAME-forwarding-rule --global --ports 80 --target-http-proxy $CLUSTER_NAME-http-proxy 

启用日志记录

gcloud compute backend-services update $CLUSTER_NAME-lb-backend --enable-logging --global 

测试

给它一些时间来部署......

IP_ADDRESS=$(gcloud compute forwarding-rules describe $CLUSTER_NAME-forwarding-rule --global --format="value(IPAddress)") ; echo $IP_ADDRESS
curl -s -I http://$IP_ADDRESS/ #404
echo curl -s -I http://$YOURDOMAIN/ #200

清理

# delete the forwarding-rule aka frontend
gcloud -q compute forwarding-rules delete $CLUSTER_NAME-forwarding-rule --global 
# delete the http proxy
gcloud -q compute target-http-proxies delete $CLUSTER_NAME-http-proxy 
# delete the url map
gcloud -q compute url-maps delete $CLUSTER_NAME-url-map 
# delete the backend
gcloud -q compute backend-services delete $CLUSTER_NAME-lb-backend --global
# delete the health check
gcloud -q compute health-checks delete app-service-80-health-check
# delete the firewall rule
gcloud -q compute firewall-rules delete $CLUSTER_NAME-lb-fw 

kubectl delete -f dummy-ingress.yaml 
kubectl delete -f dummy-app-lightweb.yaml 
helm delete ingress-nginx 

# delete the cluster
gcloud -q container clusters delete $CLUSTER_NAME --zone=$ZONE
# delete the NEG  
gcloud -q compute network-endpoint-groups delete ingress-nginx-80-neg --zone=$REGION-a
gcloud -q compute network-endpoint-groups delete ingress-nginx-80-neg --zone=$REGION-b
gcloud -q compute network-endpoint-groups delete ingress-nginx-80-neg --zone=$REGION-c
gcloud -q compute network-endpoint-groups list

Answer 1

奇怪，它通过简单地将后端添加到所有 negs 开始工作，即使在没有 ingress-nginx 的情况下也是如此。

gcloud compute backend-services add-backend $CLUSTER_NAME-lb-backend --network-endpoint-group=ingress-nginx-80-neg --network-endpoint-group-zone=europe-west2-a --balancing-mode=RATE --capacity-scaler=1.0 --max-rate-per-endpoint=1.0 --global

gcloud compute backend-services add-backend $CLUSTER_NAME-lb-backend --network-endpoint-group=ingress-nginx-80-neg --network-endpoint-group-zone=europe-west2-b --balancing-mode=RATE --capacity-scaler=1.0 --max-rate-per-endpoint=1.0 --global

gcloud compute backend-services add-backend $CLUSTER_NAME-lb-backend --network-endpoint-group=ingress-nginx-80-neg --network-endpoint-group-zone=europe-west2-c --balancing-mode=RATE --capacity-scaler=1.0 --max-rate-per-endpoint=1.0 --global