jenkins - 无法从 Metallb 下的 kubernetes 集群外部访问 Jenkins Pod

Question

我有一个运行 Jenkins Pod 的 Kubernetes 集群，其中为 Metallb 设置了服务。目前，当我尝试为集群外的 pod 访问 loadBalancerIP 时，我无法访问。我还有一个在集群上运行的 kube-verify pod，其服务也使用 Metallb。当我尝试在我的集群之外击中那个 pod 时，我可以毫无问题地击中它。

当我将 Jenkins pod 的服务切换为 NodePort 类型时，它可以正常工作，但一旦我将其切换回 LoadBalancer 类型，它就会停止工作。 Jenkins pod 和正在工作的 kube-verify pod 都在同一个节点上运行。

集群详细信息:主节点正在运行并无线连接到我的路由器。在主节点上，我设置了 dnsmasq 以及将连接从无线端口转发到以太网端口的 iptable 规则。每个节点都通过以太网通过交换机连接在一起。 Metallb 在 layer2 模式下设置，地址池与主节点无线端口的 ip 地址位于同一子网。 kube-proxy 设置为使用 strictArp 和 ipvs 模式。

Jenkins list :

apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins-sa
  namespace: "devops-tools"
  labels:
    app: jenkins
    version: v1
    tier: backend
---
apiVersion: v1
kind: Secret
metadata:
  name: jenkins-secret
  namespace: "devops-tools"
  labels:
    app: jenkins
    version: v1
    tier: backend
type: Opaque
data:
  jenkins-admin-password: ***************
  jenkins-admin-user: ********
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: jenkins
  namespace: "devops-tools"
  labels:
    app: jenkins
    version: v1
    tier: backend
data:
  jenkins.yaml: |-
    jenkins:
      authorizationStrategy:
        loggedInUsersCanDoAnything:
          allowAnonymousRead: false
      securityRealm:
        local:
          allowsSignup: false
          enableCaptcha: false
          users:
          - id: "${jenkins-admin-username}"
            name: "Jenkins Admin"
            password: "${jenkins-admin-password}"
      disableRememberMe: false
      mode: NORMAL
      numExecutors: 0
      labelString: ""
      projectNamingStrategy: "standard"
      markupFormatter:
        plainText
      clouds:
      - kubernetes:
          containerCapStr: "10"
          defaultsProviderTemplate: "jenkins-base"
          connectTimeout: "5"
          readTimeout: 15
          jenkinsUrl: "jenkins-ui:8080"
          jenkinsTunnel: "jenkins-discover:50000"
          maxRequestsPerHostStr: "32"
          name: "kubernetes"
          serverUrl: "https://kubernetes"
          podLabels:
          - key: "jenkins/jenkins-agent"
            value: "true"
          templates:
            - name: "default"
          #id: eeb122dab57104444f5bf23ca29f3550fbc187b9d7a51036ea513e2a99fecf0f
              containers:
              - name: "jnlp"
                alwaysPullImage: false
                args: "^${computer.jnlpmac} ^${computer.name}"
                command: ""
                envVars:
                - envVar:
                    key: "JENKINS_URL"
                    value: "jenkins-ui:8080"
                image: "jenkins/inbound-agent:4.11-1"
                ttyEnabled: false
                workingDir: "/home/jenkins/agent"
              idleMinutes: 0
              instanceCap: 2147483647
              label: "jenkins-agent"
              nodeUsageMode: "NORMAL"
              podRetention: Never
              showRawYaml: true
              serviceAccount: "jenkins-sa"
              slaveConnectTimeoutStr: "100"
              yamlMergeStrategy: override
      crumbIssuer:
        standard:
          excludeClientIPFromCrumb: true
    security:
      apiToken:
        creationOfLegacyTokenEnabled: false
        tokenGenerationOnCreationEnabled: false
        usageStatisticsEnabled: true
    unclassified:
      location:
        adminAddress:
        url: jenkins-ui:8080
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: jenkins-pv-volume
  labels:
    type: local
spec:
  storageClassName: local-storage
  claimRef:
    name: jenkins-pv-claim
    namespace: devops-tools
  capacity:
    storage: 16Gi
  accessModes:
    - ReadWriteMany
  local:
    path: /mnt
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - heine-cluster1
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-pv-claim
  namespace: devops-tools
  labels:
    app: jenkins
    version: v1
    tier: backend
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 8Gi
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: jenkins-cr
rules:
- apiGroups: [""]
  resources: ["*"]
  verbs: ["*"]
---
# This role is used to allow Jenkins scheduling of agents via Kubernetes plugin. 
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: jenkins-role-schedule-agents
  namespace: devops-tools
  labels:
    app: jenkins
    version: v1
    tier: backend
rules:
- apiGroups: [""]
  resources: ["pods", "pods/exec", "pods/log", "persistentvolumeclaims", "events"]
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources: ["pods", "pods/exec", "persistentvolumeclaims"]
  verbs: ["create", "delete", "deletecollection", "patch", "update"]
---
# The sidecar container which is responsible for reloading configuration changes
# needs permissions to watch ConfigMaps
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: jenkins-casc-reload
  namespace: devops-tools
  labels:
    app: jenkins
    version: v1
    tier: backend
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "watch", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: jenkins-crb
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: jenkins-cr
subjects:
- kind: ServiceAccount
  name: jenkins-sa
  namespace: "devops-tools"
---
# We bind the role to the Jenkins service account. The role binding is created in the namespace
# where the agents are supposed to run.
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: jenkins-schedule-agents
  namespace: "devops-tools"
  labels:
    app: jenkins
    version: v1
    tier: backend
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: jenkins-role-schedule-agents
subjects:
- kind: ServiceAccount
  name: jenkins-sa
  namespace: "devops-tools"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: jenkins-watch-configmaps
  namespace: "devops-tools"
  labels:
    app: jenkins
    version: v1
    tier: backend
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: jenkins-casc-reload
subjects:
- kind: ServiceAccount
  name: jenkins-sa
  namespace: "devops-tools"
---
apiVersion: v1
kind: Service
metadata:
  name: jenkins
  namespace: "devops-tools"
  labels:
    app: jenkins
    version: v1
    tier: backend
  annotations:
    metallb.universe.tf/address-pool: default
spec:
  type: LoadBalancer
  loadBalancerIP: 172.16.1.5
  ports:
  - name: ui
    port: 8080
    targetPort: 8080
  externalTrafficPolicy: Local
  selector:
    app: jenkins
---
apiVersion: v1
kind: Service
metadata: 
  name: jenkins-agent
  namespace: "devops-tools"
  labels:
    app: jenkins
    version: v1
    tier: backend
spec:
  ports:
  - name: agents
    port: 50000
    targetPort: 50000
  selector:
    app: jenkins 
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins
  namespace: "devops-tools"
  labels:
    app: jenkins
    version: v1
    tier: backend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jenkins
  template:
    metadata:
      labels:
        app: jenkins
        version: v1
        tier: backend
      annotations:
        checksum/config: c0daf24e0ec4e4cb59c8a66305181a17249770b37283ca8948e189a58e29a4a5
    spec:
      securityContext:
        runAsUser: 1000
        fsGroup: 1000
        runAsNonRoot: true
      containers:
        - name: jenkins
          image: "heineza/jenkins-master:2.323-jdk11-1"
          imagePullPolicy: Always
          args: [ "--httpPort=8080"]
          env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: JAVA_OPTS
            value: -Djenkins.install.runSetupWizard=false -Dorg.apache.commons.jelly.tags.fmt.timeZone=America/Chicago
          - name: JENKINS_SLAVE_AGENT_PORT
            value: "50000"
          ports:
          - containerPort: 8080
            name: ui
          - containerPort: 50000
            name: agents
          resources:
            limits:
              cpu: 2000m
              memory: 4096Mi
            requests:
              cpu: 50m
              memory: 256Mi
          volumeMounts:
          - mountPath: /var/jenkins_home
            name: jenkins-home
            readOnly: false
          - name: jenkins-config
            mountPath: /var/jenkins_home/jenkins.yaml
          - name: admin-secret
            mountPath: /run/secrets/jenkins-admin-username
            subPath: jenkins-admin-user
            readOnly: true
          - name: admin-secret
            mountPath: /run/secrets/jenkins-admin-password
            subPath: jenkins-admin-password
            readOnly: true
      serviceAccountName: "jenkins-sa"
      volumes:
        - name: jenkins-cache
          emptyDir: {}
        - name: jenkins-home
          persistentVolumeClaim:
            claimName: jenkins-pv-claim
        - name: jenkins-config
          configMap: 
            name: jenkins
        - name: admin-secret
          secret:
            secretName: jenkins-secret

此 Jenkins list 是 Jenkins helm-chart 生成的内容的修改版本。我编辑了我的 secret ，但在实际 list 中有 base64 编码的字符串。此外，我在部署中创建和使用的 docker 镜像使用 Jenkins 2.323-jdk11 镜像作为基础镜像，并且我刚刚安装了一些用于代码配置、kubernetes 和 Git 的插件。使用 Metallb 时，是什么阻止了 Jenkins pod 在集群外部的访问？

Answer 1

默认情况下，MetalLB 不允许重复使用/共享相同的 LoadBalancerIP 地址。

根据 MetalLB documentation :

MetalLB respects the spec.loadBalancerIP parameter, so if you want your service to be set up with a specific address, you can request it by setting that parameter.

If MetalLB does not own the requested address, or if the address is already in use by another service, assignment will fail and MetalLB will log a warning event visible in kubectl describe service <service name>.[1]

如果您需要在单个 IP 上提供服务，您可以启用选择性 IP 共享。为此，您必须添加 metallb.universe.tf/allow-shared-ip服务注释。

The value of the annotation is a “sharing key.” Services can share an IP address under the following conditions:

• They both have the same sharing key.
• They request the use of different ports (e.g. tcp/80 for one and tcp/443 for the other).
• They both use the Cluster external traffic policy, or they both point to the exact same set of pods (i.e. the pod selectors are identical). [2]

---

更新

我成功地测试了您的设置，只有一点点不同 -我需要删除:externalTrafficPolicy: Local来自 Jenkins 服务规范。

试试这个方案，如果还是不行那就是你的集群环境有问题。