kubernetes - kube-registry-proxy 不会在任何节点上暴露端口 5000

Question

我正在使用 private Docker registry addon在我的 kubernetes 集群中，我想在每个节点上公开端口 5000 以轻松地从 localhost:5000 拉取图像。所以我在每个节点上放置了一个 pod list 文件 /etc/kubernetes/manifests/kube-registry-proxy.manifest 以启动端口 5000 的本地代理。当我在裸机上手动部署 kubernetes 时它起作用几个月前的 ubuntu，但是当我尝试 kargo 时失败了，端口 5000 没有监听。

我正在使用带有 calico 网络插件的 kargo，docker registry 的配置是:

kind: PersistentVolume
apiVersion: v1
metadata:
  name: kube-system-kube-registry-pv
  labels:
    kubernetes.io/cluster-service: "true"
spec:
  capacity:
    storage: 500Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  hostPath:
    path: /registry

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: kube-registry-pvc
  namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 500Gi

apiVersion: v1
kind: ReplicationController
metadata:
  name: kube-registry-v0
  namespace: kube-system
  labels:
    k8s-app: kube-registry
    version: v0
    kubernetes.io/cluster-service: "true"
spec:
  replicas: 1
  selector:
    k8s-app: kube-registry
    version: v0
  template:
    metadata:
      labels:
        k8s-app: kube-registry
        version: v0
        kubernetes.io/cluster-service: "true"
    spec:
      containers:
      - name: registry
        image: registry:2.5.1
        resources:
          # keep request = limit to keep this container in guaranteed class
          limits:
            cpu: 100m
            memory: 100Mi
          requests:
            cpu: 100m
            memory: 100Mi
        env:
        - name: REGISTRY_HTTP_ADDR
          value: :5000
        - name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
          value: /var/lib/registry
        volumeMounts:
        - name: image-store
          mountPath: /var/lib/registry
        ports:
        - containerPort: 5000
          name: registry
          protocol: TCP
      volumes:
      - name: image-store
        persistentVolumeClaim:
          claimName: kube-registry-pvc

apiVersion: v1
kind: Service
metadata:
  name: kube-registry
  namespace: kube-system
  labels:
    k8s-app: kube-registry
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: "KubeRegistry"
spec:
  selector:
    k8s-app: kube-registry
  ports:
  - name: registry
    port: 5000
    protocol: TCP

在运行 kargo 之前，我已经创建了一个 pod list 文件 /etc/kubernetes/manifests/kube-registry-proxy.manifest:

apiVersion: v1
kind: Pod
metadata:
  name: kube-registry-proxy
  namespace: kube-system
spec:
  containers:
  - name: kube-registry-proxy
    image: gcr.io/google_containers/kube-registry-proxy:0.3
    resources:
      limits:
        cpu: 100m
        memory: 50Mi
    env:
    - name: REGISTRY_HOST
      value: kube-registry.kube-system.svc.cluster.local
    - name: REGISTRY_PORT
      value: "5000"
    - name: FORWARD_PORT
      value: "5000"
    ports:
    - name: registry
      containerPort: 5000
      hostPort: 5000

kube-registry-proxy 在所有节点上运行，但没有监听端口 5000。一些输出:

ubuntu@k8s15m1:~$ kubectl get all --all-namespaces | grep registry-proxy
kube-system   po/kube-registry-proxy-k8s15m1             1/1       Running             1          1h
kube-system   po/kube-registry-proxy-k8s15m2             1/1       Running             0          1h
kube-system   po/kube-registry-proxy-k8s15s1             1/1       Running             0          1h

ubuntu@k8s15m1:~$ docker ps | grep registry
756fcf674288        gcr.io/google_containers/kube-registry-proxy:0.3      "/usr/bin/run_proxy"     19 minutes ago      Up 19 minutes                           k8s_kube-registry-proxy.bebf6da1_kube-registry-proxy-k8s15m1_kube-system_a818b22dc7210ecd31414e328ae28e43_7221833c

ubuntu@k8s15m1:~$ docker logs 756fcf674288 | tail
waiting for kube-registry.kube-system.svc.cluster.local to come online
starting proxy

ubuntu@k8s15m1:~$ netstat -ltnp | grep 5000

ubuntu@k8s15m1:~$ curl -v localhost:5000/v1/
*   Trying 127.0.0.1...
* connect to 127.0.0.1 port 5000 failed: Connection refused
* Failed to connect to localhost port 5000: Connection refused
* Closing connection 0
curl: (7) Failed to connect to localhost port 5000: Connection refused

ubuntu@k8s15m1:~$ kubectl get po kube-registry-proxy-k8s15m1 --namespace=kube-system -o wide
NAME                          READY     STATUS    RESTARTS   AGE       IP             NODE
kube-registry-proxy-k8s15m1   1/1       Running   3          1h        10.233.69.64   k8s15m1

ubuntu@k8s15m1:~$ curl -v 10.233.69.64:5000/v1/
*   Trying 10.233.69.64...
* Connected to 10.233.69.64 (10.233.69.64) port 5000 (#0)
> GET /v1/ HTTP/1.1
> Host: 10.233.69.64:5000
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Content-Type: text/plain; charset=utf-8
< Docker-Distribution-Api-Version: registry/2.0
< X-Content-Type-Options: nosniff
< Date: Tue, 14 Mar 2017 16:41:56 GMT
< Content-Length: 19
<
404 page not found
* Connection #0 to host 10.233.69.64 left intact

Answer 1

我认为这里发生了一些事情。

首先，请注意 Kubernetes Services有 3 种口味:ClusterIP (这是默认值)，NodePort (这听起来很像你期望发生的事情)，以及 LoadBalancer (我不会进一步提及，但文档会提及)。

我希望如果您更新了您的 Service明确请求 type: NodePort ，你会更接近你的想法(但请注意，除非你改变它，NodePort ports are limited to 30000-32767。

因此:


apiVersion: v1
kind: Service
metadata:
name: kube-registry
namespace: kube-system
labels:
k8s-app: kube-registry
kubernetes.io/cluster-service: "true"
kubernetes.io/name: "KubeRegistry"
spec:
type: NodePort # <--- updated line
selector:
k8s-app: kube-registry
ports:
- name: registry
port: 5000
nodePort: 30500 # <-- you can specify, or omit this
protocol: TCP


如果您对希望服务监听的端口有意见，请随意指定它，或者干脆不写，Kubernetes 会从可用空间中选择一个。

为了完整起见，我接下来要提到这件事，但我要说的是一种不好的做法，所以请不要这样做。您还可以拥有 Pods直接监听 Node 的实际 TCP/IP 堆栈, 通过 specifying hostPort ;所以在你的情况下，它将是 hostPort: 5000正下方 containerPort: 5000 ，导致 Pod表现得像个正常人 docker -p 5000:5000命令会。但是这样做会使调度 Pods一场噩梦，所以请不要。

其次，关于来自 curl 的 404 :

根据您的 curl 命令的输出，我假设 10.233.69.x是您的服务 CIDR，它解释了端口 5000 响应任何内容的原因。该请求本着正确的精神，但是/v1/是一个不正确的 URI 尝试。 Docker Registry API docs包含有关 checking it is a V2 API instance 的部分.我的最爱curl注册表的是 https://registry.example.com/v2/_catalog因为它将返回其中每个图像的名称，确保我的凭据正确，注册表服务器运行正常，等等。

我知道要接受的内容很多，所以如果您觉得我掩盖了某些内容，请告诉我，我会尽力解决。祝你好运!