kubernetes - 使用 DaemonSet 运行 glusterfs 集群

Question

我一直在尝试使用以下方法在我的 kubernetes 集群上运行 glusterfs 集群:
glusterfs-service.json

{
  "kind": "Service",
  "apiVersion": "v1",
  "metadata": {
    "name": "glusterfs-cluster"
  },
  "spec": {
    "type": "NodePort",
    "selector": {
      "name": "gluster"
    },
    "ports": [
      {
        "port": 1
      }
    ]
  }
}

和 glusterfs-server.json :

{
  "apiVersion": "extensions/v1beta1",
  "kind": "DaemonSet",
  "metadata": {
    "labels": {
      "name": "gluster"
    },
    "name": "gluster"
  },
  "spec": {
    "selector": {
      "matchLabels": {
        "name": "gluster"
      }
    },
    "template": {
      "metadata": {
        "labels": {
          "name": "gluster"
        }
      },
      "spec": {
        "containers": [
          {
            "name": "gluster",
            "image": "gluster/gluster-centos",
            "livenessProbe": {
              "exec": {
                "command": [
                  "/bin/bash",
                  "-c",
                  "systemctl status glusterd.service"
                ]
              }
            },
            "readinessProbe": {
              "exec": {
                "command": [
                  "/bin/bash",
                  "-c",
                  "systemctl status glusterd.service"
                ]
              }
            },
            "securityContext": {
              "privileged": true
            },
            "volumeMounts": [
              {
                "mountPath": "/mnt/brick1",
                "name": "gluster-brick"
              },
              {
                "mountPath": "/etc/gluster",
                "name": "gluster-etc"
              },
              {
                "mountPath": "/var/log/gluster",
                "name": "gluster-logs"
              },
              {
                "mountPath": "/var/lib/glusterd",
                "name": "gluster-config"
              },
              {
                "mountPath": "/dev",
                "name": "gluster-dev"
              },
              {
                "mountPath": "/sys/fs/cgroup",
                "name": "gluster-cgroup"
              }
            ]
          }
        ],
        "dnsPolicy": "ClusterFirst",
        "hostNetwork": true,
        "volumes": [
          {
            "hostPath": {
              "path": "/mnt/brick1"
            },
            "name": "gluster-brick"
          },
          {
            "hostPath": {
              "path": "/etc/gluster"
            },
            "name": "gluster-etc"
          },
          {
            "hostPath": {
              "path": "/var/log/gluster"
            },
            "name": "gluster-logs"
          },
          {
            "hostPath": {
              "path": "/var/lib/glusterd"
            },
            "name": "gluster-config"
          },
          {
            "hostPath": {
              "path": "/dev"
            },
            "name": "gluster-dev"
          },
          {
            "hostPath": {
              "path": "/sys/fs/cgroup"
            },
            "name": "gluster-cgroup"
          }
        ]
      }
    }
  }
}

然后在我的 pod 定义上，我正在做:

"volumes": [
  {
    "name": "< volume name >",
    "glusterfs": {
      "endpoints": "glusterfs-cluster.default.svc.cluster.local",
      "path": "< gluster path >",
      "readOnly": false
    }
  }
]

但是 pod 创建超时，因为它无法加载卷

看起来也只有一个 glusterfs pod 正在运行

这是我的日志:
http://imgur.com/a/j2I8r

然后我尝试在我的 gluster 集群运行的同一个命名空间上运行我的 pod，我现在收到这个错误:

Operation for "\"kubernetes.io/glusterfs/01a0834e-64ab-11e6-af52-42010a840072-ssl-certificates\" (\"01a0834e-64ab-11e6-af52-42010a840072\")" failed.
No retries permitted until 2016-08-17 18:51:20.61133778 +0000 UTC (durationBeforeRetry 2m0s).
Error: MountVolume.SetUp failed for volume "kubernetes.io/glusterfs/01a0834e-64ab-11e6-af52-42010a840072-ssl-certificates" (spec.Name: "ssl-certificates") pod "01a0834e-64ab-11e6-af52-42010a840072" (UID: "01a0834e-64ab-11e6-af52-42010a840072") with: glusterfs: mount failed:
mount failed: exit status 1
Mounting arguments:
10.132.0.7:ssl_certificates /var/lib/kubelet/pods/01a0834e-64ab-11e6-af52-42010a840072/volumes/kubernetes.io~glusterfs/ssl-certificates
glusterfs [log-level=ERROR log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/ssl-certificates/caddy-server-1648321103-epvdi-glusterfs.log]
Output: Mount failed. Please check the log file for more details. the following error information was pulled from the glusterfs log to help diagnose this issue:
[2016-08-17 18:49:20.583585] E [glusterfsd-mgmt.c:1596:mgmt_getspec_cbk] 0-mgmt: failed to fetch volume file (key:ssl_certificates)
[2016-08-17 18:49:20.610531] E [glusterfsd-mgmt.c:1494:mgmt_getspec_cbk] 0-glusterfs: failed to get the 'volume file' from server

Answer 1

日志清楚地说明了发生了什么:

failed to get endpoints glusterfs-cluster [endpoints "glusterfs-cluster" not found]

因为:

  "ports": [
  {
    "port": 1
  }

在几个方面是假的。首先，端口“1”非常可疑。其次，它没有匹配的 containerPort:在 DaemonSet kubernetes 可以指向的一侧 Service -- 因此，它不会创建 Endpoints为 (podIP, protocol, port)元组。因为 glusterfs(合理地)想要联系底层 Pod s 直接，无需通过 Service ，则无法发现 Pod s，一切都戛然而止。