elasticsearch - java.io.IOException:启动k8s elasticsearch集群pod时无法在/usr/share/elasticsearch/data/nodes/0上获得锁

Question

当我在kuberntes(v1.15.2)中启动我的elasticsearch集群的pod时，pods日志显示以下错误:

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2020-03-01T05:43:27,304][INFO ][o.e.n.Node               ] [es-cluster-2] initializing ...
[2020-03-01T05:43:27,484][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [es-cluster-2] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data/k8s-logs]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.4.3.jar:6.4.3]
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.4.3.jar:6.4.3]
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.4.3.jar:6.4.3]
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.4.3.jar:6.4.3]
    at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.4.3.jar:6.4.3]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.4.3.jar:6.4.3]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.4.3.jar:6.4.3]
Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data/k8s-logs]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
    at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:247) ~[elasticsearch-6.4.3.jar:6.4.3]
    at org.elasticsearch.node.Node.<init>(Node.java:274) ~[elasticsearch-6.4.3.jar:6.4.3]
    at org.elasticsearch.node.Node.<init>(Node.java:256) ~[elasticsearch-6.4.3.jar:6.4.3]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.4.3.jar:6.4.3]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.4.3.jar:6.4.3]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.4.3.jar:6.4.3]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.4.3.jar:6.4.3]
    ... 6 more
Caused by: java.io.IOException: failed to obtain lock on /usr/share/elasticsearch/data/nodes/0
    at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:226) ~[elasticsearch-6.4.3.jar:6.4.3]
    at org.elasticsearch.node.Node.<init>(Node.java:274) ~[elasticsearch-6.4.3.jar:6.4.3]
    at org.elasticsearch.node.Node.<init>(Node.java:256) ~[elasticsearch-6.4.3.jar:6.4.3]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.4.3.jar:6.4.3]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.4.3.jar:6.4.3]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.4.3.jar:6.4.3]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.4.3.jar:6.4.3]
    ... 6 more
Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes/0/node.lock
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
    at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:178) ~[?:?]
    at java.nio.channels.FileChannel.open(FileChannel.java:292) ~[?:?]
    at java.nio.channels.FileChannel.open(FileChannel.java:340) ~[?:?]
    at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:125) ~[lucene-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:51:45]

我的Pod按状态集进行部署，并安装在NFS文件系统上。这是我的状态Yaml:

apiVersion: v1
items:
- apiVersion: apps/v1
  kind: StatefulSet
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"apps/v1","kind":"StatefulSet","metadata":{"annotations":{},"name":"es-cluster","namespace":"logging"},"spec":{"replicas":1,"selector":{"matchLabels":{"app":"elasticsearch"}},"serviceName":"elasticsearch","template":{"metadata":{"labels":{"app":"elasticsearch"}},"spec":{"containers":[{"env":[{"name":"cluster.name","value":"k8s-logs"},{"name":"node.name","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"discovery.zen.ping.unicast.hosts","value":"es-cluster-0.elasticsearch,es-cluster-1.elasticsearch,es-cluster-2.elasticsearch"},{"name":"discovery.zen.minimum_master_nodes","value":"2"},{"name":"ES_JAVA_OPTS","value":"-Xms512m -Xmx512m"}],"image":"registry.cn-hangzhou.aliyuncs.com/dabai_app_k8s/dabai_fat/elasticsearch-oss:6.4.3","name":"elasticsearch","ports":[{"containerPort":9200,"name":"rest","protocol":"TCP"},{"containerPort":9300,"name":"inter-node","protocol":"TCP"}],"resources":{"limits":{"cpu":"1000m"},"requests":{"cpu":"100m"}},"volumeMounts":[{"mountPath":"/usr/share/elasticsearch/data","name":"data"}]}],"imagePullSecrets":[{"name":"regcred"}],"initContainers":[{"command":["sh","-c","chown -R 1000:1000 /usr/share/elasticsearch/data"],"image":"docker.io/library/busybox:1.31.1","name":"fix-permissions","securityContext":{"privileged":true},"volumeMounts":[{"mountPath":"/usr/share/elasticsearch/data","name":"data"}]},{"command":["sysctl","-w","vm.max_map_count=262144"],"image":"docker.io/library/busybox:1.31.1","name":"increase-vm-max-map","securityContext":{"privileged":true}},{"command":["sh","-c","ulimit -n 65536"],"image":"docker.io/library/busybox:1.31.1","name":"increase-fd-ulimit","securityContext":{"privileged":true}}]}},"volumeClaimTemplates":[{"metadata":{"labels":{"app":"elasticsearch"},"name":"data"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"5Gi"}},"storageClassName":"es-data-db"}}]}}
    creationTimestamp: "2020-02-28T06:28:48Z"
    generation: 13
    labels:
      app: elasticsearch
    name: es-cluster
    namespace: logging
    resourceVersion: "17199681"
    selfLink: /apis/apps/v1/namespaces/logging/statefulsets/es-cluster
    uid: 06afbdc1-3ada-401e-8e62-74c6ce4aba97
  spec:
    podManagementPolicy: OrderedReady
    replicas: 3
    revisionHistoryLimit: 10
    selector:
      matchLabels:
        app: elasticsearch
    serviceName: elasticsearch
    template:
      metadata:
        creationTimestamp: null
        labels:
          app: elasticsearch
      spec:
        containers:
        - env:
          - name: cluster.name
            value: k8s-logs
          - name: node.name
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.name
          - name: discovery.zen.ping.unicast.hosts
            value: es-cluster-0.elasticsearch,es-cluster-1.elasticsearch,es-cluster-2.elasticsearch
          - name: discovery.zen.minimum_master_nodes
            value: "2"
          - name: ES_JAVA_OPTS
            value: -Xms512m -Xmx512m
          image: registry.cn-hangzhou.aliyuncs.com/dabai_app_k8s/dabai_fat/elasticsearch-oss:6.4.3
          imagePullPolicy: IfNotPresent
          name: elasticsearch
          ports:
          - containerPort: 9200
            name: rest
            protocol: TCP
          - containerPort: 9300
            name: inter-node
            protocol: TCP
          resources:
            limits:
              cpu: "1"
            requests:
              cpu: 100m
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /usr/share/elasticsearch/data
            name: data
        dnsPolicy: ClusterFirst
        imagePullSecrets:
        - name: regcred
        initContainers:
        - command:
          - sh
          - -c
          - chown -R 1000:1000 /usr/share/elasticsearch/data
          image: docker.io/library/busybox:1.31.1
          imagePullPolicy: IfNotPresent
          name: fix-permissions
          resources: {}
          securityContext:
            privileged: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /usr/share/elasticsearch/data
            name: data
        - command:
          - sysctl
          - -w
          - vm.max_map_count=262144
          image: docker.io/library/busybox:1.31.1
          imagePullPolicy: IfNotPresent
          name: increase-vm-max-map
          resources: {}
          securityContext:
            privileged: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
        - command:
          - sh
          - -c
          - ulimit -n 65536
          image: docker.io/library/busybox:1.31.1
          imagePullPolicy: IfNotPresent
          name: increase-fd-ulimit
          resources: {}
          securityContext:
            privileged: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
        restartPolicy: Always
        schedulerName: default-scheduler
        securityContext: {}
        terminationGracePeriodSeconds: 30
    updateStrategy:
      rollingUpdate:
        partition: 0
      type: RollingUpdate
    volumeClaimTemplates:
    - metadata:
        creationTimestamp: null
        labels:
          app: elasticsearch
        name: data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 5Gi
        storageClassName: es-data-db
        volumeMode: Filesystem
      status:
        phase: Pending
  status:
    collisionCount: 0
    currentReplicas: 3
    currentRevision: es-cluster-76d4866565
    observedGeneration: 13
    readyReplicas: 2
    replicas: 3
    updateRevision: es-cluster-76d4866565
    updatedReplicas: 3
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

我启动了3个Pod，每个Pod都有其自己的数据文件夹(分别由3 PV和3 PVC装入)。不可能使多节点读写一个数据文件夹。那么如何发生？我应该更改NFS文件夹权限吗？

Answer 1

根据此issue的说法，不正式支持使用NFS存储运行Elasticsearch，因为存在NFS不支持文件锁定，操作性能等方面的已知问题。但是请参阅此guide，以帮助您。