Jenkins helm chart 不更新插件

Question

我在使用 jenkins kubernetes pod 时遇到问题，它在最后一次 pod 重启(由 kubernetes 完成)后停止工作。

所以，我的日志中出现了这样的错误:

2021-02-05 11:00:55.856+0000 [id=27]    INFO    jenkins.InitReactorRunner$1#onAttained: Listed all plugins
2021-02-05 11:00:56.883+0000 [id=30]    SEVERE  jenkins.InitReactorRunner$1#onTaskFailed: Failed Loading plugin Pipeline: Multibranch v2.22 (workflow-multibranch)
java.io.IOException: Failed to load: Pipeline: Multibranch (2.22)
 - Update required: Pipeline: Job (2.36) to be updated to 2.39 or higher
    at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:952)
    at hudson.PluginManager$2$1$1.run(PluginManager.java:549)
    at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
    at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
    at jenkins.model.Jenkins$5.runTask(Jenkins.java:1131)
    at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
    at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

我看到我应该将 workflow-job 插件的版本升级到 2.39。

Jenkins 由 helm 管理。如果我从 S3 下载最新的 helm chart，我可以看到其中有一个文件夹 jenkins，我可以在其中看到 jenkins/values.yaml，如下所示:

# Default values for jenkins.
# This is a YAML-formatted file.
# Declare name/value pairs to be passed into your templates.
# name: value

## Overrides for generated resource names
# See templates/_helpers.tpl
# nameOverride:
# fullnameOverride:

Master:
  Name: jenkins-master
  Image: "jenkins/jenkins"
  ImageTag: "lts"
  ImagePullPolicy: "Always"
# ImagePullSecret: jenkins
  Component: "jenkins-master"
  NumExecutors: 0
  # configAutoReload requires UseSecurity is set to true:
  UseSecurity: true
  # SecurityRealm:
  # Optionally configure a different AuthorizationStrategy using Jenkins XML
  # AuthorizationStrategy: |-
  #    <authorizationStrategy class="hudson.security.FullControlOnceLoggedInAuthorizationStrategy">
  #      <denyAnonymousReadAccess>true</denyAnonymousReadAccess>
  #    </authorizationStrategy>
  HostNetworking: false
  # When enabling LDAP or another non-Jenkins identity source, the built-in admin account will no longer exist.
  # Since the AdminUser is used by configAutoReload, in order to use configAutoReload you must change the
  # .Master.AdminUser to a valid username on your LDAP (or other) server.  This user does not need
  # to have administrator rights in Jenkins (the default Overall:Read is sufficient) nor will it be granted any
  # additional rights.  Failure to do this will cause the sidecar container to fail to authenticate via SSH and enter
  # a restart loop.  Likewise if you disable the non-Jenkins identity store and instead use the Jenkins internal one,
  # you should revert Master.AdminUser to your preferred admin user:
  AdminUser: admin
  # AdminPassword: <defaults to random>
  OwnSshKey: false
  # If CasC auto-reload is enabled, an SSH (RSA) keypair is needed.  Can either provide your own, or leave unconfigured\false to allow a random key to be auto-generated.
  # If you choose to use your own, you must upload your decrypted RSA private key (not the public key above) to a Kubernetes secret using the following command:
  # kubectl -n <namespace> create secret generic <helm_release_name> --dry-run --from-file=jenkins-admin-private-key=~/.ssh/id_rsa -o yaml |kubectl -n <namespace> apply -f -
  # Replace ~/.ssh/id_rsa in the above command with the path to your private key file and the <helm_release_name> and <namespace> placeholders to suit.
  RollingUpdate: {}
  # Ignored if Persistence is enabled
  # maxSurge: 1
  # maxUnavailable: 25%
  resources:
    requests:
      cpu: "50m"
      memory: "256Mi"
    limits:
      cpu: "2000m"
      memory: "4096Mi"
  # Environment variables that get added to the init container (useful for e.g. http_proxy)
  # InitContainerEnv:
  #   - name: http_proxy
  #     value: "http://192.168.64.1:3128"
  # ContainerEnv:
  #   - name: http_proxy
  #     value: "http://192.168.64.1:3128"
  # Set min/max heap here if needed with:
  # JavaOpts: "-Xms512m -Xmx512m"
  # JenkinsOpts: ""
  # JenkinsUrl: ""
  # If you set this prefix and use ingress controller then you might want to set the ingress path below
  # JenkinsUriPrefix: "/jenkins"
  # Enable pod security context (must be `true` if RunAsUser or FsGroup are set)
  UsePodSecurityContext: true
  # Set RunAsUser to 1000 to let Jenkins run as non-root user 'jenkins' which exists in 'jenkins/jenkins' docker image.
  # When setting RunAsUser to a different value than 0 also set FsGroup to the same value:
  # RunAsUser: <defaults to 0>
  # FsGroup: <will be omitted in deployment if RunAsUser is 0>
  ServicePort: 8080
  # For minikube, set this to NodePort, elsewhere use LoadBalancer
  # Use ClusterIP if your setup includes ingress controller
  ServiceType: LoadBalancer
  # Master Service annotations
  ServiceAnnotations: {}
  # Master Custom Labels
  DeploymentLabels: {}
  #   foo: bar
  #   bar: foo
  # Master Service Labels
  ServiceLabels: {}
  #   service.beta.kubernetes.io/aws-load-balancer-backend-protocol: https
  # Put labels on jeknins-master pod
  PodLabels: {}
  # Used to create Ingress record (should used with ServiceType: ClusterIP)
  # HostName: jenkins.cluster.local
  # NodePort: <to set explicitly, choose port between 30000-32767
  # Enable Kubernetes Liveness and Readiness Probes
  # ~ 2 minutes to allow Jenkins to restart when upgrading plugins. Set ReadinessTimeout to be shorter than LivenessTimeout.
  HealthProbes: true
  HealthProbesLivenessTimeout: 90
  HealthProbesReadinessTimeout: 60
  HealthProbeReadinessPeriodSeconds: 10
  HealthProbeLivenessFailureThreshold: 12
  SlaveListenerPort: 50000
#  SlaveHostPort: 50000
  DisabledAgentProtocols:
    - JNLP-connect
    - JNLP2-connect
  CSRF:
    DefaultCrumbIssuer:
      Enabled: true
      ProxyCompatability: true
  CLI: false
  # Kubernetes service type for the JNLP slave service
  # SlaveListenerServiceType is the Kubernetes Service type for the JNLP slave service,
  # either 'LoadBalancer', 'NodePort', or 'ClusterIP'
  # Note if you set this to 'LoadBalancer', you *must* define annotations to secure it. By default
  # this will be an external load balancer and allowing inbound 0.0.0.0/0, a HUGE
  # security risk:  https://github.com/kubernetes/charts/issues/1341
  SlaveListenerServiceType: ClusterIP
  SlaveListenerServiceAnnotations: {}

  # Example of 'LoadBalancer' type of slave listener with annotations securing it
  # SlaveListenerServiceType: LoadBalancer
  # SlaveListenerServiceAnnotations:
  #   service.beta.kubernetes.io/aws-load-balancer-internal: "True"
  #   service.beta.kubernetes.io/load-balancer-source-ranges: "172.0.0.0/8, 10.0.0.0/8"

  # LoadBalancerSourcesRange is a list of allowed CIDR values, which are combined with ServicePort to
  # set allowed inbound rules on the security group assigned to the master load balancer
  LoadBalancerSourceRanges:
  - 0.0.0.0/0
  # Optionally assign a known public LB IP
  # LoadBalancerIP: 1.2.3.4
  # Optionally configure a JMX port
  # requires additional JavaOpts, ie
  # JavaOpts: >
  #   -Dcom.sun.management.jmxremote.port=4000
  #   -Dcom.sun.management.jmxremote.authenticate=false
  #   -Dcom.sun.management.jmxremote.ssl=false
  # JMXPort: 4000
  # Optionally configure other ports to expose in the Master container
  ExtraPorts:
  # - name: BuildInfoProxy
  #   port: 9000

  # List of plugins to be install during Jenkins master start
  OverwritePlugins: true
  InstallPlugins:
    - kubernetes:1.16.1
    - workflow-job:2.39
    - workflow-aggregator:2.6
    - workflow-basic-steps:2.18
    - credentials-binding:1.23
    - job-dsl:1.76
    - git:4.2.2
    - parameterized-trigger:2.35.2
    - slack:2.34
    - global-slack-notifier:1.5
    - ansicolor:0.6.2
    - simple-theme-plugin:0.5.1
    - aws-bucket-credentials:1.0.0
    - aws-credentials:1.28
    - ssh-agent:1.17
    #      - blueocean:1.21.0
    - basic-branch-build-strategies:1.3.2
    - buildtriggerbadge:2.10
    - rebuild:1.31
    - ghprb:1.42.0
    - antisamy-markup-formatter:1.5
    - github-oauth:0.31
    - role-strategy:2.15

  # Enable to always override the installed plugins with the values of 'Master.InstallPlugins' on upgrade or redeployment.
  # OverwritePlugins: true
  # Enable HTML parsing using OWASP Markup Formatter Plugin (antisamy-markup-formatter), useful with ghprb plugin.
  # The plugin is not installed by default, please update Master.InstallPlugins.
  # EnableRawHtmlMarkupFormatter: true
  # Used to approve a list of groovy functions in pipelines used the script-security plugin. Can be viewed under /scriptApproval
  # ScriptApproval:
  #   - "method groovy.json.JsonSlurperClassic parseText java.lang.String"
  #   - "new groovy.json.JsonSlurperClassic"
  # List of groovy init scripts to be executed during Jenkins master start
  InitScripts:
  #  - |
  #    print 'adding global pipeline libraries, register properties, bootstrap jobs...'
  # Kubernetes secret that contains a 'credentials.xml' for Jenkins
  # CredentialsXmlSecret: jenkins-credentials
  # Kubernetes secret that contains files to be put in the Jenkins 'secrets' directory,
  # useful to manage encryption keys used for credentials.xml for instance (such as
  # master.key and hudson.util.Secret)
  # SecretsFilesSecret: jenkins-secrets
  # Jenkins XML job configs to provision
  # Jobs:
  #   test: |-
  #     <<xml here>>

  # Below is the implementation of Jenkins Configuration as Code.  Add a key under ConfigScripts for each configuration area,
  # where each corresponds to a plugin or section of the UI.  Each key (prior to | character) is just a label, and can be any value.
  # Keys are only used to give the section a meaningful name.  The only restriction is they may only contain RFC 1123 \ DNS label
  # characters: lowercase letters, numbers, and hyphens.  The keys become the name of a configuration yaml file on the master in
  # /var/jenkins_home/casc_configs (by default) and will be processed by the Configuration as Code Plugin.  The lines after each |
  # become the content of the configuration yaml file.  The first line after this is a JCasC root element, eg jenkins, credentials,
  # etc.  Best reference is https://<jenkins_url>/configuration-as-code/reference.  The example below creates a welcome message:
  JCasC:
    enabled: false
    PluginVersion: 1.5
    SupportPluginVersion: 1.5
    ConfigScripts:
      welcome-message: |
        jenkins:
          systemMessage: Welcome to our CI\CD server.  This Jenkins is configured and managed 'as code'.

  Sidecars:
    configAutoReload:
      # If enabled: true, Jenkins Configuration as Code will be reloaded on-the-fly without a reboot.  If false or not-specified,
      # jcasc changes will cause a reboot and will only be applied at the subsequent start-up.  Auto-reload uses the Jenkins CLI
      # over SSH to reapply config when changes to the ConfigScripts are detected.  The admin user (or account you specify in
      # Master.AdminUser) will have a random SSH private key (RSA 4096) assigned unless you specify OwnSshKey: true.  This will be saved to a k8s secret.
      enabled: false
      image: shadwell/k8s-sidecar:0.0.2
      imagePullPolicy: IfNotPresent
      resources:
        #   limits:
        #     cpu: 100m
        #     memory: 100Mi
        #   requests:
        #     cpu: 50m
        #     memory: 50Mi
      # SSH port value can be set to any unused TCP port.  The default, 1044, is a non-standard SSH port that has been chosen at random.
      # Is only used to reload jcasc config from the sidecar container running in the Jenkins master pod.
      # This TCP port will not be open in the pod (unless you specifically configure this), so Jenkins will not be
      # accessible via SSH from outside of the pod.  Note if you use non-root pod privileges (RunAsUser & FsGroup),
      # this must be > 1024:
      sshTcpPort: 1044
      # label that the configmaps with dashboards are marked with:
      label: jenkins_config
      # folder in the pod that should hold the collected dashboards:
      folder: /var/jenkins_home/casc_configs
      # If specified, the sidecar will search for dashboard config-maps inside this namespace.
      # Otherwise the namespace in which the sidecar is running will be used.
      # It's also possible to specify ALL to search in all namespaces:
      # searchNamespace:

    # Allows you to inject additional/other sidecars
    other:
    ## The example below runs the client for https://smee.io as sidecar container next to Jenkins,
    ## that allows to trigger build behind a secure firewall.
    ## https://jenkins.io/blog/2019/01/07/webhook-firewalls/#triggering-builds-with-webhooks-behind-a-secure-firewall
    ##
    ## Note: To use it you should go to https://smee.io/new and update the url to the generete one.
    # - name: smee
    #   image: docker.io/twalter/smee-client:1.0.2
    #   args: ["--port", "{{ .Values.Master.ServicePort }}", "--path", "/github-webhook/", "--url", "https://smee.io/new"]
    #   resources:
    #     limits:
    #       cpu: 50m
    #       memory: 128Mi
    #     requests:
    #       cpu: 10m
    #       memory: 32Mi
  # Node labels and tolerations for pod assignment
  # ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector
  # ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#taints-and-tolerations-beta-feature
  NodeSelector: {}
  Tolerations: {}
  PodAnnotations: {}

  # The below two configuration-related values are deprecated and replaced by Jenkins Configuration as Code (see above
  # JCasC key).  They will be deleted in an upcoming version.
  CustomConfigMap: false
  # By default, the configMap is only used to set the initial config the first time
  # that the chart is installed.  Setting `OverwriteConfig` to `true` will overwrite
  # the jenkins config with the contents of the configMap every time the pod starts.
  # This will also overwrite all init scripts
  OverwriteConfig: false

  ingress:
    enabled: false
    labels: {}
    annotations: {}
    # kubernetes.io/ingress.class: nginx
    # kubernetes.io/tls-acme: "true"
    # Set this path to JenkinsUriPrefix above or use annotations to rewrite path
    # path: "/jenkins"
    tls:
    # - secretName: jenkins.cluster.local
    #   hosts:
    #     - jenkins.cluster.local
  AdditionalConfig: {}

  # Master.HostAliases allows for adding entries to Pod /etc/hosts:
  # https://kubernetes.io/docs/concepts/services-networking/add-entries-to-pod-etc-hosts-with-host-aliases/
  HostAliases: []
  # - ip: 192.168.50.50
  #   hostnames:
  #     - something.local
  # - ip: 10.0.50.50
  #   hostnames:
  #     - other.local

Agent:
  Enabled: true
  Image: jenkins/jnlp-slave
  ImageTag: 3.27-1
  CustomJenkinsLabels: []
# ImagePullSecret: jenkins
  Component: "jenkins-slave"
  Privileged: false
  resources:
    requests:
      cpu: "200m"
      memory: "256Mi"
    limits:
      cpu: "200m"
      memory: "256Mi"
  # You may want to change this to true while testing a new image
  AlwaysPullImage: false
  # Controls how slave pods are retained after the Jenkins build completes
  # Possible values: Always, Never, OnFailure
  PodRetention: Never
  # You can define the volumes that you want to mount for this container
  # Allowed types are: ConfigMap, EmptyDir, HostPath, Nfs, Pod, Secret
  # Configure the attributes as they appear in the corresponding Java class for that type
  # https://github.com/jenkinsci/kubernetes-plugin/tree/master/src/main/java/org/csanchez/jenkins/plugins/kubernetes/volumes
  # Pod-wide ennvironment, these vars are visible to any container in the slave pod
  envVars:
  # - name: PATH
  #   value: /usr/local/bin
  volumes:
  # - type: Secret
  #   secretName: mysecret
  #   mountPath: /var/myapp/mysecret
  NodeSelector: {}
  # Key Value selectors. Ex:
  # jenkins-agent: v1

  # Executed command when side container gets started
  Command:
  Args:
  # Side container name
  SideContainerName: jnlp
  # Doesn't allocate pseudo TTY by default
  TTYEnabled: false
  # Max number of spawned agent
  ContainerCap: 10
  # Pod name
  PodName: default

Persistence:
  Enabled: true
  ## A manually managed Persistent Volume and Claim
  ## Requires Persistence.Enabled: true
  ## If defined, PVC must be created manually before volume will be bound
  # ExistingClaim:
  ## jenkins data Persistent Volume Storage Class
  ## If defined, storageClassName: <storageClass>
  ## If set to "-", storageClassName: "", which disables dynamic provisioning
  ## If undefined (the default) or set to null, no storageClassName spec is
  ##   set, choosing the default provisioner.  (gp2 on AWS, standard on
  ##   GKE, AWS & OpenStack)
  ##
  # StorageClass: "-"
  Annotations: {}
  AccessMode: ReadWriteOnce
  Size: 8Gi
  volumes:
  #  - name: nothing
  #    emptyDir: {}
  mounts:
  #  - mountPath: /var/nothing
  #    name: nothing
  #    readOnly: true

NetworkPolicy:
  # Enable creation of NetworkPolicy resources.
  Enabled: false
  # For Kubernetes v1.4, v1.5 and v1.6, use 'extensions/v1beta1'
  # For Kubernetes v1.7, use 'networking.k8s.io/v1'
  ApiVersion: networking.k8s.io/v1

## Install Default RBAC roles and bindings
rbac:
  install: false
  serviceAccountName: default
  # Role reference
  roleRef: cluster-admin
  # Role kind (Role or ClusterRole)
  roleKind: ClusterRole
  # Role binding kind (RoleBinding or ClusterRoleBinding)
  roleBindingKind: ClusterRoleBinding

## Backup cronjob configuration
## Ref: https://github.com/nuvo/kube-tasks
backup:
  # Backup must use RBAC
  # So by enabling backup you are enabling RBAC specific for backup
  enabled: false
  # Schedule to run jobs. Must be in cron time format
  # Ref: https://crontab.guru/
  schedule: "0 2 * * *"
  annotations:
    # Example for authorization to AWS S3 using kube2iam
    # Can also be done using environment variables
    iam.amazonaws.com/role: jenkins
  image:
    repository: nuvo/kube-tasks
    tag: 0.1.2
  # Additional arguments for kube-tasks
  # Ref: https://github.com/nuvo/kube-tasks#simple-backup
  extraArgs: []
  # Add additional environment variables
  env:
  # Example environment variable required for AWS credentials chain
  - name: AWS_REGION
    value: us-east-1
  resources:
    requests:
      memory: 1Gi
      cpu: 1
    limits:
      memory: 1Gi
      cpu: 1
  # Destination to store the backup artifacts
  # Supported cloud storage services: AWS S3, Minio S3, Azure Blob Storage
  # Additional support can added. Visit this repository for details
  # Ref: https://github.com/nuvo/skbn
  destination: s3://nuvo-jenkins-data/backup

我的 requirements.yaml 文件在图表中看起来像

dependencies:
  - name: jenkins
    version: 0.35.1
    repository: https://kubernetes-charts.storage.googleapis.com/
  - name: istio-components
    version: 0.4.1
    repository: s3://helm-chart-repository/

我尝试在 InstallPlugins 下更新插件版本并推送 helm chart 并列出它(我可以看到它已经更新)，但它对错误没有影响，就像插件没有更新在全部。

关于如何继续调试这个问题有什么建议吗？

Answer 1

我能够通过删除/var/jenkins_home 下的/plugins 文件夹然后重新启动我的 Jenkins pod 来解决这个问题。这创建了一个新的/plugins 文件夹，然后我不再有任何插件依赖性问题。