docker - Gitlab CI 服务和 docker hub 身份验证

Question

由于 docker hub 上的 new restrictions 未经身份验证的拉取，您如何为 gitlab-ci 服务验证您的 docker hub 帐户？
这是来自 the gitlab documentation 的示例 CI 配置:

# from official documentation 
services:
  - postgres:12.2 # <---- this will fail at some point because it's a non-authenticated pull

variables:
  POSTGRES_DB: nice_marmot
  POSTGRES_USER: runner
  POSTGRES_PASSWORD: ""
  POSTGRES_HOST_AUTH_METHOD: trust

这会在一段时间后导致以下错误:

ERROR: Preparation failed: Error response from daemon: 
toomanyrequests: You have reached your pull rate limit. 
You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit (executor_docker.go:188:1s)

由于服务在脚本运行之前被拉取，我们不能在脚本部分使用 docker login。我在 gitlab 中找不到任何关于 url auth 或环境变量 auth 的文档。
一个理想的解决方案不需要对 gitlab-ci 服务器或 gitlab-ci 运行器的管理员访问权限，也不需要使用 pull_policy = never 设置自定义运行器(我们最终这样做了，但它通过单个运行器瓶颈大大减慢了我们的 CI e2e 测试)

Answer 1

还使用 GitLab 13.7(2020 年 12 月)检查改进的依赖代理是否可以提供帮助:

Avoid Docker rate limits and speed up your pipelines

For faster and more reliable builds, you can use the Dependency Proxy to cache the container images hosted on Docker Hub.

But, when Docker started to enforce rate limits on pull requests from Docker Hub, you noticed that even when your image was pulled from the cache, Docker counted it against your limit.
That’s because the Dependency Proxy was only caching the image’s layers (or blobs) and not the manifest, which contains information about how to build a given image.
Since the manifest is required, a pull request was still required. This also means that if Docker Hub was unavailable, you couldn’t pull your image.

Moving forward, the Dependency Proxy will cache both the image’s layers and manifest.

So, the first time you pull alpine:latest, the image will be added to the Dependency Proxy cache and count as one pull against your rate limit.
The next time you pull alpine:latest, it will be pulled from the cache, even if Docker Hub is unavailable and will not count against your rate limit.

Don’t forget, as of milestone 13.6, the Dependency Proxy is available in Core. So, give it a try and let us know what you think. Or better yet, consider contributing to one of the open issues.

See Documentation and Issue.

和:
仍然使用 GitLab 13.7(2020 年 12 月)

Use pre-defined variables with the Dependency Proxy

By proxying and caching container images from Docker Hub, the Dependency Proxy helps you to improve the performance of your pipelines.

Even though the proxy is intended to be heavily used with CI/CD, to use the feature, you had to define your own variables or hard-code values in your gitlab.ci-yml file.
This made it difficult to get started for individuals, and prevented it from being a scalable solution, especially for organizations with many different groups and projects.

Moving forward, you can use pre-defined environment variables as an intuitive way to use the Dependency Proxy. The following variables are supported:

• CI_DEPENDENCY_PROXY_USER: a CI user for logging in to the Dependency Proxy.
• CI_DEPENDENCY_PROXY_PASSWORD: a CI password for logging in to the Dependency Proxy.
• CI_DEPENDENCY_PROXY_SERVER: the server for logging in to the Dependency Proxy.
• CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX: the image prefix for pulling images through the Dependency Proxy.

Give it a try and let us know what you think!

See Documentation and Issue.

这甚至适用于 private projects(2020 年 12 月)

Use the Dependency Proxy with private projects

You can use the GitLab Dependency Proxy to proxy and cache container images from Docker Hub. Until recently the feature was only available for public groups, preventing many of you from being able to use it.

You can now use the Dependency Proxy with private projects.
You can reduce your reliance on Docker Hub by caching your container images for future use.

Because the Dependency Proxy is storing Docker images in a space associated with your group, you must authenticate with your GitLab username and password, or with your personal access token with the scope set to at least read_registry.

See Documentation and Issue.

使用 GitLab 13.9(2021 年 2 月):

Automatically authenticate when using the Dependency Proxy

By proxying and caching container images from Docker Hub, the Dependency Proxy helps you to improve the performance of your pipelines.

Even though the proxy is intended to be heavily used with CI/CD, to use the feature, you had to add your credentials to the DOCKER_AUTH_CONFIG CI/CD variable or manually run docker login in your pipeline. These solutions worked fine, but when you consider how many .gitlab-ci.yml files that you need to update, it would be better if the GitLab Runner could automatically authenticate for you.

Since the Runner is already able to automatically authenticate with the integrated GitLab Container Registry, we were able to leverage that functionality to help you automatically authenticate with the Dependency Proxy.

Now it’s easier to use the Dependency Proxy to proxy and cache your container images from Docker Hub and start having faster, more reliable builds.

See Documentation and Issue.

参见 GitLab 13.10(2021 年 3 月)

Use the Dependency Proxy with 'containerd' and Docker 20+

The GitLab Dependency Proxy is a local proxy you can use for your frequently-accessed upstream images from Docker Hub. In the case of CI/CD, the Dependency Proxy receives a request and returns the upstream image from a registry, acting as a pull-through cache. This helps to reduce your CI minutes and increase reliability.

However, you haven’t been able to pull images by digest, which as an immutable identifier ensures you are using the exact version of a specific image and tag. Since both containerd and Docker 20+ depend on pull-by-digest, this meant that many of you were blocked from using the Dependency Proxy.

We are happy to say that you can now pull your container images from Docker Hub by digest. You can use the Dependency Proxy by adding the URL to your .gitlab-ci.yml file, manually pulling the image from the command line, or using a Dockerfile. Check out the documentation and start saving time on your builds.

See Documentation and Issue.