docker - 无法从 docker build 中的工件注册表安装私有(private)依赖项

Question

我正在尝试安装一个私有(private)的 python 包，该包已上传到 docker 容器内的工件注册表(以将其部署在 cloudrun 上)。

我过去曾在云函数中成功使用过该包，因此我确信该包有效。

cloudbuild.yaml

steps:
- name: 'gcr.io/cloud-builders/docker'
  args: [ 'build', '-t', 'gcr.io/${_PROJECT}/${_SERVICE_NAME}:$SHORT_SHA', '--network=cloudbuild', '.', '--progress=plain']

Dockerfile

FROM python:3.8.6-slim-buster

ENV APP_PATH=/usr/src/app
ENV PORT=8080

# Copy requirements.txt to the docker image and install packages
RUN apt-get update && apt-get install -y cython 

RUN pip install --upgrade pip

# Set the WORKDIR to be the folder
RUN mkdir -p $APP_PATH

COPY / $APP_PATH

WORKDIR $APP_PATH

RUN pip install -r requirements.txt --no-color
RUN pip install --extra-index-url https://us-west1-python.pkg.dev/my-project/my-package/simple/ my-package==0.2.3 # This line is where the bug occurs


# Expose port 
EXPOSE $PORT

# Use gunicorn as the entrypoint
CMD exec gunicorn --bind 0.0.0.0:8080 app:app

我添加的权限是:

• cloudbuild 默认服务帐户 (project-number@cloudbuild.gserviceaccount.com):Artifact Registry Reader
• 运行 cloudbuild 的服务帐户:Artifact Registry Reader
• 运行应用程序的服务帐户:Artifact Registry Reader

云构建错误:

Step 10/12 : RUN pip install --extra-index-url https://us-west1-python.pkg.dev/my-project/my-package/simple/ my-package==0.2.3
---> Running in b2ead00ccdf4
Looking in indexes: https://pypi.org/simple, https://us-west1-python.pkg.dev/muse-speech-devops/gcp-utils/simple/
User for us-west1-python.pkg.dev: [91mERROR: Exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
status = run_func(*args)
File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
return func(self, options, args)
File "/usr/local/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 340, in run
requirement_set = resolver.resolve(
File "/usr/local/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 94, in resolve
result = self._result = resolver.resolve(
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 481, in resolve
state = resolution.resolve(requirements, max_rounds=max_rounds)
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 348, in resolve
self._add_to_criteria(self.state.criteria, r, parent=None)
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 172, in _add_to_criteria
if not criterion.candidates:
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/structs.py", line 151, in __bool__

Answer 1

从您的回溯日志中，我们可以看到 Cloud Build 没有向私有(private)存储库进行身份验证的凭据:

Step 10/12 : RUN pip install --extra-index-url https://us-west1-python.pkg.dev/my-project/my-package/simple/ my-package==0.2.3
---> Running in b2ead00ccdf4
Looking in indexes: https://pypi.org/simple, https://us-west1-python.pkg.dev/muse-speech-devops/gcp-utils/simple/
User for us-west1-python.pkg.dev: [91mERROR: Exception: //<-ASKING FOR USERNAME

我将一个简单的包上传到私有(private) Artifact Registry 存储库以在构建容器时对此进行测试，并且也收到了相同的消息。由于您似乎正在使用 service account key 进行身份验证，用户名和密码需要存储在 pip.conf 中:

pip.conf

[global]
extra-index-url = https://_json_key_base64:KEY@LOCATION-python.pkg.dev/PROJECT/REPOSITORY/simple/

因此，此文件需要在构建过程中可用。 Multi-stage docker builds 在这里非常有用，可以确保配置 key 不会暴露，因为我们可以选择将哪些文件放入最终镜像(配置 key 仅在用于从私有(private)存储库下载包时出现):

示例 Dockerfile

# Installing packages in a separate image
FROM python:3.8.6-slim-buster as pkg-build

# Target Python environment variable to bind to pip.conf
ENV PIP_CONFIG_FILE /pip.conf

WORKDIR /packages/
COPY requirements.txt /

# Copying the pip.conf key file only during package downloading
COPY ./config/pip.conf /pip.conf

# Packages are downloaded to the /packages/ directory
RUN pip download -r /requirements.txt
RUN pip download --extra-index-url https://LOCATION-python.pkg.dev/PROJECT/REPO/simple/ PACKAGES

# Final image that will be deployed
FROM python:3.8.6-slim-buster

ENV PYTHONUNBUFFERED True
ENV APP_HOME /app

WORKDIR /packages/
# Copying ONLY the packages from the previous build
COPY --from=pkg-build /packages/ /packages/

# Installing the packages from the copied files
RUN pip install --no-index --find-links=/packages/ /packages/*

WORKDIR $APP_HOME
COPY ./src/main.py ./

# Executing sample flask web app 
CMD exec gunicorn --bind :$PORT --workers 1 --threads 8 --timeout 0 main:app

我上面的 dockerfile 基于这个 related thread ，我可以确认这些包是从我的私有(private) Artifact Registry 存储库正确下载的，而且 pip.conf 文件不存在于生成的图像中。