python - 如何正确缓存 Python 依赖项

Question

我正在尝试在 Dockerfile 中构建 Python 项目。我想缓存依赖项，然后稍后使用该缓存，如下所示:

RUN pip3 download -d "/pth/to/downloaded/files" -r /temp/requirements.txt -c /temp/constraints.txt

# much later on in the Dockerfile:

RUN pip3 install --download-cache="/pth/to/downloaded/files" -r requirements.txt -c constraints.txt

---

问题:假设 pip3 download 命令是正确的，当我查看 pip3 --help 输出时，我不再看到 --download-cache 选项- 是否有一个新选项可以与 pip3 install 一起使用来引用 pip3 download 生成的依赖项缓存？

现在我收到此错误:

Usage: pip install [options]

no such option: --download-cache

Answer 1

--download-cache选项在 pip 版本 8 中被删除，因为它现在默认使用缓存。所以你根本不需要指定这个选项。我不确定 pip download -d <dir> 的目的是什么选项是，但显然它没有在目标目录中创建缓存。您可以省略 -d <dir>选项也是。以下 Dockerfile 有效:

FROM python:3.7
COPY constraints.txt requirements.txt ./
RUN pip3 download -d .pipcache -r requirements.txt -c constraints.txt
COPY test.txt ./
RUN pip3 install -r requirements.txt -c constraints.txt

如果您添加 --cache-dir <dir> 两个 download和 install命令，它也会起作用。所以下面的 Dockerfile 也可以:

FROM python:3.7
COPY constraints.txt requirements.txt ./
RUN pip3 download --cache-dir ./tmp/pipcache -r requirements.txt -c constraints.txt
COPY test.txt ./
RUN pip3 install --cache-dir ./tmp/pipcache -r requirements.txt -c constraints.txt

示例输出(pep8 中只有 pylint 和 requirements.txt):

第一次运行:

Sending build context to Docker daemon  5.632kB
Step 1/5 : FROM python:3.7
 ---> a4cc999cf2aa
Step 2/5 : COPY constraints.txt requirements.txt ./
 ---> 411eaa3d36ff
Step 3/5 : RUN pip3 download -r requirements.txt -c constraints.txt
 ---> Running in 6b489df74137
Collecting pep8==1.7.1 (from -c constraints.txt (line 17))
  Downloading https://files.pythonhosted.org/packages/42/3f/669429ce58de2c22d8d2c542752e137ec4b9885fff398d3eceb1a7f5acb4/pep8-1.7.1-py2.py3-none-any.whl (41kB)
  Saved /pep8-1.7.1-py2.py3-none-any.whl
Collecting pylint==2.3.1 (from -c constraints.txt (line 22))
  Downloading https://files.pythonhosted.org/packages/60/c2/b3f73f4ac008bef6e75bca4992f3963b3f85942e0277237721ef1c151f0d/pylint-2.3.1-py3-none-any.whl (765kB)
  Saved /pylint-2.3.1-py3-none-any.whl
Collecting mccabe==0.6.1 (from -c constraints.txt (line 14))
  Downloading https://files.pythonhosted.org/packages/87/89/479dc97e18549e21354893e4ee4ef36db1d237534982482c3681ee6e7b57/mccabe-0.6.1-py2.py3-none-any.whl
  Saved /mccabe-0.6.1-py2.py3-none-any.whl
Collecting astroid==2.2.5 (from -c constraints.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/d5/ad/7221a62a2dbce5c3b8c57fd18e1052c7331adc19b3f27f1561aa6e620db2/astroid-2.2.5-py3-none-any.whl (193kB)
  Saved /astroid-2.2.5-py3-none-any.whl
Collecting isort==4.3.19 (from -c constraints.txt (line 10))
  Downloading https://files.pythonhosted.org/packages/ae/ae/5ef4b57e15489754b73dc908b656b02ab0e6d37b190ac78dd498be8b577d/isort-4.3.19-py2.py3-none-any.whl (42kB)
  Saved /isort-4.3.19-py2.py3-none-any.whl
Collecting lazy-object-proxy==1.4.1 (from -c constraints.txt (line 12))
  Downloading https://files.pythonhosted.org/packages/43/a5/1b19b094ad19bce55b5b6d434020f5537b424fd2b3cff0fbef23d7bb5a95/lazy_object_proxy-1.4.1-cp37-cp37m-manylinux1_x86_64.whl (49kB)
  Saved /lazy_object_proxy-1.4.1-cp37-cp37m-manylinux1_x86_64.whl
Collecting wrapt==1.11.1 (from -c constraints.txt (line 39))
  Downloading https://files.pythonhosted.org/packages/67/b2/0f71ca90b0ade7fad27e3d20327c996c6252a2ffe88f50a95bba7434eda9/wrapt-1.11.1.tar.gz
  Saved /wrapt-1.11.1.tar.gz
Collecting six==1.12.0 (from -c constraints.txt (line 28))
  Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
  Saved /six-1.12.0-py2.py3-none-any.whl
Collecting typed-ast==1.3.5 (from -c constraints.txt (line 37))
  Downloading https://files.pythonhosted.org/packages/17/9e/00918af7bdd616decb5b7ad06a9cd0a4a247d2fccaa630ab448a57e68b98/typed_ast-1.3.5-cp37-cp37m-manylinux1_x86_64.whl (736kB)
  Saved /typed_ast-1.3.5-cp37-cp37m-manylinux1_x86_64.whl
Successfully downloaded pep8 pylint mccabe astroid isort lazy-object-proxy wrapt six typed-ast
Removing intermediate container 6b489df74137
 ---> 8ac3be432c58
Step 4/5 : COPY test.txt ./
 ---> 5cac20851967
Step 5/5 : RUN pip3 install -r requirements.txt -c constraints.txt
 ---> Running in 394847f09e9b
Collecting pep8==1.7.1 (from -c constraints.txt (line 17))
  Using cached https://files.pythonhosted.org/packages/42/3f/669429ce58de2c22d8d2c542752e137ec4b9885fff398d3eceb1a7f5acb4/pep8-1.7.1-py2.py3-none-any.whl
Collecting pylint==2.3.1 (from -c constraints.txt (line 22))
  Using cached https://files.pythonhosted.org/packages/60/c2/b3f73f4ac008bef6e75bca4992f3963b3f85942e0277237721ef1c151f0d/pylint-2.3.1-py3-none-any.whl
Collecting astroid==2.2.5 (from -c constraints.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/d5/ad/7221a62a2dbce5c3b8c57fd18e1052c7331adc19b3f27f1561aa6e620db2/astroid-2.2.5-py3-none-any.whl
Collecting mccabe==0.6.1 (from -c constraints.txt (line 14))
  Using cached https://files.pythonhosted.org/packages/87/89/479dc97e18549e21354893e4ee4ef36db1d237534982482c3681ee6e7b57/mccabe-0.6.1-py2.py3-none-any.whl
Collecting isort==4.3.19 (from -c constraints.txt (line 10))
  Using cached https://files.pythonhosted.org/packages/ae/ae/5ef4b57e15489754b73dc908b656b02ab0e6d37b190ac78dd498be8b577d/isort-4.3.19-py2.py3-none-any.whl
Collecting lazy-object-proxy==1.4.1 (from -c constraints.txt (line 12))
  Using cached https://files.pythonhosted.org/packages/43/a5/1b19b094ad19bce55b5b6d434020f5537b424fd2b3cff0fbef23d7bb5a95/lazy_object_proxy-1.4.1-cp37-cp37m-manylinux1_x86_64.whl
Collecting six==1.12.0 (from -c constraints.txt (line 28))
  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting wrapt==1.11.1 (from -c constraints.txt (line 39))
  Using cached https://files.pythonhosted.org/packages/67/b2/0f71ca90b0ade7fad27e3d20327c996c6252a2ffe88f50a95bba7434eda9/wrapt-1.11.1.tar.gz
Collecting typed-ast==1.3.5 (from -c constraints.txt (line 37))
  Using cached https://files.pythonhosted.org/packages/17/9e/00918af7bdd616decb5b7ad06a9cd0a4a247d2fccaa630ab448a57e68b98/typed_ast-1.3.5-cp37-cp37m-manylinux1_x86_64.whl
Building wheels for collected packages: wrapt
  Building wheel for wrapt (setup.py): started
  Building wheel for wrapt (setup.py): finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/89/67/41/63cbf0f6ac0a6156588b9587be4db5565f8c6d8ccef98202fc
Successfully built wrapt
Installing collected packages: lazy-object-proxy, six, wrapt, typed-ast, astroid, isort, mccabe, pep8, pylint
Successfully installed astroid-2.2.5 isort-4.3.19 lazy-object-proxy-1.4.1 mccabe-0.6.1 pep8-1.7.1 pylint-2.3.1 six-1.12.0 typed-ast-1.3.5 wrapt-1.11.1
Removing intermediate container 394847f09e9b
 ---> 68e65a214a32
Successfully built 68e65a214a32
Successfully tagged test:latest

第二次运行(在更改 test.txt 以触发 Docker 第 4 层和第 5 层的重建之后):

Sending build context to Docker daemon  5.632kB
Step 1/5 : FROM python:3.7
 ---> a4cc999cf2aa
Step 2/5 : COPY constraints.txt requirements.txt ./
 ---> Using cache
 ---> 411eaa3d36ff
Step 3/5 : RUN pip3 download -r requirements.txt -c constraints.txt
 ---> Using cache
 ---> 8ac3be432c58
Step 4/5 : COPY test.txt ./
 ---> 7ab5814153b7
Step 5/5 : RUN pip3 install -r requirements.txt -c constraints.txt
 ---> Running in 501da787ab07
Collecting pep8==1.7.1 (from -c constraints.txt (line 17))
  Using cached https://files.pythonhosted.org/packages/42/3f/669429ce58de2c22d8d2c542752e137ec4b9885fff398d3eceb1a7f5acb4/pep8-1.7.1-py2.py3-none-any.whl
Collecting pylint==2.3.1 (from -c constraints.txt (line 22))
  Using cached https://files.pythonhosted.org/packages/60/c2/b3f73f4ac008bef6e75bca4992f3963b3f85942e0277237721ef1c151f0d/pylint-2.3.1-py3-none-any.whl
Collecting astroid==2.2.5 (from -c constraints.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/d5/ad/7221a62a2dbce5c3b8c57fd18e1052c7331adc19b3f27f1561aa6e620db2/astroid-2.2.5-py3-none-any.whl
Collecting mccabe==0.6.1 (from -c constraints.txt (line 14))
  Using cached https://files.pythonhosted.org/packages/87/89/479dc97e18549e21354893e4ee4ef36db1d237534982482c3681ee6e7b57/mccabe-0.6.1-py2.py3-none-any.whl
Collecting isort==4.3.19 (from -c constraints.txt (line 10))
  Using cached https://files.pythonhosted.org/packages/ae/ae/5ef4b57e15489754b73dc908b656b02ab0e6d37b190ac78dd498be8b577d/isort-4.3.19-py2.py3-none-any.whl
Collecting typed-ast==1.3.5 (from -c constraints.txt (line 37))
  Using cached https://files.pythonhosted.org/packages/17/9e/00918af7bdd616decb5b7ad06a9cd0a4a247d2fccaa630ab448a57e68b98/typed_ast-1.3.5-cp37-cp37m-manylinux1_x86_64.whl
Collecting six==1.12.0 (from -c constraints.txt (line 28))
  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting wrapt==1.11.1 (from -c constraints.txt (line 39))
  Using cached https://files.pythonhosted.org/packages/67/b2/0f71ca90b0ade7fad27e3d20327c996c6252a2ffe88f50a95bba7434eda9/wrapt-1.11.1.tar.gz
Collecting lazy-object-proxy==1.4.1 (from -c constraints.txt (line 12))
  Using cached https://files.pythonhosted.org/packages/43/a5/1b19b094ad19bce55b5b6d434020f5537b424fd2b3cff0fbef23d7bb5a95/lazy_object_proxy-1.4.1-cp37-cp37m-manylinux1_x86_64.whl
Building wheels for collected packages: wrapt
  Building wheel for wrapt (setup.py): started
  Building wheel for wrapt (setup.py): finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/89/67/41/63cbf0f6ac0a6156588b9587be4db5565f8c6d8ccef98202fc
Successfully built wrapt
Installing collected packages: typed-ast, six, wrapt, lazy-object-proxy, astroid, isort, mccabe, pep8, pylint
Successfully installed astroid-2.2.5 isort-4.3.19 lazy-object-proxy-1.4.1 mccabe-0.6.1 pep8-1.7.1 pylint-2.3.1 six-1.12.0 typed-ast-1.3.5 wrapt-1.11.1
Removing intermediate container 501da787ab07
 ---> b377fe561e97
Successfully built b377fe561e97
Successfully tagged test:latest

注意:official documentation很有帮助。