python - 如何在 Google App Engine 上安装 'python3-dev' 以使用 'pymatting_aot.aot'

Question

我正在尝试在使用 pymatting_aot.aot 的 Google App Engine 上运行 Python 3.8 脚本

在我的本地 Ubuntu 计算机上运行此脚本时，它可以正常工作，没有任何问题，但当我尝试在 GAE 上运行它时，出现以下错误:

Traceback (most recent call last):
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/pymatting_aot/cc.py", line 21, in <module>
    import pymatting_aot.aot
ModuleNotFoundError: No module named 'pymatting_aot.aot'

此github链接:https://github.com/danielgatis/rembg/issues/35 ，表明我需要安装 python3-dev 来通过 apt-get install -y python3-dev

修复问题

我在我的 GAE 控制台上尝试了 sudo apt-get install -y python3-dev 并得到了:

machine is ephemeral and no system-wide change will persist beyond session end.

To suppress this warning, create an empty ~/.cloudshell/no-apt-get-warning file.
The command will automatically proceed in 5 seconds or on any key.

Visit https://cloud.google.com/shell/help for more information.
********************************************************************************
Reading package lists... Done
Building dependency tree       
Reading state information... Done
python3-dev is already the newest version (3.7.3-1).
0 upgraded, 0 newly installed, 0 to remove and 37 not upgraded.

需求.txt

-f https://download.pytorch.org/whl/torch_stable.html
numpy==1.19.4
torch==1.7.0+cpu
torchvision==0.8.1+cpu
pymatting==1.1.1
scikit-image==0.17.2
waitress==1.4.4
scipy==1.5.4
hsh==1.1.0
flask==1.1.2
filetype==1.0.7
matplotlib==3.1.1
tqdm==4.51.0
requests==2.25.0
fastapi==0.62.0
Pillow==8.0.1
skimage==0.0
uvicorn==0.11.6
gunicorn==20.0.4
python-multipart==0.0.5

问题仍然存在，知道如何让它发挥作用吗？

Answer 1

问题是 pymatting_aot.aot 确实无法导入，需要先编译。

您可以通过首先在代码中导入 pymatting_aot.cc 或将此导入合并到应用程序引擎的 Dockerfile 中来编译模块，以便容器附带预编译的共享对象库。

这里是一个快速概述，以了解它是如何工作的(或者更确切地说，看看它在编译后如何工作):

root@dcda6f2673cb:/# pip install pymatting

Collecting pymatting
  Downloading PyMatting-1.1.2-py3-none-any.whl (48 kB)
     |████████████████████████████████| 48 kB 1.4 MB/s
Collecting numba!=0.49.0
  Downloading numba-0.52.0-cp38-cp38-manylinux2014_x86_64.whl (3.2 MB)
     |████████████████████████████████| 3.2 MB 2.9 MB/s
Requirement already satisfied: setuptools in /usr/local/lib/python3.8/site-packages (from numba!=0.49.0->pymatting) (51.1.0)
Collecting llvmlite<0.36,>=0.35.0
  Downloading llvmlite-0.35.0-cp38-cp38-manylinux2010_x86_64.whl (25.3 MB)
     |████████████████████████████████| 25.3 MB 10.9 MB/s
Collecting numpy>=1.16.0
  Downloading numpy-1.19.4-cp38-cp38-manylinux2010_x86_64.whl (14.5 MB)
     |████████████████████████████████| 14.5 MB 11.3 MB/s
Collecting pillow>=5.2.0
  Downloading Pillow-8.0.1-cp38-cp38-manylinux1_x86_64.whl (2.2 MB)
     |████████████████████████████████| 2.2 MB 3.8 MB/s
Collecting scipy>=1.1.0
  Downloading scipy-1.5.4-cp38-cp38-manylinux1_x86_64.whl (25.8 MB)
     |████████████████████████████████| 25.8 MB 12.0 MB/s
Installing collected packages: numpy, llvmlite, scipy, pillow, numba, pymatting
Successfully installed llvmlite-0.35.0 numba-0.52.0 numpy-1.19.4 pillow-8.0.1 pymatting-1.1.2 scipy-1.5.4

root@dcda6f2673cb:/# python
Python 3.8.7 (default, Dec 22 2020, 18:46:25)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> import pymatting_aot.aot
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'pymatting_aot.aot'

>>> import pymatting_aot.cc
Failed to import ahead-of-time-compiled modules.
This is expected on first import.
Compiling modules and trying again.
This might take a minute.
Successfully imported ahead-of-time-compiled modules.

>>> import pymatting_aot.aot
>>> pymatting_aot.aot
<module 'pymatting_aot.aot' from '/usr/local/lib/python3.8/site-packages/pymatting_aot/aot.cpython-38-x86_64-linux-gnu.so'>

编辑1:事实上，在查看 cc.py 之后，您甚至不需要在导入 pymatting_aot.cc 后导入 pymatting_aot.aot，因为它会为您加载。

编辑 2:以下是如何从 Dockerfile 中为应用程序引擎的自定义 Flex 环境嵌入编译 .so:

~ cat Dockerfile
FROM python:3.8
RUN pip install pymatting && python -c 'import pymatting_aot.cc'

测试:

~ docker run -it --rm test:latest
Python 3.8.7 (default, Dec 22 2020, 18:46:25)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pymatting_aot.aot
>>>