python - NVIDIA Jetson Nano的PyTorch的Yocto Warrior Bitbake食谱

Question

我正在尝试为Python 3 PyTorch创建一个简单的Yocto Python食谱。目标是Yocto从meta-tegra层为NVIDIA Jetson Nano制作的SD卡镜像。没有这些配方，我就可以成功地从meta-tegra编译并引导镜像。

NVIDIA本身已经编译并发布了“.whl” Python软件包，可在以下位置找到它们:https://devtalk.nvidia.com/default/topic/1048776/official-tensorflow-for-jetson-nano-/

使用他们的构建说明，我尝试编写一个Bitbake配方来安装PyTorch，如下所示:

SUMMARY = "Facebook PyTorch AI"
DESCRIPTION = "Facebook PyTorch AI"
HOMEPAGE = "https://pytorch.org/"
LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://LICENSE;md5=acf4d595f99e159bf31797aa872aef57"

inherit pypi
inherit setuptools3

#BBCLASSEXTEND = "native nativesdk"

DEPENDS += "python3-pytest-runner-native python3-pyyaml-native cmake-native"

do_configure_prepend() {
  USE_NCCL=0
  USE_DISTRIBUTED=0
  TORCH_CUDA_ARCH_LIST="5.3;6.2;7.2"
}

do_compile_prepend() {
  USE_NCCL=0
  USE_DISTRIBUTED=0
  TORCH_CUDA_ARCH_LIST="5.3;6.2;7.2"
}

SRC_URI = "gitsm://github.com/pytorch/pytorch.git;protocol=https"
SRCREV = "a3346e100e7f4e7ec90f18b7befcccc47d5a1c82"

S = "${WORKDIR}/git"

问题是我遇到了以下错误。我认为这与我的环境变量未被获取有关，因为错误包括“-DUSE_CUDA = False”，但是当我查看bitbake环境(bitbake -e)时，它似乎在那里。

这是错误:

**NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
ERROR: python3-torch-1.1.0-r0 do_compile: 'python3 setup.py build ' execution failed.
ERROR: python3-torch-1.1.0-r0 do_compile: Function failed: do_compile (log file is located at /home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/temp/log.do_compile.27625)
ERROR: Logfile of failure stored in: /home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/temp/log.do_compile.27625
Log data follows:
| DEBUG: Executing shell function do_compile
| -- The CXX compiler identification is GNU 7.2.1
| -- The C compiler identification is GNU 7.2.1
| -- Check for working CXX compiler: /home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/usr/bin/aarch64-poky-linux/aarch64-poky-linux-g++
| -- Check for working CXX compiler: /home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/usr/bin/aarch64-poky-linux/aarch64-poky-linux-g++ -- works
| -- Detecting CXX compiler ABI info
| -- Detecting CXX compiler ABI info - done
| -- Detecting CXX compile features
| -- Detecting CXX compile features - done
| -- Check for working C compiler: /home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/usr/bin/aarch64-poky-linux/aarch64-poky-linux-gcc
| -- Check for working C compiler: /home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/usr/bin/aarch64-poky-linux/aarch64-poky-linux-gcc -- works
| -- Detecting C compiler ABI info
| -- Detecting C compiler ABI info - done
| -- Detecting C compile features
| -- Detecting C compile features - done
| -- Not forcing any particular BLAS to be found
| -- Performing Test COMPILER_WORKS
| -- Performing Test COMPILER_WORKS - Failed
| CMake Error at cmake/MiscCheck.cmake:52 (message):
|   Could not run a simple program built with your compiler.  If you are trying
|   to use -fsanitize=address, make sure libasan is properly installed on your
|   system (you can confirm if the problem is this by attempting to build and
|   run a small program.)
| Call Stack (most recent call first):
|   CMakeLists.txt:271 (include)
| 
| 
| -- Configuring incomplete, errors occurred!
| See also "/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/git/build/CMakeFiles/CMakeOutput.log".
| See also "/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/git/build/CMakeFiles/CMakeError.log".
| Building wheel torch-1.2.0a0+a3346e1
| -- Building version 1.2.0a0+a3346e1
| cmake -DBUILD_AR=ar -DBUILD_AS=as  -DBUILD_CC=gcc  -DBUILD_CCLD=gcc  -DBUILD_CFLAGS=-isystem/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/usr/include -O2 -pipe -DBUILD_CPP=gcc  -E -DBUILD_CPPFLAGS=-isystem/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/usr/include -DBUILD_CXX=g++  -DBUILD_CXXFLAGS=-isystem/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/usr/include -O2 -pipe -DBUILD_FC=gfortran  -DBUILD_LD=ld  -DBUILD_LDFLAGS=-L/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/usr/lib                         -L/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/lib                         -Wl,--enable-new-dtags                         -Wl,-rpath-link,/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/usr/lib                         -Wl,-rpath-link,/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/lib                         -Wl,-rpath,/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/usr/lib                         -Wl,-rpath,/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/lib                         -Wl,-O1 -Wl,--allow-shlib-undefined -Wl,--dynamic-linker=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2 -DBUILD_NM=nm -DBUILD_PYTHON=True -DBUILD_RANLIB=ranlib -DBUILD_STRIP=strip -DBUILD_TEST=True -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS= -O2 -pipe -g -feliminate-unused-debug-types -fdebug-prefix-map=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0=/usr/src/debug/python3-torch/1.1.0-r0                      -fdebug-prefix-map=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot=                      -fdebug-prefix-map=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native=   -DCMAKE_C_FLAGS= -O2 -pipe -g -feliminate-unused-debug-types -fdebug-prefix-map=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0=/usr/src/debug/python3-torch/1.1.0-r0                      -fdebug-prefix-map=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot=                      -fdebug-prefix-map=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native=   -DCMAKE_EXE_LINKER_FLAGS=-Wl,-O1 -Wl,--hash-style=gnu -Wl,--as-needed -fstack-protector-strong -Wl,-z,relro,-z,now -DCMAKE_INSTALL_PREFIX=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/git/torch -DCMAKE_PREFIX_PATH=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot/usr/lib/python3.7/site-packages -DCMAKE_SHARED_LINKER_FLAGS=-Wl,-O1 -Wl,--hash-style=gnu -Wl,--as-needed -fstack-protector-strong -Wl,-z,relro,-z,now -DINSTALL_TEST=True -DPYTHON_EXECUTABLE=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/usr/bin/python3-native/python3 -DPYTHON_INCLUDE_DIR=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot/usr/include/python3.7m -DPYTHON_LIBRARY=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot/usr/lib/libpython3.7m.so.1.0 -DTORCH_BUILD_VERSION=1.2.0a0+a3346e1 -DUSE_CUDA=False -DUSE_DISTRIBUTED=True -DUSE_FBGEMM=True -DUSE_NCCL=False -DUSE_NUMPY=True -DUSE_QNNPACK=True -DUSE_SYSTEM_EIGEN_INSTALL=OFF -DUSE_SYSTEM_NCCL=False /home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/git
| Traceback (most recent call last):
|   File "setup.py", line 749, in <module>
|     build_deps()
|   File "setup.py", line 323, in build_deps
|     cmake=cmake)
|   File "/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/git/tools/build_pytorch_libs.py", line 61, in build_caffe2
|     rerun_cmake)
|   File "/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/git/tools/setup_helpers/cmake.py", line 318, in generate
|     self.run(args, env=my_env)
|   File "/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/git/tools/setup_helpers/cmake.py", line 93, in run
|     check_call(command, cwd=self.build_dir, env=env)
|   File "/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/usr/lib/python3.7/subprocess.py", line 347, in check_call
|     raise CalledProcessError(retcode, cmd)
| subprocess.CalledProcessError: Command '['cmake', '-DBUILD_AR=ar', '-DBUILD_AS=as ', '-DBUILD_CC=gcc ', '-DBUILD_CCLD=gcc ', '-DBUILD_CFLAGS=-isystem/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/usr/include -O2 -pipe', '-DBUILD_CPP=gcc  -E', '-DBUILD_CPPFLAGS=-isystem/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/usr/include', '-DBUILD_CXX=g++ ', '-DBUILD_CXXFLAGS=-isystem/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/usr/include -O2 -pipe', '-DBUILD_FC=gfortran ', '-DBUILD_LD=ld ', '-DBUILD_LDFLAGS=-L/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/usr/lib                         -L/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/lib                         -Wl,--enable-new-dtags                         -Wl,-rpath-link,/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/usr/lib                         -Wl,-rpath-link,/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/lib                         -Wl,-rpath,/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/usr/lib                         -Wl,-rpath,/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/lib                         -Wl,-O1 -Wl,--allow-shlib-undefined -Wl,--dynamic-linker=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2', '-DBUILD_NM=nm', '-DBUILD_PYTHON=True', '-DBUILD_RANLIB=ranlib', '-DBUILD_STRIP=strip', '-DBUILD_TEST=True', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_CXX_FLAGS= -O2 -pipe -g -feliminate-unused-debug-types -fdebug-prefix-map=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0=/usr/src/debug/python3-torch/1.1.0-r0                      -fdebug-prefix-map=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot=                      -fdebug-prefix-map=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native=  ', '-DCMAKE_C_FLAGS= -O2 -pipe -g -feliminate-unused-debug-types -fdebug-prefix-map=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0=/usr/src/debug/python3-torch/1.1.0-r0                      -fdebug-prefix-map=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot=                      -fdebug-prefix-map=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native=  ', '-DCMAKE_EXE_LINKER_FLAGS=-Wl,-O1 -Wl,--hash-style=gnu -Wl,--as-needed -fstack-protector-strong -Wl,-z,relro,-z,now', '-DCMAKE_INSTALL_PREFIX=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/git/torch', '-DCMAKE_PREFIX_PATH=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot/usr/lib/python3.7/site-packages', '-DCMAKE_SHARED_LINKER_FLAGS=-Wl,-O1 -Wl,--hash-style=gnu -Wl,--as-needed -fstack-protector-strong -Wl,-z,relro,-z,now', '-DINSTALL_TEST=True', '-DPYTHON_EXECUTABLE=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/usr/bin/python3-native/python3', '-DPYTHON_INCLUDE_DIR=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot/usr/include/python3.7m', '-DPYTHON_LIBRARY=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot/usr/lib/libpython3.7m.so.1.0', '-DTORCH_BUILD_VERSION=1.2.0a0+a3346e1', '-DUSE_CUDA=False', '-DUSE_DISTRIBUTED=True', '-DUSE_FBGEMM=True', '-DUSE_NCCL=False', '-DUSE_NUMPY=True', '-DUSE_QNNPACK=True', '-DUSE_SYSTEM_EIGEN_INSTALL=OFF', '-DUSE_SYSTEM_NCCL=False', '/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/git']' returned non-zero exit status 1.
| ERROR: 'python3 setup.py build ' execution failed.
| WARNING: exit code 1 from a shell command.
| ERROR: Function failed: do_compile (log file is located at /home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/temp/log.do_compile.27625)
ERROR: Task (/home/ubuntu/Desktop/comp-jetson-yocto/layers/meta-comp/recipes-python/python3-torch/python3-torch_1.1.0.bb:do_compile) failed with exit code '1'
NOTE: Tasks Summary: Attempted 1770 tasks of which 1763 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
  /home/ubuntu/Desktop/comp-jetson-yocto/layers/meta-comp/recipes-python/python3-torch/python3-torch_1.1.0.bb:do_compile
Summary: There was 1 WARNING message shown.
Summary: There were 2 ERROR messages shown, returning a non-zero exit code.**

Answer 1

问题的根本原因是

if (NOT INTERN_BUILD_MOBILE)
  # ---[ Check that our programs run.  This is different from the native CMake
  # compiler check, which just tests if the program compiles and links.  This is
  # important because with ASAN you might need to help the compiled library find
  # some dynamic libraries.
  cmake_push_check_state(RESET)
  CHECK_C_SOURCE_RUNS("
  int main() { return 0; }
  " COMPILER_WORKS)
  if (NOT COMPILER_WORKS)
    # Force cmake to retest next time around
    unset(COMPILER_WORKS CACHE)
    message(FATAL_ERROR
        "Could not run a simple program built with your compiler. "
        "If you are trying to use -fsanitize=address, make sure "
        "libasan is properly installed on your system (you can confirm "
        "if the problem is this by attempting to build and run a "
        "small program.)")
  endif()
  cmake_pop_check_state()
endif()

这是正在运行的程序编译的。
但是我们在这里交叉编译。
你可以尝试设置INTERN_BUILD_MOBILE

set(INTERN_BUILD_MOBILE ON)

我没有直接来源会建立在杰森·纳诺(Jetson-Nano)上
但是紧随其后可能会帮助您取得进展。
您可以引用 this pull request进行android的交叉编译。

很高兴回答您可能需要的其他任何问题。