python - 如何在 macOS 12 上的 M1 Mac 上安装 trax、jax、jaxlib？

Question

trax

trax 的新手，我尝试在本地运行它(macOS 12.1、Apple Silicon ARM M1 处理器、8GB RAM、Anaconda)，但我遇到了一些问题。

在使用 python 3.8.5 的环境中，我在 (Anaconda) conda 环境中安装了运行 pip3 install tr​​ax==1.3.9 的 trax。后来，我在尝试使用 from trax import layers as tl

在我的代码中导入 trax 层时遇到了问题

RuntimeError: This version of jaxlib was built using AVX instructions, which your CPU and/or operating system do not support. You may be able work around this issue by building jaxlib from source.

我赶紧用python 3.10开始新的conda环境。然后在尝试安装 trax 时遇到问题:

macos ERROR: Could not find a version that satisfies the requirement tensorflow-text (from trax) (from versions: none)
ERROR: No matching distribution found for tensorflow-text

然后我用 python 3.9 创建了一个新环境。安装很顺利，但在导入图层时遇到了同样的错误:

RuntimeError: This version of jaxlib was built using AVX instructions, which your CPU and/or operating system do not support. You may be able work around this issue by building jaxlib from source.

在trax之前安装jaxlib, jax

然后我尝试从 these instructions 之后的源代码构建 jaxlib 并收到此错误:

Building XLA and installing it in the jaxlib source tree...
./bazel-4.2.1-darwin-x86_64 run --verbose_failures=true --config=avx_posix --config=mkl_open_source_only :build_wheel -- --output_path=/my path/jax/dist --cpu=x86_64
ERROR: bazel does not currently work properly from paths containing spaces (/my path/jax).
b''
Traceback (most recent call last):
  File "/my path/jax/build/build.py", line 524, in <module>
    main()
  File "/my path/jax/build/build.py", line 519, in main
    shell(command)
  File "/my path/jax/build/build.py", line 53, in shell
    output = subprocess.check_output(cmd)
  File "/myuserpath/opt/anaconda3/envs/mytraxenv/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/myuserpath/opt/anaconda3/envs/mytraxenv/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['./bazel-4.2.1-darwin-x86_64', 'run', '--verbose_failures=true', '--config=avx_posix', '--config=mkl_open_source_only', ':build_wheel', '--', '--output_path=/my path/jax/dist', '--cpu=x86_64']' returned non-zero exit status 36.

强调我最初错过的部分:bazel 目前无法在包含空格的路径 (/my path/jax) 中正常工作。

我将我的 /my path/ 目录移动到没有空格的路径 /mypath/。删除并重新下载 jax 目录。尽管如此，使用 python build/build.py 构建(针对 CPU)失败了:

ERROR: /private/var/tmp/_bazel_a/2caf512c3c5e3f3f654bc58b48b8333a/external/llvm-project/llvm/BUILD.bazel:610:11: Generating code from table: include/llvm/IR/Intrinsics.td @llvm-project//llvm:intrinsic_XCore_gen__gen_intrinsic_enums__intrinsic_prefix_xcore_genrule failed: (Illegal instruction): bash failed: error executing command 
  (cd /private/var/tmp/_bazel_a/2caf512c3c5e3f3f654bc58b48b8333a/execroot/__main__ && \
  exec env - \
    PATH=/myuserpath/opt/anaconda3/envs/mytraxenv/bin:/myuserpath/opt/anaconda3/condabin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/Library/Apple/usr/bin \
  /bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; bazel-out/darwin-opt/bin/external/llvm-project/llvm/llvm-tblgen -I external/llvm-project/llvm/include -I external/llvm-project/clang/include -I $(dirname external/llvm-project/llvm/include/llvm/IR/Intrinsics.td) -gen-intrinsic-enums -intrinsic-prefix=xcore external/llvm-project/llvm/include/llvm/IR/Intrinsics.td  -o bazel-out/darwin-opt/bin/external/llvm-project/llvm/include/llvm/IR/IntrinsicsXCore.h')
Execution platform: @local_execution_config_platform//:platform
/bin/bash: line 1: 11140 Illegal instruction: 4  bazel-out/darwin-opt/bin/external/llvm-project/llvm/llvm-tblgen -I external/llvm-project/llvm/include -I external/llvm-project/clang/include -I $(dirname external/llvm-project/llvm/include/llvm/IR/Intrinsics.td) -gen-intrinsic-enums -intrinsic-prefix=xcore external/llvm-project/llvm/include/llvm/IR/Intrinsics.td -o bazel-out/darwin-opt/bin/external/llvm-project/llvm/include/llvm/IR/IntrinsicsXCore.h
Target //build:build_wheel failed to build
INFO: Elapsed time: 620.950s, Critical Path: 45.35s
INFO: 589 processes: 132 internal, 457 local.
FAILED: Build did NOT complete successfully
ERROR: Build failed. Not running target
FAILED: Build did NOT complete successfully
b''
Traceback (most recent call last):
  File "/mypath/jax/build/build.py", line 524, in <module>
    main()
  File "/mypath/jax/build/build.py", line 519, in main
    shell(command)
  File "/mypath/jax/build/build.py", line 53, in shell
    output = subprocess.check_output(cmd)
  File "/myuserpath/opt/anaconda3/envs/mytraxenv/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/myuserpath/opt/anaconda3/envs/mytraxenv/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['./bazel-4.2.1-darwin-x86_64', 'run', '--verbose_failures=true', '--config=avx_posix', '--config=mkl_open_source_only', ':build_wheel', '--', '--output_path=/mypath/jax/dist', '--cpu=x86_64']' returned non-zero exit status 1.

再次尝试了几次(删除并重新下载了 jax 目录)，.../lib/python3.9/subprocess.py 中的同一行 528 似乎导致了问题，但输出, 虽然主要是上述内容，但有时略有不同。让我怀疑内存有问题，因为我(诚然)几周没有重新启动我的机器并且它开始无响应地变慢。

我将我的 XCode 命令行工具更新到版本 12.2(非常肯定)。

我重新启动了我的 (8GB) 机器。删除并重新下载 jax 目录。我用自制软件安装了 bazel 5.0.0 版，以防有帮助。我有点担心它一直在为我的 ARM 处理器下载 x86 版本。 bazel 安装顺利。

从 these instructions 重新开始。不过，jaxlib 构建明确表示它需要较早 (4.2.1) 版本的 bazel 并像以前一样下载了它:

b'\x1b[31mERROR: The project you\'re trying to build requires Bazel 4.2.1 (specified in /mypath/jax/.bazelversion), but it wasn\'t found in /opt/homebrew/Cellar/bazel/5.0.0/libexec/bin.\x1b[0m\n\nBazel binaries for all official releases can be downloaded from here:\n  https://github.com/bazelbuild/bazel/releases\n\nYou can download the required version directly using this command:\n  (cd "/opt/homebrew/Cellar/bazel/5.0.0/libexec/bin" && curl -fLO https://releases.bazel.build/4.2.1/release/bazel-4.2.1-darwin-x86_64 && chmod +x bazel-4.2.1-darwin-x86_64)\n'

再次，同一行 528 中出现不同的错误。现在显示完整运行:

b'\x1b[31mERROR: The project you\'re trying to build requires Bazel 4.2.1 (specified in /mypath/jax/.bazelversion), but it wasn\'t found in /opt/homebrew/Cellar/bazel/5.0.0/libexec/bin.\x1b[0m\n\nBazel binaries for all official releases can be downloaded from here:\n  https://github.com/bazelbuild/bazel/releases\n\nYou can download the required version directly using this command:\n  (cd "/opt/homebrew/Cellar/bazel/5.0.0/libexec/bin" && curl -fLO https://releases.bazel.build/4.2.1/release/bazel-4.2.1-darwin-x86_64 && chmod +x bazel-4.2.1-darwin-x86_64)\n'
Downloading bazel from: https://github.com/bazelbuild/bazel/releases/download/4.2.1/bazel-4.2.1-darwin-x86_64
bazel-4.2.1-darwin-x86_64 [########################################] 100%
Bazel binary path: ./bazel-4.2.1-darwin-x86_64
Bazel version: 4.2.1
Python binary path: /myuserpath/opt/anaconda3/envs/mytraxenv/bin/python
Python version: 3.9
NumPy version: 1.21.2
MKL-DNN enabled: yes
Target CPU: x86_64
Target CPU features: release
CUDA enabled: no
TPU enabled: no
ROCm enabled: no

Building XLA and installing it in the jaxlib source tree...
./bazel-4.2.1-darwin-x86_64 run --verbose_failures=true --config=avx_posix --config=mkl_open_source_only :build_wheel -- --output_path=/mypath/jax/dist --cpu=x86_64
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=0 --terminal_columns=80
INFO: Reading rc options for 'run' from /mypath/jax/.bazelrc:
  Inherited 'common' options: --experimental_repo_remote_exec
INFO: Reading rc options for 'run' from /mypath/jax/.bazelrc:
  Inherited 'build' options: --apple_platform_type=macos --macos_minimum_os=10.9 --announce_rc --define open_source_build=true --spawn_strategy=standalone --enable_platform_specific_config --define=no_aws_support=true --define=no_gcp_support=true --define=no_hdfs_support=true --define=no_kafka_support=true --define=no_ignite_support=true --define=grpc_no_ares=true -c opt --config=short_logs --copt=-DMLIR_PYTHON_PACKAGE_PREFIX=jaxlib.mlir.
INFO: Reading rc options for 'run' from /mypath/jax/.jax_configure.bazelrc:
  Inherited 'build' options: --strategy=Genrule=standalone --repo_env PYTHON_BIN_PATH=/myuserpath/opt/anaconda3/envs/mytraxenv/bin/python --action_env=PYENV_ROOT --python_path=/myuserpath/opt/anaconda3/envs/mytraxenv/bin/python --distinct_host_configuration=false
INFO: Found applicable config definition build:short_logs in file /mypath/jax/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:avx_posix in file /mypath/jax/.bazelrc: --copt=-mavx --host_copt=-mavx
INFO: Found applicable config definition build:mkl_open_source_only in file /mypath/jax/.bazelrc: --define=tensorflow_mkldnn_contraction_kernel=1
INFO: Found applicable config definition build:macos in file /mypath/jax/.bazelrc: --config=posix
INFO: Found applicable config definition build:posix in file /mypath/jax/.bazelrc: --copt=-fvisibility=hidden --copt=-Wno-sign-compare --cxxopt=-std=c++14 --host_cxxopt=-std=c++14
Loading: 
Loading: 0 packages loaded
Analyzing: target //build:build_wheel (1 packages loaded, 0 targets configured)
Analyzing: target //build:build_wheel (8 packages loaded, 286 targets configured)
Analyzing: target //build:build_wheel (10 packages loaded, 4271 targets configured)
Analyzing: target //build:build_wheel (10 packages loaded, 4271 targets configured)
Analyzing: target //build:build_wheel (11 packages loaded, 4494 targets configured)
INFO: Analyzed target //build:build_wheel (11 packages loaded, 7417 targets configured).

INFO: Found 1 target...
[0 / 3] [Prepa] BazelWorkspaceStatusAction stable-status.txt
[21 / 233] Compiling src/google/protobuf/generated_enum_util.cc; 2s local ... (6 actions, 3 running)
[28 / 233] Compiling src/google/protobuf/extension_set.cc; 4s local ... (8 actions, 7 running)
[114 / 523] Compiling src/google/protobuf/generated_message_util.cc; 2s local ... (8 actions running)
[140 / 594] Compiling platform/c++11/src/nsync_semaphore_mutex.cc; 0s local ... (8 actions, 7 running)
[237 / 597] Compiling src/google/protobuf/util/message_differencer.cc; 0s local ... (8 actions running)
[252 / 597] Compiling src/google/protobuf/util/message_differencer.cc; 4s local ... (8 actions running)
[272 / 597] Compiling src/google/protobuf/descriptor.pb.cc; 8s local ... (8 actions running)
[296 / 597] Compiling src/google/protobuf/descriptor.cc; 12s local ... (8 actions running)
[315 / 597] Compiling src/google/protobuf/descriptor_database.cc; 2s local ... (5 actions running)
[487 / 1,963] Compiling src/compiler/python_generator.cc; 2s local ... (8 actions running)
[579 / 3,117] Compiling tensorflow/compiler/xla/service/cpu/runtime_single_threaded_conv3d.cc; 0s local ... (8 actions, 7 running)
[619 / 3,173] Compiling tensorflow/compiler/xla/service/cpu/runtime_single_threaded_conv3d.cc; 8s local ... (8 actions running)
[687 / 3,270] Compiling tensorflow/compiler/xla/service/cpu/runtime_single_threaded_conv3d.cc; 17s local ... (8 actions running)
[774 / 3,464] Compiling tensorflow/compiler/xla/service/cpu/runtime_single_threaded_conv2d.cc; 27s local ... (8 actions running)
[1,204 / 4,860] Compiling tensorflow/compiler/xla/service/cpu/runtime_single_threaded_conv2d.cc; 39s local ... (8 actions running)
[1,255 / 4,916] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 52s local ... (8 actions running)
[1,340 / 5,042] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 68s local ... (8 actions running)
[1,456 / 5,156] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 86s local ... (8 actions, 7 running)
[1,661 / 5,704] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 107s local ... (8 actions, 7 running)
[1,688 / 5,704] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 132s local ... (8 actions, 7 running)
[1,721 / 5,704] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 160s local ... (8 actions, 7 running)
[2,106 / 6,584] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 192s local ... (8 actions, 7 running)
[2,342 / 7,067] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 231s local ... (8 actions, 7 running)
[2,378 / 7,067] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 274s local ... (8 actions, 7 running)
[2,422 / 7,067] Compiling src/cpu/rnn/ref_rnn.cpp; 75s local ... (8 actions, 7 running)
[2,452 / 7,067] Compiling src/cpu/x64/gemm/f32/jit_avx512_core_f32_copy_at_kern_part1_autogen.cpp; 54s local ... (8 actions, 7 running)
[2,500 / 7,067] Compiling src/cpu/x64/gemm/f32/jit_avx512_core_f32_copy_at_kern_part1_autogen.cpp; 119s local ... (8 actions, 7 running)
[2,577 / 7,067] Compiling src/common/memory_zero_pad.cpp; 74s local ... (8 actions, 7 running)
ERROR: /private/var/tmp/_bazel_a/f5e9a3325f07a1f02c52d821857db47c/external/org_tensorflow/tensorflow/compiler/xla/BUILD:69:17: ProtoCompile external/org_tensorflow/tensorflow/compiler/xla/xla.pb.h failed: (Illegal instruction): protoc failed: error executing command 
  (cd /private/var/tmp/_bazel_a/f5e9a3325f07a1f02c52d821857db47c/execroot/__main__ && \
  exec env - \
    PATH=/myuserpath/opt/anaconda3/envs/mytraxenv/bin:/myuserpath/opt/anaconda3/condabin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/Library/Apple/usr/bin \
  bazel-out/darwin-opt/bin/external/com_google_protobuf/protoc '--cpp_out=bazel-out/darwin-opt/bin/external/org_tensorflow' -Iexternal/org_tensorflow -Ibazel-out/darwin-opt/bin/external/org_tensorflow -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/any_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/source_context_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/type_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/api_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/descriptor_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/compiler_plugin_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/duration_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/empty_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/field_mask_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/struct_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/timestamp_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/wrappers_proto external/org_tensorflow/tensorflow/compiler/xla/xla.proto)
Execution platform: @local_execution_config_platform//:platform
Target //build:build_wheel failed to build
INFO: Elapsed time: 631.189s, Critical Path: 283.35s
INFO: 2563 processes: 935 internal, 1628 local.
FAILED: Build did NOT complete successfully
ERROR: Build failed. Not running target
FAILED: Build did NOT complete successfully
b''
Traceback (most recent call last):
  File "/mypath/jax/build/build.py", line 524, in <module>
    main()
  File "/mypath/jax/build/build.py", line 519, in main
    shell(command)
  File "/mypath/jax/build/build.py", line 53, in shell
    output = subprocess.check_output(cmd)
  File "/myuserpath/opt/anaconda3/envs/mytraxenv/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/myuserpath/opt/anaconda3/envs/mytraxenv/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['./bazel-4.2.1-darwin-x86_64', 'run', '--verbose_failures=true', '--config=avx_posix', '--config=mkl_open_source_only', ':build_wheel', '--', '--output_path=/mypath/jax/dist', '--cpu=x86_64']' returned non-zero exit status 1.

我不再尝试构建 jaxlib。在 late October 2021 中，发布了一个兼容 M1 的 jaxlib wheel，所以我尝试了:

pip install -U pip
pip install -U https://storage.googleapis.com/jax-releases/mac/jaxlib-0.1.74-cp39-none-macosx_11_0_arm64.whl

但是得到了

ERROR: jaxlib-0.1.74-cp39-none-macosx_11_0_arm64.whl is not a supported wheel on this platform.

试过 upgrading python from 3.9 to 3.10 ，但得到了同样的信息。

调试，在python中，运行

import platform
print(platform.machine())

这表明我的 python 仍在 x86 架构上运行，因为 "Anaconda doesn't yet provide packages for M1/ARM"。

[在下面自己的答案中看到成功(到目前为止)jaxlib 安装]

回到trax

成功安装 jaxlib 和 jax 后，尝试使用(miniforge 的)conda install tr​​ax 安装 trax我得到:

Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.

PackagesNotFoundError: The following packages are not available from current channels:

  - trax

Current channels:

  - https://conda.anaconda.org/conda-forge/osx-arm64
  - https://conda.anaconda.org/conda-forge/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

我尝试安装为 trax itself suggests 但在构建 hp5y 时遇到错误:

ERROR: Failed building wheel for h5py
ERROR: Could not build wheels for h5py, which is required to install pyproject.toml-based projects

现在调查这个错误，并再次考虑使用 miniforge 来解决这个问题，但使用 Anaconda 的 channel 来安装 trax。我会在到达时进行更新。

任何指点或任何设法通过任何不同方式在 M1 Mac 上安装 trax 的人？

Answer 1

jaxlib + jax

在 someone claimed success using miniforge 之后, 我读了this并观看了this阐明一起使用 Anaconda 和 miniforge。

我用 Apple's arm64 : Apple Silicon method 安装了 miniforge .出于某种原因，当我运行 miniforge 的 conda init 时，即使我使用的是 zsh shell，它也会在 ~/.bash_profile 中设置初始化代码。我尝试手动将代码放在 ~/.zprofile 中，但它不会加载到交互式 shell 中，所以我最终将它放在 Anaconda 放置其初始化代码的地方，在 ~/zshrc.

这使 miniforge 成为默认管理器。按照上面非常有用的视频，我创建了一个 ~/.start_anaconda.sh 脚本，这样我就可以使用 Anaconda 作为替代方案。

使用 miniforge

• 使用 conda create -n mytraxenv python=3 创建了一个新的 conda 环境 mytraxenv 目前有 python 3.10.2

• 激活环境:conda activate mytraxenv

• 运行 conda install numpy 和 conda install six 以确保 numpy。我的mytraxenv环境中安装了six和wheel(前两者之一安装)

• 再次尝试，稍微更新版本(来自 here ):

  pip 安装 -U pippip 安装-U https://storage.googleapis.com/jax-releases/mac/jaxlib-0.1.75-cp310-none-macosx_11_0_arm64.whl

这在安装 jaxlib 时有效!

然后，我关注了these instructions安装 jax:

pip install --upgrade pip
pip install --upgrade "jax[cpu]"

同样有效。请注意，在 python 中运行 import jax 时，它当前会发出警告:

/mytraxenv/lib/python3.10/site-packages/jax/_src/lib/__init__.py:32: UserWarning: JAX on Mac ARM machines is experimental and minimally tested. Please see https://github.com/google/jax/issues/5501 in the event of problems.
  warnings.warn("JAX on Mac ARM machines is experimental and minimally tested. "

trax

尚未安装 trax 成功。