python - 无法加载 XGBoost 库 (libxgboost.so)

Question

问题

在无服务器项目中，我想将 XGBoost 导入到用 Python 编写的 lambda 中。但是当我尝试调用 lambda 时，我在 CloudWatch 上看到了这个错误:

[ERROR] XGBoostError: XGBoost Library (libxgboost.so) could not be loaded.
Likely causes:
* OpenMP runtime is not installed (vcomp140.dll or libgomp-1.dll for Windows, libgomp.so for UNIX-like OSes)
* You are running 32-bit Python on a 64-bit OS
Error message(s): ['libgomp.so.1: cannot open shared object file: No such file or directory']

我尝试了什么？

我在运行 sls deploy在 macOS 上，我添加了 dockerizePip: true到我的 serverless.yml

我尝试通过使用自定义 Dockerfile 修复丢失的依赖项:

FROM lambci/lambda:build-python3.6

RUN apt-get update && apt-get install libaio1

I also must specify dockerExtraFiles 中的库路径但我不知道 libgomp.so 应该在 Linux 上的什么位置。所以，我坚持这一点。

我的代码

无服务器.yml:

app: improve
org: kvadrug
service: testservice

provider:
  name: aws
  runtime: python3.8
  versionFunctions: false
  stage: dev
  region: us-west-2
  timeout: 30

plugins:
  - serverless-python-requirements

custom:
  pythonRequirements:
    dockerFile: Dockerfile
    zip: true
    dockerizePip: true

functions:
  hello:
    handler: hello.hello
    events:
      - http:
          path: hello
          method: post
          private: true

包.json:

{
    "name": "testservice",
    "version": "1.0.0",
    "description": "Test service",
    "dependencies": {},
    "devDependencies": {
        "serverless-python-requirements": "^5.1.0"
    }
}

要求.txt:

xgboost==1.0.2

Answer 1

工作配置 - Python 3.7

在 Mojave 10.14.6 (18G4032) 和 Docker v2.1.0.2 (37877) 上测试。

脚步:

sls requirements clean
rm -Rf ~/Library/Caches/serverless-python-requirements/
sls deploy

要求.txt:

xgboost==1.0.2

无服务器.yml:

service: xgboost
provider:
  name: aws
  timeout:60
  runtime: python3.7

plugins:
  - serverless-python-requirements

custom:
  pythonRequirements:
    zip: true
    dockerizePip: non-linux
    dockerExtraFiles:
      - /usr/lib64/libgomp.so.1

functions:
  hello:
    handler: handler.hello
    events:
      - http:
          path: hello
          method: post
          private: true

处理程序.py:

    try:
        import unzip_requirements
    except ImportError:
        pass

    import sys
    import glob
    import os

    def hello(event, context):
        pkgdir = '/tmp/sls-py-req/'

        print("-------- Sys Path --------")
        for p in sys.path:
            print(p)

        if os.path.exists(pkgdir):
            print("-------- Pkg Dir ----------")
            os.chdir(pkgdir)
            for file in glob.glob("*"):
                print(file)

        print("-------- Var Task ----------")
        os.chdir("/var/task")
        for file in glob.glob("*"):
            print(file)

        try:
            import xgboost as xgb
            print(xgb)
        except Exception as ex:
            template = "An exception of type {0} occurred. Arguments:\n{1!r}"
            message = template.format(type(ex).__name__, ex.args)
            print(message)

        return True

Python 3.8

出于某种原因，运行 Python 3.8 的 Lambda 会忽略“sys.path”上的“/tmp/sls-py-req”条目。因此，您需要手动将库文件“libgomp.so.1”添加到应用程序的根目录。

要尝试一下，请将 serverless.yml 上的“runtime: python3.7”更新为“runtime: python3.8”，然后执行以下步骤:

sls requirements clean
rm -Rf ~/Library/Caches/serverless-python-requirements/
sls package
cp .serverless/requirements/libgomp.so.1 ./
sls deploy

仅供引用 - 如何在 Lambda docker 镜像中找到库位置

跑:

docker run --rm -ti --entrypoint /bin/sh -u 0 lambci/lambda\:build-python3.8

然后:

find / -name libgomp.so.1