python - 使用 Bazel 为 AWS Lambda 创建 Python zip

Question

我有一个包含一组 Python AWS lambda 的 monorepo，我正在使用 Bazel用于构建和打包 lambda。我现在正在尝试使用 Bazel 创建一个符合预期 AWS Lambdas packaging 的 zip 文件我可以上传到 Lambda。想知道使用 Bazel 执行此操作的最佳方法是什么？

以下是我到目前为止尝试过的一些不同的东西:

尝试 1:py_binary

构建.bazel

py_binary(
name = "main_binary",
srcs = glob(["*.py"]),
main = "main.py",
visibility = ["//appcode/api/transaction_details:__subpackages__"],
deps = [
        requirement("Faker"),
    ],
)

问题:

这会生成以下内容:

• main_binary(python 可执行文件)
• main_binary.runfiles
• main_binary.runfiles_manifest

Lambda 期望处理程序的格式为 lambda_function.lambda_handler。由于 main_binary 是可执行文件而不是 python 文件，它不会公开实际的处理程序方法，并且 lambda 会因为找不到它而崩溃。我尝试更新处理程序配置以简单地指向 main_binary 但它爆炸了，因为它需要两个参数(即 lambda_function.lambda_handler)。

尝试 2:py_library + pkg_zip

构建.bazel

py_library(
name = "main",
srcs = glob(["*.py"]),
visibility = ["//appcode/api/transaction_details:__subpackages__"],
deps = [
        requirement("Faker"),
    ],
)

pkg_zip(
name = "main_zip",
srcs =["//appcode/api/transaction_details/src:main" ],
)

问题:

这会生成一个 zip 文件，其中包含:

• 主要文件
• __init__.py

zip 文件现在包含 main.py 但不包含其运行时依赖项。因此 lambda 爆炸了，因为它找不到 Faker。

其他尝试:

我也尝试过使用 --build_python_zip 标志以及带有通用规则的 @bazel_tools//tools/zip:zipper 但它们都会导致与前两次尝试类似的结果。

Answer 1

我们将 @bazel_tools//tools/zip:zipper 与自定义规则一起使用。我们还使用 rules_nodejs 引入 serverless 并通过 bazel 运行它，这导致包构建在运行 sls deploy 之前发生。

我们使用 rules_python 中的 pip_parse。我不确定下面的 _short_path 函数是否适用于 pip_install 或其他机制。

支持文件过滤，虽然有点笨拙。理想情况下，zip 生成将由单独的二进制文件(即 Python 脚本)处理，这将允许使用正则表达式/globs/等进行过滤。 Bazel 不支持 Starlark 中的正则表达式，所以我们使用自己的东西。

我摘录了一段:

lambda.bzl

"""
Support for serverless deployments.
"""

def contains(pattern):
    return "contains:" + pattern

def startswith(pattern):
    return "startswith:" + pattern

def endswith(pattern):
    return "endswith:" + pattern

def _is_ignored(path, patterns):
    for p in patterns:
        if p.startswith("contains:"):
            if p[len("contains:"):] in path:
                return True
        elif p.startswith("startswith:"):
            if path.startswith(p[len("startswith:"):]):
                return True
        elif p.startswith("endswith:"):
            if path.endswith(p[len("endswith:"):]):
                return True
        else:
            fail("Invalid pattern: " + p)

    return False

def _short_path(file_):
    # Remove prefixes for external and generated files.
    # E.g.,
    #   ../py_deps_pypi__pydantic/pydantic/__init__.py -> pydantic/__init__.py
    short_path = file_.short_path
    if short_path.startswith("../"):
        second_slash = short_path.index("/", 3)
        short_path = short_path[second_slash + 1:]
    return short_path

def _py_lambda_zip_impl(ctx):
    deps = ctx.attr.target[DefaultInfo].default_runfiles.files

    f = ctx.outputs.output

    args = []
    for dep in deps.to_list():
        short_path = _short_path(dep)

        # Skip ignored patterns
        if _is_ignored(short_path, ctx.attr.ignore):
            continue

        args.append(short_path + "=" + dep.path)

    ctx.actions.run(
        outputs = [f],
        inputs = deps,
        executable = ctx.executable._zipper,
        arguments = ["cC", f.path] + args,
        progress_message = "Creating archive...",
        mnemonic = "archiver",
    )

    out = depset(direct = [f])
    return [
        DefaultInfo(
            files = out,
        ),
        OutputGroupInfo(
            all_files = out,
        ),
    ]

_py_lambda_zip = rule(
    implementation = _py_lambda_zip_impl,
    attrs = {
        "target": attr.label(),
        "ignore": attr.string_list(),
        "_zipper": attr.label(
            default = Label("@bazel_tools//tools/zip:zipper"),
            cfg = "host",
            executable = True,
        ),
        "output": attr.output(),
    },
    executable = False,
    test = False,
)

def py_lambda_zip(name, target, ignore, **kwargs):
    _py_lambda_zip(
        name = name,
        target = target,
        ignore = ignore,
        output = name + ".zip",
        **kwargs
    )

BUILD.bazel

load("@npm_serverless//serverless:index.bzl", "serverless")
load(":lambda.bzl", "contains", "endswith", "py_lambda_zip", "startswith")

py_binary(
    name = "my_lambda_app",
    ...
)

py_lambda_zip(
    name = "lambda_archive",
    ignore = [
        contains("/__pycache__/"),
        endswith(".pyc"),
        endswith(".pyo"),
        
        # Ignore boto since it's provided by Lambda.
        startswith("boto3/"),
        startswith("botocore/"),

        # With the move to hermetic toolchains, the zip gets a lib/ directory containing the
        # python runtime. We don't need that.
        startswith("lib/"),
    ],
    target = ":my_lambda_app",

    # Only allow building on linux, since we don't want to upload a lambda zip file
    # with e.g. macos compiled binaries.
    target_compatible_with = [
        "@platforms//os:linux",
    ],
)

# The sls command requires that serverless.yml be in its working directory, and that the yaml file
# NOT be a symlink. So this target builds a directory containing a copy of serverless.yml, and also 
# symlinks the generated lambda_archive.zip in the same directory.
#
# It also generates a chdir.js script that we instruct node to execute to change to the proper working directory.
genrule(
    name = "sls_files",
    srcs = [
        "lambda_archive.zip",
        "serverless.yml",
    ],
    outs = [
        "sls_files/lambda_archive.zip",
        "sls_files/serverless.yml",
        "sls_files/chdir.js",
    ],
    cmd = """
        mkdir -p $(@D)/sls_files
        cp $(location serverless.yml) $(@D)/sls_files/serverless.yml
        cp -P $(location lambda_archive.zip) $(@D)/sls_files/lambda_archive.zip

        echo "const fs = require('fs');" \
             "const path = require('path');" \
             "process.chdir(path.dirname(fs.realpathSync(__filename)));" > $(@D)/sls_files/chdir.js
    """,
)

# Usage:
#   bazel run //:sls -- deploy <more args>
serverless(
    name = "sls",
    args = ["""--node_options=--require=./$(location sls_files/chdir.js)"""],
    data = [
        "sls_files/chdir.js",
        "sls_files/serverless.yml",
        "sls_files/lambda_archive.zip",
    ],
)

serverless.yml

service: my-app

package:
  artifact: lambda_archive.zip

# ... other config ...