python - TensorFlow 如何通过 array_ops.cc 生成 gen_array_ops.py？

Question

TensorFlow 自动生成代码。我很好奇 TF 是如何生成的 gen_array_ops.py来自 array_ops.cc ?

生成的python文件在python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py

"""Python wrappers around TensorFlow ops.

This file is MACHINE GENERATED! Do not edit.
Original C++ source file: array_ops.cc
"""
...
...

Answer 1

Python 代码生成是在构建时通过 Bazel 完成的。您可以在 tensorflow/tensorflow.bzl 中找到相关定义，我将在这里只发布标题:

# Generates a Python library target wrapping the ops registered in "deps".
#
# Args:
#   name: used as the name of the generated target and as a name component of
#     the intermediate files.
#   out: name of the python file created by this rule. If None, then
#     "ops/gen_{name}.py" is used.
#   hidden: Optional list of ops names to make private in the Python module.
#     It is invalid to specify both "hidden" and "op_whitelist".
#   visibility: passed to py_library.
#   deps: list of dependencies for the intermediate tool used to generate the
#     python target. NOTE these `deps` are not applied to the final python
#     library target itself.
#   require_shape_functions: leave this as False.
#   hidden_file: optional file that contains a list of op names to make private
#     in the generated Python module. Each op name should be on a line by
#     itself. Lines that start with characters that are invalid op name
#     starting characters are treated as comments and ignored.
#   generated_target_name: name of the generated target (overrides the
#     "name" arg)
#   op_whitelist: if not empty, only op names in this list will be wrapped. It
#     is invalid to specify both "hidden" and "op_whitelist".
#   cc_linkopts: Optional linkopts to be added to tf_cc_binary that contains the
#     specified ops.

def tf_gen_op_wrapper_py(
        name,
        out = None,
        hidden = None,
        visibility = None,
        deps = [],
        require_shape_functions = False,
        hidden_file = None,
        generated_target_name = None,
        op_whitelist = [],
        cc_linkopts = [],
        api_def_srcs = []):
    # ...

这是通过 tf_gen_op_wrapper_private_py 间接调用的您可以在 tensorflow/python/build_defs.bzl 中找到.对于 array_ops 的情况，您会在 tensorflow/python/BUILD 中找到它:

tf_gen_op_wrapper_private_py(
    name = "array_ops_gen",
    visibility = [
        "//learning/brain/python/ops:__pkg__",
        "//tensorflow/compiler/tests:__pkg__",
        "//tensorflow/contrib/quantization:__pkg__",
        "//tensorflow/python/kernel_tests:__pkg__",
    ],
)

这条规则有什么作用？它调用一个程序，您可以在 tensorflow/python/framework/python_op_gen_main.cc 找到该程序的源代码。 (这是主要入口点，它使用其他相邻的源文件)。本质上，它是一个通过 REGISTER_OP 注册的操作的程序。宏(在 tensorflow/core/framework/op.h 中定义)并相应地生成 Python 代码。我现在无法详细说明，但如果您想了解详细信息，您应该能够浏览代码。