python - 模块属性更新不会传播到 Windows 上的子进程

Question

我遇到了一些与 Windows 上的模块属性更新没有传播到 Windows 上的子进程相关的问题。

下面的片段说明了这个问题:

import functools
import multiprocessing
import os
from contextlib import contextmanager

_DOMAIN_RANGE_SCALE = 'reference'


def get_domain_range_scale():
    return _DOMAIN_RANGE_SCALE


def set_domain_range_scale(scale='Reference'):
    global _DOMAIN_RANGE_SCALE

    scale = str(scale).lower()

    _DOMAIN_RANGE_SCALE = scale


class domain_range_scale(object):
    def __init__(self, scale):
        self._scale = scale
        self._previous_scale = get_domain_range_scale()

    def __enter__(self):
        set_domain_range_scale(self._scale)

        return self

    def __exit__(self, *args):
        set_domain_range_scale(self._previous_scale)

    def __call__(self, function):
        @functools.wraps(function)
        def wrapper(*args, **kwargs):
            with self:
                return function(*args, **kwargs)

        return wrapper


@contextmanager
def multiprocessing_pool(*args, **kwargs):
    pool = multiprocessing.Pool(*args, **kwargs)

    yield pool

    pool.terminate()


def test_domain_range_scale(*args):
    print('Domain Range Scale Inner: {0}, PID: {1}'.format(
        get_domain_range_scale(), os.getpid()))


if __name__ == '__main__':
    for scale in ('reference', '1', '100'):
        with domain_range_scale(scale):
            print('*' * 79)
            print('Domain Range Scale Outer: {0}, PID: {1}'.format(
                get_domain_range_scale(), os.getpid()))
            with multiprocessing_pool(processes=4) as pool:
                pool.map(test_domain_range_scale, range(10))

Linux/macOS 上的输出

*******************************************************************************
Domain Range Scale Outer: reference, PID: 93989
Domain Range Scale Inner: reference, PID: 93990
Domain Range Scale Inner: reference, PID: 93992
Domain Range Scale Inner: reference, PID: 93993
Domain Range Scale Inner: reference, PID: 93991
Domain Range Scale Inner: reference, PID: 93990
Domain Range Scale Inner: reference, PID: 93991
Domain Range Scale Inner: reference, PID: 93990
Domain Range Scale Inner: reference, PID: 93993
Domain Range Scale Inner: reference, PID: 93991
Domain Range Scale Inner: reference, PID: 93992
*******************************************************************************
Domain Range Scale Outer: 1, PID: 93989
Domain Range Scale Inner: 1, PID: 93994
Domain Range Scale Inner: 1, PID: 93995
Domain Range Scale Inner: 1, PID: 93996
Domain Range Scale Inner: 1, PID: 93997
Domain Range Scale Inner: 1, PID: 93994
Domain Range Scale Inner: 1, PID: 93995
Domain Range Scale Inner: 1, PID: 93996
Domain Range Scale Inner: 1, PID: 93994
Domain Range Scale Inner: 1, PID: 93997
Domain Range Scale Inner: 1, PID: 93995
*******************************************************************************
Domain Range Scale Outer: 100, PID: 93989
Domain Range Scale Inner: 100, PID: 93998
Domain Range Scale Inner: 100, PID: 93999
Domain Range Scale Inner: 100, PID: 94000
Domain Range Scale Inner: 100, PID: 94001
Domain Range Scale Inner: 100, PID: 93998
Domain Range Scale Inner: 100, PID: 93999
Domain Range Scale Inner: 100, PID: 94000
Domain Range Scale Inner: 100, PID: 94001
Domain Range Scale Inner: 100, PID: 93998
Domain Range Scale Inner: 100, PID: 93999

在 Windows 上输出

*******************************************************************************
Domain Range Scale Outer: reference, PID: 6524
Domain Range Scale Inner: reference, PID: 2124
Domain Range Scale Inner: reference, PID: 2124
Domain Range Scale Inner: reference, PID: 5476
Domain Range Scale Inner: reference, PID: 4872
Domain Range Scale Inner: reference, PID: 1932
*******************************************************************************
Domain Range Scale Outer: 1, PID: 6524
Domain Range Scale Inner: reference, PID: 2716
Domain Range Scale Inner: reference, PID: 2716
Domain Range Scale Inner: reference, PID: 1012
Domain Range Scale Inner: reference, PID: 1852
Domain Range Scale Inner: reference, PID: 6544
*******************************************************************************
Domain Range Scale Outer: 100, PID: 6524
Domain Range Scale Inner: reference, PID: 7456
Domain Range Scale Inner: reference, PID: 7456
Domain Range Scale Inner: reference, PID: 7456
Domain Range Scale Inner: reference, PID: 7456
Domain Range Scale Inner: reference, PID: 5944

Answer 1

您的问题在于 Windows 不支持“fork”作为新进程的启动方法(仅支持“spawn”)。全局变量不随“spawn”继承。当您在 _DOMAIN_RANGE_SCALE = 'reference' 下方放置打印语句时，您将看到 Windows 上的子进程将再次运行该脚本直到 if __name__ == '__main__': 当他们导入需要的函数时。

您必须使用 Pool 的initializer参数在进程启动后显式注册全局变量。

...

def init_global(scale):
    global _DOMAIN_RANGE_SCALE
    _DOMAIN_RANGE_SCALE = scale

if __name__ == '__main__':

...
        with multiprocessing.Pool(processes=4,
                                  initializer=init_global,
                                  initargs=(scale,)) as pool:

            pool.map(test_domain_range_scale, range(10))
...