python - 嵌入 Python、在 Grand Central Dispatch 中使用非 Python 线程会导致 Py_Finalize 错误

Question

这发生在 Python 3.4.3 中，来自非 python 创建的线程:

https://docs.python.org/3.4/c-api/init.html#non-python-created-threads

有几个关于 SO 的问题，可以从 C/C++ 应用程序中查看类似的问题。

AssertionError (3.X only) when calling Py_Finalize with threads

Fatal error during Py_Finalize in embedded Python application

PyEval_InitThreads in Python 3: How/when to call it? (the saga continues ad nauseum)

但这些都没有专门涉及 Grand Central Dispatch。我不知道这是否重要，因为它可能只是引擎盖下的线程。

然而，尝试应用这些帖子中的知识仍然给我带来了问题。

这就是我目前所处的位置，我有一个代表我的 Python 运行时的 obj-c 类，并且我有以下相关方法:

- (void)initialize
{
    Py_Initialize();
    PyEval_InitThreads();

    PyObject* sysPath = PySys_GetObject((char*)"path");

    for(NSString * path in self.pythonPath){
        PyList_Append(sysPath, objc_convert_string(path));
    }

    // not calling PyEval_SaveThread 
    // causes beginTask below to die on 
    // PyEval_AcquireThread

    // self.threadState = PyThreadState_Get();
    // release the GIL, this shouldn't need to be
    // done, as I understand it, 
    // but as the comment above states, if I don't
    // beginTask will fail at PyEval_AcquireThread
    self.threadState = PyEval_SaveThread();
    self.running = YES;
}

这就是我初始化 Python 的方式。然后我通过以下方式调用 python 命令:

- (void)beginTask:(nonnull void (^)(void))task completion:(nullable void (^)(void))completion
{
    dispatch_async(self.pythonQueue, ^{

        PyInterpreterState * mainInterpreterState = self.threadState->interp;
        PyThreadState * myThreadState = PyThreadState_New(mainInterpreterState);
        PyEval_AcquireThread(myThreadState);

        // Perform any Py_* related functions here
        task();

        PyEval_ReleaseThread(PyThreadState_Get());

        if (completion){
            dispatch_async(dispatch_get_main_queue(), ^{
                completion();
            });
        }

    });
}

在进行长时间运行的操作的情况下，我有一些代码执行相当多的 Jinja 模板渲染并保存到文件系统。我想在完成后进行清理，因此我调用 Py_Finalize 然后重新初始化，使用上述方法输入我的问题:

- (void)finalize
{
    PyEval_RestoreThread(self.threadState);

    // Problems here
    Py_Finalize();

    self.running = NO;
}

这会导致 Python 中出现以下错误:

Exception ignored in: <module 'threading' from '/Users/blitz6/Library/Developer/Xcode/DerivedData/Live-cblwxzmbsdpraeebpregeuhikwzh/Build/Products/Debug/LivePython.framework/Resources/python3.4/lib/python3.4/threading.py'>
Traceback (most recent call last):
  File "/Users/blitz6/Library/Developer/Xcode/DerivedData/Live-cblwxzmbsdpraeebpregeuhikwzh/Build/Products/Debug/LivePython.framework/Resources/python3.4/lib/python3.4/threading.py", line 1296, in _shutdown
    _main_thread._delete()
  File "/Users/blitz6/Library/Developer/Xcode/DerivedData/Live-cblwxzmbsdpraeebpregeuhikwzh/Build/Products/Debug/LivePython.framework/Resources/python3.4/lib/python3.4/threading.py", line 1019, in _delete
    del _active[get_ident()]
KeyError: 140735266947840

我尝试了几种不同的方法来处理这个问题，包括在 beginTask 中使用 PyGILState_Ensure() 和 PyGILState_Release(myState);:

- (void)beginTask:(nonnull void (^)(void))task completion:(nullable void (^)(void))completion
{

    dispatch_async(self.pythonQueue, ^{

        PyGILState_STATE state = PyGILState_Ensure();

        task();

        PyGILState_Release(state);

        if (completion){
            dispatch_async(dispatch_get_main_queue(), ^{
                completion();
            });
        }
    });
}

但这会导致上面的finalize方法中出现此错误:

Exception ignored in: <module 'threading' from '/Users/blitz6/Library/Developer/Xcode/DerivedData/Live-cblwxzmbsdpraeebpregeuhikwzh/Build/Products/Debug/LivePython.framework/Resources/python3.4/lib/python3.4/threading.py'>
Traceback (most recent call last):
  File "/Users/blitz6/Library/Developer/Xcode/DerivedData/Live-cblwxzmbsdpraeebpregeuhikwzh/Build/Products/Debug/LivePython.framework/Resources/python3.4/lib/python3.4/threading.py", line 1289, in _shutdown
    assert tlock.locked()
AssertionError: 

所以我几乎陷入了困境:如何Py_Finalize而不出现某种错误。显然我不明白一些事情。当我到达断点时，我还可以通过 xcode 确认我的 dispatch_async block 正在从非主线程的另一个线程运行。

更新

修补了一下，我发现，这个:

PyObject* module = PyImport_ImportModule("requests");

当我在另一个线程上时，当我 Py_Finalize

时会导致此错误

Exception ignored in: <module 'threading' from '/Users/blitz6/Library/Developer/Xcode/DerivedData/Live-cblwxzmbsdpraeebpregeuhikwzh/Build/Products/Debug/LivePython.framework/Resources/python3.4/lib/python3.4/threading.py'>
Traceback (most recent call last):
  File "/Users/blitz6/Library/Developer/Xcode/DerivedData/Live-cblwxzmbsdpraeebpregeuhikwzh/Build/Products/Debug/LivePython.framework/Resources/python3.4/lib/python3.4/threading.py", line 1296, in _shutdown
    _main_thread._delete()
  File "/Users/blitz6/Library/Developer/Xcode/DerivedData/Live-cblwxzmbsdpraeebpregeuhikwzh/Build/Products/Debug/LivePython.framework/Resources/python3.4/lib/python3.4/threading.py", line 1019, in _delete
    del _active[get_ident()]
KeyError: 140735266947840

如果我导入:

PyObject* module = PyImport_ImportModule("os");
OR
PyObject* module = PyImport_ImportModule("json");

作为示例，一切都运行良好。当我开始导入自己的模块时，我遇到了问题。

在Py_Finalize内部，wait_for_thread_shutdown();是我遇到这个问题的地方。我猜，根据评论，这与:

/* Wait until threading._shutdown completes, provided the threading module was imported in the first place. The shutdown routine will wait until all non-daemon "threading" threads have completed. */

具体在wait_for_thread_shutdown中:

PyThreadState *tstate = PyThreadState_GET();
PyObject *threading = PyMapping_GetItemString(tstate->interp->modules,
                                              "threading");
if (threading == NULL) {
    /* threading not imported */
    PyErr_Clear();
    return;
}

tstate 返回 NULL，但 threading 不是 NULL，它会跳过 PyErr_Clear() 代码路径并执行:

result = _PyObject_CallMethodId(threading, &PyId__shutdown, "");
if (result == NULL) {
    PyErr_WriteUnraisable(threading);
}
else {
    Py_DECREF(result);
}
Py_DECREF(threading);

如果我只是:

PyObject* module = PyImport_ImportModule("json");

然后，PyErr_WriteUnraisable(threading);在wait_for_thread_shutdown中执行

result = _PyObject_CallMethodId(threading, &PyId__shutdown, "");
if (result == NULL) {
    PyErr_WriteUnraisable(threading);
}

Answer 1

如前所述，我正在思考我的代码，并且错误地思考了嵌入式 Python 解释器。它与 GCD 无关，而与我对嵌入式 Python 某些方面的误解有关。我可能仍然有误解，但是下面的逻辑符合我对我认为应该做的事情的期望，并且我的错误消失了。

我试图做的是运行一次性任务，而不是保留任何东西。我认为我需要在这里线程，这让我绊倒了。

我正在做的事情是获取 GIL，然后导入一些使用 threading 模块的 Python 代码。当您这样做时，解释器会注册您已引入线程模块，就像当您 Py_Finalize 时，它会跳过一些循环以确保所有可能存在或不存在的子线程都被关闭向下。我实际上是把地毯从下面拉出来，这导致了我的错误。相反，我需要完成的工作更有利于 Py_NewInterpreter。当您调用 Py_EndInterpreter 时，它会运行与 Py_Finalize 完全相同的线程关闭过程，但与自身隔离。

所以我最终的 GCD 一劳永逸代码如下所示:

初始化

- (void)initialize
{
    Py_Initialize();
    PyEval_InitThreads();

    [self updateSysPath];

    // Release the GIL
    self.threadState = PyEval_SaveThread();
    self.running = YES;

}

任务执行

- (void)beginTask:(nonnull void (^)(Runtime * __nonnull))task completion:(nullable void (^)(Runtime * __nonnull))completion
{
    dispatch_async(self.pythonQueue, ^{

        PyInterpreterState * mainInterpreterState = self.threadState->interp;
        PyThreadState * taskState = PyThreadState_New(mainInterpreterState);

        // Acquire the GIL
        PyEval_AcquireThread(taskState);

        PyThreadState* tstate = Py_NewInterpreter();
        [self updateSysPath];

        task(self);

        // when Py_EndInterpreter is called, the current 
        // thread state is set to NULL, 
        // so we need to put ourselves back on
        // the taskState, and release the GIL
        Py_EndInterpreter(tstate);
        PyThreadState_Swap(taskState);

        // release the GIL
        PyEval_ReleaseThread(taskState);

        // cleanup
        PyThreadState_Clear(taskState);
        PyThreadState_Delete(taskState);

        if (completion){
            dispatch_async(dispatch_get_main_queue(), ^{
                completion(self);
            });
        }
    });
}

最终确定

- (void)finalize
{
    // acquire the GIL
    PyEval_RestoreThread(self.threadState);
    Py_Finalize();
    self.running = NO;
}