python - 从 C 并行调用 python

Question

我需要从我的 C 代码中调用一个 Python 函数。它工作得很好，但是当我想进行并行化时，它就崩溃了。请参阅以下最小 C 代码:

#include <Python.h>
#include <stdio.h>

int main(void)
{
  double Z = 1.;
  double k = 1.;
  double l = 1.;
  double eta = -Z/k;

  Py_Initialize();

  PyObject* pName = PyString_FromString("mpmath");
  PyObject* pModule = PyImport_Import(pName);
  PyObject* pFunc = PyObject_GetAttrString(pModule, "coulombf");

  PyObject* pl = PyFloat_FromDouble(l);
  PyObject* peta = PyFloat_FromDouble(eta);

  int i;
#pragma omp parallel for private(i)
  for(i=0; i<10000; i++)
  {
    double r = 0.01*i;
    PyObject* prho = PyFloat_FromDouble(k*r);
    PyObject* pArgs = PyTuple_Pack(3, pl, peta, prho);
    PyObject* pValue = PyObject_CallObject(pFunc, pArgs);
    double value = PyFloat_AsDouble(pValue);
    printf("r=%.2f\tf=%.6f\n",r,value);
  }

  Py_Finalize();
  return 0;
}

我们把这个文件命名为testPython.c，你可以用gcc -fopenmp testPython.c -o testPython -I/usr/include/python2.7 -L/usr编译它/lib64/python2.7/config -lpython2.7.

现在使用 ./testPython 运行它，您会看到这样的错误:Fatal Python error: GC object already tracked。 (有时，错误信息会有所不同。)

但是，如果您编译它而将 -fopenmp 排除在外，该程序将完美运行。

我该如何克服这个问题？谢谢!

编辑:

正如 Natecat、John Bollinger 和 Olaf 的回答，多线程不太可能大大加快处理速度，但多处理确实可以加快计算速度。纯python脚本如下:

import numpy
from mpmath import coulombf
from multiprocessing import Pool

Z = 1.
k = 1.
l = 1.
eta = -Z/k

def coulombF(r):
    return coulombf(l,eta,k*r)

pool = Pool(12)
result = pool.map_async(coulombF, numpy.arange(0.,100.,0.01))
print(result.get())

但是我如何在 C 中做到这一点？我还没找到路。

Answer 1

@Natecat 的回答基本上是正确的，只是缺少细节和细微差别。 The docs of Python's C API给一个更完整的画面。假设这是您正在使用的 Python 实现，您需要了解以下内容:

The Python interpreter is not fully thread-safe. In order to support multi-threaded Python programs, there’s a global lock, called the global interpreter lock or GIL, that must be held by the current thread before it can safely access Python objects. Without the lock, even the simplest operations could cause problems in a multi-threaded program [...].

Therefore, the rule exists that only the thread that has acquired the GIL may operate on Python objects or call Python/C API functions. In order to emulate concurrency of execution, the interpreter regularly tries to switch threads (see sys.setswitchinterval()). The lock is also released around potentially blocking I/O operations like reading or writing a file, so that other Python threads can run in the meantime.

和

when threads are created from C (for example by a third-party library with its own thread management), they don’t hold the GIL, nor is there a thread state structure for them.

注意:OpenMP 就是这种情况。

If you need to call Python code from these threads [...] you must first register these threads with the interpreter by creating a thread state data structure, then acquiring the GIL, and finally storing their thread state pointer, before you can start using the Python/C API. When you are done, you should reset the thread state pointer, release the GIL, and finally free the thread state data structure.

The PyGILState_Ensure() and PyGILState_Release() functions do all of the above automatically. The typical idiom for calling into Python from a C thread is:

PyGILState_STATE gstate;
gstate = PyGILState_Ensure();

/* Perform Python actions here. */
result = CallSomeFunction();
/* evaluate result or handle exception */

/* Release the thread. No Python API allowed beyond this point. */
PyGILState_Release(gstate);

您必须实现该模式以允许多个 OpenMP 线程安全地对同一个 CPython 解释器进行并发调用，但是您不太可能从并行化中获得太多好处，因为各种 OpenMP 线程将在很大程度上被阻止并发运行。