python - 从 C 调用 python 回调时如何修复 "SystemError: null argument to internal routine"错误

Question

介绍

我正在编写一个 python 应用程序，它使用一个用 C 编写的库。当某些事件在 C 级别发生时，将调用 Python 回调。

这是我的 python 回调定义的一部分:

def callback(str1, str2, cdata, flag):
  print("PYTHON HANDLER")
  ...
  print(">>EXIT<<")

在将回调 传递给 C 之前，它使用以下代码进行转换:

FuncType = ctypes.CFUNCTYPE(None, ctypes.c_char_p, ctypes.c_char_p,
                            ctypes.c_void_p, ctypes.c_bool)

ctype_function_wrapper = FuncType(callback)
cfunction_pointer = ctypes.cast(ctype_function_wrapper, ctypes.c_void_p)
py_callback_ptr = cfunction_pointer.value

之后 py_callback_ptr 被传递到 C 库并保存为 (void *) 指针。

当事件发生时，调用 C 级 py_callback_ptr。这是一段在 C 级别执行此操作的代码:

...
printf("Acquire GIL\r\n");
PyGILState_STATE gstate;
gstate = PyGILState_Ensure();
printf("Callback()\r\n");
((void (*)(char*, char*, DataStructure*, bool))py_callback_ptr)(
    str1, str2, value, bool_flag);
printf("Callback return\r\n");
PyGILState_Release(gstate);
printf("Release GIL\r\n");
...

大部分时间它按预期工作并调用回调。我在标准输出中得到这个输出:

...
Acquire GIL
Callback()
PYTHON HANDLER
>>EXIT<<
Callback return
Release GIL
...

但有时我会将此输出与 stderr 输出混合在一起。它向我展示了回调周围的 C 代码正常执行但回调本身没有执行:

...
Acquire GIL
Callback()
SystemError: null argument to internal routine      # from stderr
Callback return
Release GIL
...

研究

我在 Python 和 found the explanation 中寻找过 SystemError

Raised when the interpreter finds an internal error, but the situation does not look so serious to cause it to abandon all hope. The associated value is a string indicating what went wrong (in low-level terms).

所以这是在向 stderr 写入一些字符串的解释器代码中捕获的错误。我决定接听电话并进行调查。

调试

我试图在 gdb 中捕获这个错误。我已经设置了一个断点，它使用以下命令停止在 stderr 输出上执行程序:

(gdb) b write if 2==$rdi

还捕获了正常回调执行的回溯。他们在这里。

正常回溯的一部分:

#0  write () at ../sysdeps/unix/syscall-template.S:81
#1  0x0000000000437735 in _Py_write_impl (gil_held=1, count=4, buf=<optimized out>, fd=2) at Python/fileutils.c:1331
#2  _Py_write (fd=2, buf=0x961100, count=<optimized out>) at Python/fileutils.c:1393
#3  0x00000000005b73ce in _io_FileIO_write_impl (b=0x7ffff3916b00, b=0x7ffff3916b00, self=0x7ffff7f0c3a8) at ./Modules/_io/fileio.c:857
#4  _io_FileIO_write (self=0x7ffff7f0c3a8, arg=<optimized out>) at ./Modules/_io/clinic/fileio.c.h:251
#5  0x00000000004aa139 in _PyCFunction_FastCallDict (func_obj=func_obj@entry=0x7ffff4f9d828, args=args@entry=0x7ffff3916c40, nargs=nargs@entry=1, kwargs=kwargs@entry=0x0) at Objects/methodobject.c:209
#6  0x0000000000451b68 in _PyObject_FastCallDict (func=0x7ffff4f9d828, args=0x7ffff3916c40, nargs=1, kwargs=0x0) at Objects/abstract.c:2313
#7  0x0000000000452b86 in PyObject_CallMethodObjArgs (callable=0x7ffff4f9d828, name=<optimized out>) at Objects/abstract.c:2759
#8  0x00000000005bb24a in _bufferedwriter_raw_write (self=self@entry=0x7ffff7fd0b48, start=<optimized out>, len=4) at ./Modules/_io/bufferedio.c:1853
#9  0x00000000005bbd0f in _bufferedwriter_flush_unlocked (self=self@entry=0x7ffff7fd0b48) at ./Modules/_io/bufferedio.c:1898
#10 buffered_flush_and_rewind_unlocked (self=self@entry=0x7ffff7fd0b48) at ./Modules/_io/bufferedio.c:827
#11 0x00000000005bbf27 in buffered_flush (self=0x7ffff7fd0b48, args=<optimized out>) at ./Modules/_io/bufferedio.c:854
#12 0x00000000004aa1ff in _PyCFunction_FastCallDict (func_obj=func_obj@entry=0x7ffff4f9d5a0, args=args@entry=0x7ffff3916f30, nargs=nargs@entry=0, kwargs=kwargs@entry=0x0) at Objects/methodobject.c:192
#13 0x0000000000451b68 in _PyObject_FastCallDict (func=0x7ffff4f9d5a0, args=0x7ffff3916f30, nargs=0, kwargs=0x0) at Objects/abstract.c:2313
#14 0x0000000000452b86 in PyObject_CallMethodObjArgs (callable=0x7ffff4f9d5a0, name=<optimized out>) at Objects/abstract.c:2759
#15 0x00000000005c1924 in _io_TextIOWrapper_write_impl (text=0x7ffff7f30a40, self=0x7ffff7f7d708) at ./Modules/_io/textio.c:1367
#16 _io_TextIOWrapper_write (self=0x7ffff7f7d708, arg=<optimized out>) at ./Modules/_io/clinic/textio.c.h:216
#17 0x00000000004aa139 in _PyCFunction_FastCallDict (func_obj=func_obj@entry=0x7ffff4f9d5e8, args=args@entry=0x7ffff3917148, nargs=nargs@entry=1, kwargs=kwargs@entry=0x0) at Objects/methodobject.c:209
#18 0x0000000000451b68 in _PyObject_FastCallDict (func=func@entry=0x7ffff4f9d5e8, args=args@entry=0x7ffff3917148, nargs=nargs@entry=1, kwargs=kwargs@entry=0x0) at Objects/abstract.c:2313
#19 0x000000000047a4cc in PyFile_WriteObject (flags=1, f=<optimized out>, v=0x7ffff7f30a40) at Objects/fileobject.c:150
#20 PyFile_WriteString (s=s@entry=0x5e4ac5 "\n", f=<optimized out>) at Objects/fileobject.c:174
#21 0x000000000053b103 in builtin_print (self=<optimized out>, args=0x7ffff65df748, kwds=<optimized out>) at Python/bltinmodule.c:1814
#22 0x00000000004aa3d9 in PyCFunction_Call (func=func@entry=0x7ffff7fcdf30, args=args@entry=0x7ffff65df748, kwds=kwds@entry=0x7ffff4f9d8b8) at Objects/methodobject.c:98
#23 0x0000000000548572 in do_call_core (kwdict=0x7ffff4f9d8b8, callargs=0x7ffff65df748, func=0x7ffff7fcdf30) at Python/ceval.c:5116
#24 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3404
#25 0x000000000053f401 in PyEval_EvalFrameEx (throwflag=0, f=0x7ffff4fa1048) at Python/ceval.c:754
#26 _PyEval_EvalCodeWithName (_co=0x7ffff7f089c0, globals=globals@entry=0x7ffff7f4c360, locals=locals@entry=0x0, args=<optimized out>, argcount=1, kwnames=0x0, kwargs=kwargs@entry=0x7ffff7ed4d60, kwcount=0, kwstep=kwstep@entry=1, 
    defs=0x0, defcount=defcount@entry=0, kwdefs=0x0, closure=0x0, name=name@entry=0x7ffff7f0f3b0, qualname=0x7ffff7f0f3b0) at Python/ceval.c:4166
#27 0x000000000053f6ff in fast_function (kwnames=0x0, nargs=<optimized out>, stack=<optimized out>, func=0x7ffff7f36e18) at Python/ceval.c:4992
#28 call_function (pp_stack=pp_stack@entry=0x7ffff3917550, oparg=<optimized out>, kwnames=kwnames@entry=0x0) at Python/ceval.c:4872
#29 0x000000000054138b in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3335
#30 0x000000000053e791 in PyEval_EvalFrameEx (throwflag=0, f=0x7ffff7ed4bb8) at Python/ceval.c:754
#31 _PyFunction_FastCall (co=<optimized out>, args=<optimized out>, nargs=4, globals=globals@entry=0x7ffff7f4c360) at Python/ceval.c:4933
#32 0x000000000053f927 in fast_function (kwnames=0x0, nargs=<optimized out>, stack=<optimized out>, func=0x7ffff4f9ed08) at Python/ceval.c:4968
#33 call_function (pp_stack=pp_stack@entry=0x7ffff3917770, oparg=<optimized out>, kwnames=kwnames@entry=0x0) at Python/ceval.c:4872
#34 0x000000000054138b in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3335
#35 0x000000000053f401 in PyEval_EvalFrameEx (throwflag=0, f=0x7ffff4fa0048) at Python/ceval.c:754
#36 _PyEval_EvalCodeWithName (_co=0x7ffff65d0810, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0x7ffff6569330, argcount=4, kwnames=kwnames@entry=0x0, kwargs=kwargs@entry=0x0, kwcount=kwcount@entry=0, 
    kwstep=kwstep@entry=2, defs=defs@entry=0x0, defcount=defcount@entry=0, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x7ffff7e9e208, name=name@entry=0x0, qualname=qualname@entry=0x0) at Python/ceval.c:4166
#37 0x00000000005401dd in PyEval_EvalCodeEx (_co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0x7ffff6569330, argcount=<optimized out>, kws=kws@entry=0x0, kwcount=kwcount@entry=0, 
    defs=defs@entry=0x0, defcount=defcount@entry=0, kwdefs=0x0, closure=0x7ffff7e9e208) at Python/ceval.c:4187
#38 0x000000000048101c in function_call (func=0x7ffff4f9eea0, arg=0x7ffff6569318, kw=0x0) at Objects/funcobject.c:604
#39 0x0000000000451820 in PyObject_Call (func=0x7ffff4f9eea0, args=args@entry=0x7ffff6569318, kwargs=<optimized out>) at Objects/abstract.c:2261
#40 0x0000000000540696 in PyEval_CallObjectWithKeywords (func=func@entry=0x7ffff4f9eea0, args=args@entry=0x7ffff6569318, kwargs=kwargs@entry=0x0) at Python/ceval.c:4771
#41 0x00000000004516e7 in PyObject_CallObject (o=o@entry=0x7ffff4f9eea0, a=a@entry=0x7ffff6569318) at Objects/abstract.c:2187
#42 0x00007ffff4d54d2c in _CallPythonObject (pArgs=<optimized out>, flags=257, converters=0x7ffff656a908, callable=0x7ffff4f9eea0, setfunc=0x0, restype=0x7ffff4f631d0 <ffi_type_void>, mem=0x7ffff3917c20)
    at /home/ns-sergeev/.pythons/src/Python-3.6.8/Modules/_ctypes/callbacks.c:234
#43 closure_fcn (cif=<optimized out>, resp=0x7ffff3917c20, args=0x7ffff3917a80, userdata=<optimized out>) at /home/ns-sergeev/.pythons/src/Python-3.6.8/Modules/_ctypes/callbacks.c:296
#44 0x00007ffff4b42a6b in ffi_closure_unix64_inner (closure=0x7ffff7ff2010, rvalue=0x7ffff3917c20, reg_args=0x7ffff3917b70, argp=0x7ffff3917c40 "P\r") at ../src/x86/ffi64.c:667
#45 0x00007ffff4b42de4 in ffi_closure_unix64 () at ../src/x86/unix64.S:229
...
C-library frames are omitted

回调调用错误的回溯:

#0  write () at ../sysdeps/unix/syscall-template.S:81
#1  0x0000000000437735 in _Py_write_impl (gil_held=1, count=47, buf=<optimized out>, fd=2) at Python/fileutils.c:1331
#2  _Py_write (fd=2, buf=0x961100, count=<optimized out>) at Python/fileutils.c:1393
#3  0x00000000005b73ce in _io_FileIO_write_impl (b=0x7ffff3916e80, b=0x7ffff3916e80, self=0x7ffff7f0c3a8) at ./Modules/_io/fileio.c:857
#4  _io_FileIO_write (self=0x7ffff7f0c3a8, arg=<optimized out>) at ./Modules/_io/clinic/fileio.c.h:251
#5  0x00000000004aa139 in _PyCFunction_FastCallDict (func_obj=func_obj@entry=0x7ffff4f9e7e0, args=args@entry=0x7ffff3916fc0, nargs=nargs@entry=1, 
    kwargs=kwargs@entry=0x0) at Objects/methodobject.c:209
#6  0x0000000000451b68 in _PyObject_FastCallDict (func=0x7ffff4f9e7e0, args=0x7ffff3916fc0, nargs=1, kwargs=0x0) at Objects/abstract.c:2313
#7  0x0000000000452b86 in PyObject_CallMethodObjArgs (callable=0x7ffff4f9e7e0, name=<optimized out>) at Objects/abstract.c:2759
#8  0x00000000005bb24a in _bufferedwriter_raw_write (self=self@entry=0x7ffff7fd0b48, start=<optimized out>, len=47) at ./Modules/_io/bufferedio.c:1853
#9  0x00000000005bbd0f in _bufferedwriter_flush_unlocked (self=self@entry=0x7ffff7fd0b48) at ./Modules/_io/bufferedio.c:1898
#10 buffered_flush_and_rewind_unlocked (self=self@entry=0x7ffff7fd0b48) at ./Modules/_io/bufferedio.c:827
#11 0x00000000005bbf27 in buffered_flush (self=0x7ffff7fd0b48, args=<optimized out>) at ./Modules/_io/bufferedio.c:854
#12 0x00000000004aa1ff in _PyCFunction_FastCallDict (func_obj=func_obj@entry=0x7ffff4f9e558, args=args@entry=0x7ffff39172b0, nargs=nargs@entry=0, 
    kwargs=kwargs@entry=0x0) at Objects/methodobject.c:192
#13 0x0000000000451b68 in _PyObject_FastCallDict (func=0x7ffff4f9e558, args=0x7ffff39172b0, nargs=0, kwargs=0x0) at Objects/abstract.c:2313
#14 0x0000000000452b86 in PyObject_CallMethodObjArgs (callable=0x7ffff4f9e558, name=<optimized out>) at Objects/abstract.c:2759
#15 0x00000000005c1924 in _io_TextIOWrapper_write_impl (text=0x7ffff7f30a40, self=0x7ffff7f7d708) at ./Modules/_io/textio.c:1367
#16 _io_TextIOWrapper_write (self=0x7ffff7f7d708, arg=<optimized out>) at ./Modules/_io/clinic/textio.c.h:216
#17 0x00000000004aa139 in _PyCFunction_FastCallDict (func_obj=func_obj@entry=0x7ffff4f9e5a0, args=args@entry=0x7ffff39174c8, nargs=nargs@entry=1, 
    kwargs=kwargs@entry=0x0) at Objects/methodobject.c:209
#18 0x0000000000451b68 in _PyObject_FastCallDict (func=func@entry=0x7ffff4f9e5a0, args=args@entry=0x7ffff39174c8, nargs=nargs@entry=1, 
    kwargs=kwargs@entry=0x0) at Objects/abstract.c:2313
#19 0x000000000047a4cc in PyFile_WriteObject (flags=1, f=0x7ffff7f30a40, v=0x7ffff7f30a40) at Objects/fileobject.c:150
#20 PyFile_WriteString (s=s@entry=0x5e4ac5 "\n", f=f@entry=0x7ffff7f7d708) at Objects/fileobject.c:174
#21 0x0000000000425790 in print_exception (value=<optimized out>, f=0x7ffff7f7d708) at Python/pythonrun.c:819
#22 print_exception_recursive (f=f@entry=0x7ffff7f7d708, value=value@entry=0x7ffff4f9a1a8, seen=seen@entry=0x7ffff650ee48) at Python/pythonrun.c:891
---Type <return> to continue, or q <return> to quit---
#23 0x0000000000426683 in PyErr_Display (exception=<optimized out>, value=0x7ffff4f9a1a8, tb=<optimized out>) at Python/pythonrun.c:925
#24 0x000000000043199a in sys_excepthook (self=<optimized out>, args=<optimized out>) at ./Python/sysmodule.c:230
#25 0x00000000004aa1ac in _PyCFunction_FastCallDict (func_obj=func_obj@entry=0x7ffff7f61f78, args=args@entry=0x7ffff39176c0, nargs=nargs@entry=3, 
    kwargs=kwargs@entry=0x0) at Objects/methodobject.c:234
#26 0x0000000000451b68 in _PyObject_FastCallDict (func=0x7ffff7f61f78, args=args@entry=0x7ffff39176c0, nargs=nargs@entry=3, kwargs=kwargs@entry=0x0)
    at Objects/abstract.c:2313
#27 0x00000000004267f5 in PyErr_PrintEx (set_sys_last_vars=set_sys_last_vars@entry=1) at Python/pythonrun.c:669
#28 0x0000000000426a9a in PyErr_Print () at Python/pythonrun.c:532
#29 0x00007ffff4d54b78 in PrintError (msg=msg@entry=0x7ffff4d5bc7a "BUG: PySequence_Length")
    at /home/ns-sergeev/.pythons/src/Python-3.6.8/Modules/_ctypes/callbacks.c:91
#30 0x00007ffff4d54e8e in _CallPythonObject (pArgs=0x7ffff3917a80, flags=4353, converters=0x0, callable=0x0, setfunc=0x0, 
    restype=0x7ffff4f631d0 <ffi_type_void>, mem=0x7ffff3917c20) at /home/ns-sergeev/.pythons/src/Python-3.6.8/Modules/_ctypes/callbacks.c:149
#31 closure_fcn (cif=<optimized out>, resp=0x7ffff3917c20, args=0x7ffff3917a80, userdata=<optimized out>)
    at /home/ns-sergeev/.pythons/src/Python-3.6.8/Modules/_ctypes/callbacks.c:296
#32 0x00007ffff4b42a6b in ffi_closure_unix64_inner (closure=0x7ffff7ff0cc0, rvalue=0x7ffff3917c20, reg_args=0x7ffff3917b70, 
    argp=0x7ffff3917c40 "\240\021\243") at ../src/x86/ffi64.c:667
#33 0x00007ffff4b42de4 in ffi_closure_unix64 () at ../src/x86/unix64.S:229
...
C-library frames are omitted

初步调查结果

似乎 _CallPythonObject 的参数 converters 和 callable 不正常(由于某种原因更改或损坏)。

来自正常回溯的第 42 帧。正常值和调用的回调。

#42 0x00007ffff4d54d2c in _CallPythonObject (pArgs=<optimized out>, flags=257,
    converters=0x7ffff656a908, callable=0x7ffff4f9eea0, setfunc=0x0,
    restype=0x7ffff4f631d0 <ffi_type_void>, mem=0x7ffff3917c20)

来自错误回溯的第 30 帧。零值和回调被放弃。

#30 0x00007ffff4d54e8e in _CallPythonObject (pArgs=0x7ffff3917a80, flags=4353, 
    converters=0x0, callable=0x0, setfunc=0x0,
    restype=0x7ffff4f631d0 <ffi_type_void>, mem=0x7ffff3917c20)

有人可以帮助解决这个问题或建议进一步挖掘和调试它的方法吗？

Answer 1

您应该将 cfunction_pointer 而不是 py_callback_ptr = cfunction_pointer.value 传递给您的库。

最小工作示例:

cside.h:

#pragma once

#ifdef __cplusplus
extern "C" {
#endif

typedef struct DataStructure {
  int a;
} DataStructure;

typedef void (*callback)(char*, char*, DataStructure*, bool);

typedef void (*pcallback)(char*, char*, void*, bool);

int calc0(int, callback cb);

int calc1(int, pcallback cb);


#ifdef __cplusplus
}
#endif

请注意，您不需要在 C 代码中的任何地方定义将 DataStructure 替换为 void 的签名。无论如何，我已经将它包括在内以使用您问题中的类型转换。

cside.cpp:

#include "cside.h"
#include "Python.h"

const char cstring1[] = "foo";
const char cstring2[] = "bar";

const bool bool_flag = false;

char* str1 = const_cast<char*>(&cstring1[0]);
char* str2 = const_cast<char*>(&cstring2[0]);

int calc0(int k, callback cb) {
  printf("Acquire GIL\r\n");
  PyGILState_STATE gstate;
  gstate = PyGILState_Ensure();
  printf("Callback()\r\n");

  DataStructure s;

  // Execute callback with the signature known in C
  cb(str1, str2, &s, bool_flag);

  printf("Callback return\r\n");
  PyGILState_Release(gstate);
  printf("Release GIL\r\n");
  return k+2;
}

int calc1(int k, pcallback cb) {
  printf("Acquire GIL\r\n");
  PyGILState_STATE gstate;
  gstate = PyGILState_Ensure();
  printf("Callback()\r\n");

  DataStructure s;

  // Cast to correct signature
  ((void (*)(char*, char*, DataStructure*, bool))cb)(
      str1, str2, &s, bool_flag);

  printf("Callback return\r\n");
  PyGILState_Release(gstate);
  printf("Release GIL\r\n");
  return k+2;
}

编译:

g++ -I. -fPIC -shared -o libcside.so cside.cpp -I/usr/include/python2.7 -lpython2.7

在 Python 中测试:

import ctypes

libc = ctypes.CDLL('libcside.so')

def callback(str1, str2, cdata, flag):
  print("PYTHON HANDLER")
  print(">>EXIT<<")

FuncType = ctypes.CFUNCTYPE(None, ctypes.c_char_p, ctypes.c_char_p,
                            ctypes.c_void_p, ctypes.c_bool)

ctype_function_wrapper = FuncType(callback)
cfunction_pointer = ctypes.cast(ctype_function_wrapper, ctypes.c_void_p)
py_callback_ptr = cfunction_pointer.value

libc.calc0(2, cfunction_pointer)
libc.calc1(2, cfunction_pointer)

#libc.calc(2, py_callback_ptr) # segfaults