python-3.x - python3.PyImport_ImportModule(name) 在第二次调用时会发出 fatal error

Question

环境 :

MacOS(Catalina 版本 10.15.4)

Python3.7.6

Go1.13.8

我想使用 go-python3 调用一个用 Python3 编写的算法，但是如上所述，当我第二次调用这个算法时会产生一个 fatal error 。从输出消息来看，似乎是 PyImport_ImportModule 导致了这个错误。

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0xa pc=0x91256a3]

runtime stack:
runtime.throw(0x4967a75, 0x2a)
        /usr/local/go/src/runtime/panic.go:774 +0x72
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:378 +0x47c

goroutine 41 [syscall]:
runtime.cgocall(0x4637740, 0xc000063c18, 0x48a4660)
        /usr/local/go/src/runtime/cgocall.go:128 +0x5b fp=0xc000063be8 sp=0xc000063bb0 pc=0x4004d0b
github.com/DataDog/go-python3._Cfunc_PyImport_ImportModule(0x8061d90, 0x0)
        _cgo_gotypes.go:3780 +0x4a fp=0xc000063c18 sp=0xc000063be8 pc=0x462c2fa
github.com/DataDog/go-python3.PyImport_ImportModule(0x49501f5, 0x8, 0x0)
        /Users/zhao/go/pkg/mod/github.com/!data!dog/go-python3@v0.0.0-20191126174558-6ed25e33b3c4/import.go:24 +0x87 fp=0xc000063c80 sp=0xc000063c18 pc=0x462e267
PPGServer/pkg/algo.ImportModule(0x4964926, 0x26, 0x49501f5, 0x8, 0x1)
        /Users/zhao/go/src/PPGServer/pkg/algo/ppg.go:42 +0x4cb fp=0xc000063d98 sp=0xc000063c80 pc=0x46332db
PPGServer/pkg/algo.CalcPre(0xc0003560c0, 0xd, 0x0, 0x0)
....

这是示例代码。
PyImport_ImportModule 的包装器 :

// ImportModule will import python module from given directory
func ImportModule(dir, name string) *python3.PyObject {
    fmt.Println("python3.PyImport_ImportModule before")
    sysModule := python3.PyImport_ImportModule("sys") // import sys
    fmt.Println("python3.PyImport_ImportModule success")
    path := sysModule.GetAttrString("path")           // path = sys.path
    ob := python3.PyList_GetItem(path, 1)
    fmt.Println("check:", python3.PyUnicode_Check(ob))
    fmt.Println("path:", python3.PyUnicode_AsUTF8(ob))
    fmt.Println("sysModule.GetAttrString success")
    python3.PyList_Insert(path, 0, python3.PyUnicode_FromString("/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages"))
    python3.PyList_Insert(path, 0, python3.PyUnicode_FromString(dir))
    fmt.Println("After module insert:", python3.PyUnicode_AsUTF8(python3.PyList_GetItem(path, 0)))
    fmt.Println("module name:", name)
    return python3.PyImport_ImportModule(name) 
}

每次在 goroutine 中调用算法。

func CalcPre(dataFilePath string) (sbpI int, dbpI int) {
    python3.Py_Initialize()
    if !python3.Py_IsInitialized() {
        fmt.Println("Error initializing the python interpreter")
        os.Exit(1)
    }
    gstate = python3.PyGILState_Ensure()
    fmt.Println("Py_Initialize success")
    vbp := ImportModule("/Users/zhao/Desktop/lab/ppython", "value_bp")
    fmt.Println("ImportModule success")

    b := vbp.GetAttrString("estimate")
    fmt.Printf("[FUNC] b = %#v\n", b)
    bArgs := python3.PyTuple_New(1)
    python3.PyTuple_SetItem(bArgs, 0, python3.PyUnicode_FromString(dataFilePath))
    re := b.Call(bArgs, python3.Py_None)
    sbp := python3.PyTuple_GetItem(re, 0)
    dbp := python3.PyTuple_GetItem(re, 1)
    defer func() {
        python3.Py_Finalize()
        fmt.Println("python3.Py_Finalize()")
    }()
    sbpI = python3.PyLong_AsLong(sbp)
    dbpI = python3.PyLong_AsLong(dbp)
    python3.PyGILState_Release(gstate)
    return
}

func Calc(dataFilePath string) {
    CalcPre(dataFilePath)
}

像这样的示例调用者: go Calc("aaa.csv") 。
要重现这一点，请使用上面的代码和上面的环境，将这些代码放入一个 goroutine，如 go Calc("aaa.csv") 。为简单起见，您可以删除算法部分，只保留骨架。
为简单起见，您也可以使用此代码来重现此问题:

func CalcPre(dataFilePath string) (sbpI int, dbpI int) {
    python3.Py_Initialize()
    if !python3.Py_IsInitialized() {
        fmt.Println("Error initializing the python interpreter")
        os.Exit(1)
    }
    gstate = python3.PyGILState_Ensure()
    ImportModule("/Users/zhao/Desktop/lab/ppython", "value_bp")
    defer func() {
        python3.Py_Finalize()
        fmt.Println("python3.Py_Finalize()")
    }()
    
    return
}

Answer 1

这是实现相同目的的选项。
只需使用 os/exec包 ~~~

使用exec.Command设置可执行二进制文件及其参数，
示例代码:cmd := exec.Command("python3", "test.py", "./srcdata/aaa.csv")

使用cmd.Dir设置命令工作目录；

使用output := cmd.CombinedOutput()运行此命令并获取其标准输出信息；

随心所欲地处理output .

完整片段如下:

func Calc(dataFilePath string) (sI int, dI int) {
    cmd := exec.Command("python3", "test.py", "./srcdata/"+dataFilePath)
    cmd.Dir = "/Users/username/Desktop/lab/ppython"
    output, e := cmd.CombinedOutput()
    if e != nil {
        fmt.Println("Python Execution Error :", e)
    }
    result := string(output)
    strs := strings.Split(result, "\n")
    sI, e = strconv.Atoi(strs[0])
    dI, e = strconv.Atoi(strs[1])
    return
}

无论如何，这可以为我处理大多数跨语言操作。