python - Linux 上的 MATLAB MEX 文件无法在 libpython.2.7.so 中找到符号

Question

我一直在研究 project使用 libpython2.7.so 将 Python 解释器作为 MEX 文件嵌入到 MATLAB 中，但是每当我尝试导入作为扩展模块实现的 Python 模块时都会遇到问题。例如，尝试导入 itertools 会导致以下错误:

>> py_import itertools
ImportError: /usr/lib64/python2.7/lib-dynload/itertoolsmodule.so: undefined symbol: PyTuple_Type
??? Error using ==> pymex_fns
Python exception inside py_import.

Error in ==> py_import at 24
    py_obj = pymex_fns(py_function_t.IMPORT, name);

无论我是否在调用 Py_Initialize() 之前清除 LD_LIBRARY_PATH 以及调用 ldd(对于 itertools.so，在本例中)从 MATLAB 内部不会导致任何 (not found) 消息。下面，我粘贴了在 LD_DEBUG=libs 设置下运行 MATLAB 的结果，首先是在 MATLAB 启动时设置了 LD_LIBRARY_PATH，然后在运行 setenv('LD_LIBRARY_PATH ', '') 在尝试导入之前。

如何解决这个问题，并允许动态加载扩展模块？

使用 MATLAB 默认 LD_LIBRARY_PATH:

>> py_import itertools           
      3018: find library=libpython2.7.so.1.0 [0]; searching
      3018:  search path=/usr/local/MATLAB/R2011a/bin/glnxa64/../../bin/glnxa64:/usr/local/MATLAB/R2011a/bin/glnxa64:/usr/local/MATLAB/R2011a/bin/glnxa64/../../sys/os/glnxa64      (RPATH from file /usr/local/MATLAB/R2011a/bin/glnxa64/MATLAB)
      3018:   trying file=/usr/local/MATLAB/R2011a/bin/glnxa64/../../bin/glnxa64/libpython2.7.so.1.0
      3018:   trying file=/usr/local/MATLAB/R2011a/bin/glnxa64/libpython2.7.so.1.0
      3018:   trying file=/usr/local/MATLAB/R2011a/bin/glnxa64/../../sys/os/glnxa64/libpython2.7.so.1.0
      3018:  search path=/usr/local/MATLAB/R2011a/sys/os/glnxa64        (LD_LIBRARY_PATH)
      3018:   trying file=/usr/local/MATLAB/R2011a/sys/os/glnxa64/libpython2.7.so.1.0
      3018:  search path=/usr/local/MATLAB/R2011a/bin/glnxa64       (RPATH from file /usr/local/MATLAB/R2011a/bin/glnxa64/MATLAB)
      3018:   trying file=/usr/local/MATLAB/R2011a/bin/glnxa64/libpython2.7.so.1.0
      3018:  search path=/usr/local/MATLAB/R2011a/extern/lib/glnxa64:/usr/local/MATLAB/R2011a/runtime/glnxa64:/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64/native_threads:/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64/server:/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64     (LD_LIBRARY_PATH)
      3018:   trying file=/usr/local/MATLAB/R2011a/extern/lib/glnxa64/libpython2.7.so.1.0
      3018:   trying file=/usr/local/MATLAB/R2011a/runtime/glnxa64/libpython2.7.so.1.0
      3018:   trying file=/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64/native_threads/libpython2.7.so.1.0
      3018:   trying file=/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64/server/libpython2.7.so.1.0
      3018:   trying file=/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64/libpython2.7.so.1.0
      3018:  search cache=/etc/ld.so.cache
      3018:   trying file=/lib64/libpython2.7.so.1.0
      3018: 
      3018: find library=libutil.so.1 [0]; searching
      3018:  search path=/usr/local/MATLAB/R2011a/bin/glnxa64/../../bin/glnxa64:/usr/local/MATLAB/R2011a/bin/glnxa64:/usr/local/MATLAB/R2011a/bin/glnxa64/../../sys/os/glnxa64      (RPATH from file /usr/local/MATLAB/R2011a/bin/glnxa64/MATLAB)
      3018:   trying file=/usr/local/MATLAB/R2011a/bin/glnxa64/../../bin/glnxa64/libutil.so.1
      3018:   trying file=/usr/local/MATLAB/R2011a/bin/glnxa64/libutil.so.1
      3018:   trying file=/usr/local/MATLAB/R2011a/bin/glnxa64/../../sys/os/glnxa64/libutil.so.1
      3018:  search path=/usr/local/MATLAB/R2011a/sys/os/glnxa64        (LD_LIBRARY_PATH)
      3018:   trying file=/usr/local/MATLAB/R2011a/sys/os/glnxa64/libutil.so.1
      3018:  search path=/usr/local/MATLAB/R2011a/bin/glnxa64       (RPATH from file /usr/local/MATLAB/R2011a/bin/glnxa64/MATLAB)
      3018:   trying file=/usr/local/MATLAB/R2011a/bin/glnxa64/libutil.so.1
      3018:  search path=/usr/local/MATLAB/R2011a/extern/lib/glnxa64:/usr/local/MATLAB/R2011a/runtime/glnxa64:/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64/native_threads:/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64/server:/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64     (LD_LIBRARY_PATH)
      3018:   trying file=/usr/local/MATLAB/R2011a/extern/lib/glnxa64/libutil.so.1
      3018:   trying file=/usr/local/MATLAB/R2011a/runtime/glnxa64/libutil.so.1
      3018:   trying file=/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64/native_threads/libutil.so.1
      3018:   trying file=/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64/server/libutil.so.1
      3018:   trying file=/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64/libutil.so.1
      3018:  search cache=/etc/ld.so.cache
      3018:   trying file=/lib64/libutil.so.1
      3018: 
      3018: 
      3018: calling init: /lib64/libutil.so.1
      3018: 
      3018: 
      3018: calling init: /lib64/libpython2.7.so.1.0
      3018: 
      3018: 
      3018: calling init: /home/cgranade/academics/software-projects/pymex-embed/src/pymex_fns.mexa64
      3018: 
      3018: /home/cgranade/academics/software-projects/pymex-embed/src/pymex_fns.mexa64: error: symbol lookup error: undefined symbol: mexLibrary (fatal)
      3018: /usr/lib64/python2.7/lib-dynload/itertoolsmodule.so: error: symbol lookup error: undefined symbol: PyTuple_Type (fatal)
??? Error using ==> pymex_fns
Python exception inside py_import.

Error in ==> py_import at 24
    py_obj = pymex_fns(py_function_t.IMPORT, name);

清除LD_LIBRARY_PATH:

>> py_import itertools           
      3125: find library=libpython2.7.so.1.0 [0]; searching
      3125:  search path=/usr/local/MATLAB/R2011a/bin/glnxa64/../../bin/glnxa64:/usr/local/MATLAB/R2011a/bin/glnxa64:/usr/local/MATLAB/R2011a/bin/glnxa64/../../sys/os/glnxa64      (RPATH from file /usr/local/MATLAB/R2011a/bin/glnxa64/MATLAB)
      3125:   trying file=/usr/local/MATLAB/R2011a/bin/glnxa64/../../bin/glnxa64/libpython2.7.so.1.0
      3125:   trying file=/usr/local/MATLAB/R2011a/bin/glnxa64/libpython2.7.so.1.0
      3125:   trying file=/usr/local/MATLAB/R2011a/bin/glnxa64/../../sys/os/glnxa64/libpython2.7.so.1.0
      3125:  search path=/usr/local/MATLAB/R2011a/sys/os/glnxa64        (LD_LIBRARY_PATH)
      3125:   trying file=/usr/local/MATLAB/R2011a/sys/os/glnxa64/libpython2.7.so.1.0
      3125:  search path=/usr/local/MATLAB/R2011a/bin/glnxa64       (RPATH from file /usr/local/MATLAB/R2011a/bin/glnxa64/MATLAB)
      3125:   trying file=/usr/local/MATLAB/R2011a/bin/glnxa64/libpython2.7.so.1.0
      3125:  search path=/usr/local/MATLAB/R2011a/extern/lib/glnxa64:/usr/local/MATLAB/R2011a/runtime/glnxa64:/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64/native_threads:/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64/server:/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64     (LD_LIBRARY_PATH)
      3125:   trying file=/usr/local/MATLAB/R2011a/extern/lib/glnxa64/libpython2.7.so.1.0
      3125:   trying file=/usr/local/MATLAB/R2011a/runtime/glnxa64/libpython2.7.so.1.0
      3125:   trying file=/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64/native_threads/libpython2.7.so.1.0
      3125:   trying file=/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64/server/libpython2.7.so.1.0
      3125:   trying file=/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64/libpython2.7.so.1.0
      3125:  search cache=/etc/ld.so.cache
      3125:   trying file=/lib64/libpython2.7.so.1.0
      3125: 
      3125: find library=libutil.so.1 [0]; searching
      3125:  search path=/usr/local/MATLAB/R2011a/bin/glnxa64/../../bin/glnxa64:/usr/local/MATLAB/R2011a/bin/glnxa64:/usr/local/MATLAB/R2011a/bin/glnxa64/../../sys/os/glnxa64      (RPATH from file /usr/local/MATLAB/R2011a/bin/glnxa64/MATLAB)
      3125:   trying file=/usr/local/MATLAB/R2011a/bin/glnxa64/../../bin/glnxa64/libutil.so.1
      3125:   trying file=/usr/local/MATLAB/R2011a/bin/glnxa64/libutil.so.1
      3125:   trying file=/usr/local/MATLAB/R2011a/bin/glnxa64/../../sys/os/glnxa64/libutil.so.1
      3125:  search path=/usr/local/MATLAB/R2011a/sys/os/glnxa64        (LD_LIBRARY_PATH)
      3125:   trying file=/usr/local/MATLAB/R2011a/sys/os/glnxa64/libutil.so.1
      3125:  search path=/usr/local/MATLAB/R2011a/bin/glnxa64       (RPATH from file /usr/local/MATLAB/R2011a/bin/glnxa64/MATLAB)
      3125:   trying file=/usr/local/MATLAB/R2011a/bin/glnxa64/libutil.so.1
      3125:  search path=/usr/local/MATLAB/R2011a/extern/lib/glnxa64:/usr/local/MATLAB/R2011a/runtime/glnxa64:/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64/native_threads:/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64/server:/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64     (LD_LIBRARY_PATH)
      3125:   trying file=/usr/local/MATLAB/R2011a/extern/lib/glnxa64/libutil.so.1
      3125:   trying file=/usr/local/MATLAB/R2011a/runtime/glnxa64/libutil.so.1
      3125:   trying file=/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64/native_threads/libutil.so.1
      3125:   trying file=/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64/server/libutil.so.1
      3125:   trying file=/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64/libutil.so.1
      3125:  search cache=/etc/ld.so.cache
      3125:   trying file=/lib64/libutil.so.1
      3125: 
      3125: 
      3125: calling init: /lib64/libutil.so.1
      3125: 
      3125: 
      3125: calling init: /lib64/libpython2.7.so.1.0
      3125: 
      3125: 
      3125: calling init: /home/cgranade/academics/software-projects/pymex-embed/src/pymex_fns.mexa64
      3125: 
      3125: /home/cgranade/academics/software-projects/pymex-embed/src/pymex_fns.mexa64: error: symbol lookup error: undefined symbol: mexLibrary (fatal)
      3125: /usr/lib64/python2.7/lib-dynload/itertoolsmodule.so: error: symbol lookup error: undefined symbol: PyTuple_Type (fatal)

Answer 1

尝试从 MATLAB 内部和系统 shell 运行常规 python:

外壳

$ LD_DEBUG_OUTPUT=./ld_debug_out LD_DEBUG=all python -c 'import numpy'

MATLAB

>> !LD_DEBUG_OUTPUT=./ld_debug_out LD_DEBUG=all python -c 'import numpy'

并检查输出文件(也许 all 信息太多了!)。请注意，我导入了 numpy，因为 itertools 在我的安装中没有相应的共享库..

---

一个想法是在调用 Py_Initialize() 之前尝试在 MEX 函数中手动加载共享库:

dlopen('libpython2.7.so', RTLD_GLOBAL|RTLD_LAZY);

---

我在 Windows 机器上，但上次玩这个时我也遇到了一些问题。我记得一旦导入，某些 C 扩展模块就无法在同一 session 中卸载和重新加载(在嵌入 Python 的托管进程的生命周期内)。查看这些 remarks及相关PEP .