python - 如何为Pyjulia 0.5.2添加处理器(addprocs)并将Julia函数的值返回给Python？

Question

有谁知道在 Python 中调用 Julia 函数时如何为 Pyjulia 添加处理器？我需要使用 Python 调用使用多个内核的 Julia 函数并将值返回给 Python。

Python 代码，testJulia_addproc.py:

import julia
j = julia.Julia()
print("Core number = {}".format(j.nprocs()))
j.include("test_addproc.jl")
a = j.eval("test_addproc()")
print a

Julia 代码test_addproc.jl:

function test_addproc()
    println("line 1")
    np = nprocs()
    println("line 2")
    if np < 12
        println("line 3")
        addprocs(12-np)
        println("line 4")
    end
    println("line 5")
    return(1+1)
end

==
$ python testJulia_addproc.py
我收到了这个错误。

WARNING: redefining constant JULIA_HOME
Core number = 1
line 1
line 2
line 3
ERROR (unhandled task failure): could not spawn setenv(`/anaconda/lib/python2.7/site-packages/julia-0.1.1-py2.7.egg/julia/../fake-julia/julia -Ccore2 -J/Applications/Julia-0.5.app/Contents/Resources/julia/lib/julia/sys.dylib --compile=yes --depwarn=yes --bind-to 127.0.0.1 --worker pjVTa5WBhFbDB7vs`; dir="/Users/chlin/Dropbox/Projects/StackOverflow"): no such file or directory (ENOENT)
 in _jl_spawn(::String, ::Array{String,1}, ::Ptr{Void}, ::Base.Process, ::Base.DevNullStream, ::Base.PipeEndpoint, ::Base.TTY) at ./process.jl:321
 in #424 at ./process.jl:478 [inlined]
 in setup_stdio(::Base.##424#425{Cmd,Ptr{Void},Base.Process}, ::Tuple{Base.DevNullStream,Pipe,Base.TTY}) at ./process.jl:466
 in #spawn#423(::Nullable{Base.ProcessChain}, ::Function, ::Cmd, ::Tuple{Base.DevNullStream,Pipe,Base.TTY}, ::Bool, ::Bool) at ./process.jl:477
 in (::Base.#kw##spawn)(::Array{Any,1}, ::Base.#spawn, ::Cmd, ::Tuple{Base.DevNullStream,Pipe,Base.TTY}, ::Bool, ::Bool) at ./<missing>:0
 in #spawn#420(::Nullable{Base.ProcessChain}, ::Function, ::Base.CmdRedirect, ::Tuple{Base.DevNullStream,Pipe,Base.TTY}, ::Bool, ::Bool) at ./process.jl:359
 in (::Base.#kw##spawn)(::Array{Any,1}, ::Base.#spawn, ::Base.CmdRedirect, ::Tuple{Base.DevNullStream,Pipe,Base.TTY}, ::Bool, ::Bool) at ./<missing>:0
 in open(::Base.CmdRedirect, ::String, ::Base.DevNullStream) at ./process.jl:539
 in launch(::Base.LocalManager, ::Dict{Any,Any}, ::Array{WorkerConfig,1}, ::Condition) at ./managers.jl:318
 in (::Base.##666#670{Base.LocalManager,Dict{Any,Any},Array{WorkerConfig,1},Condition})() at ./event.jl:68
could not spawn setenv(`/anaconda/lib/python2.7/site-packages/julia-0.1.1-py2.7.egg/julia/../fake-julia/julia -Ccore2 -J/Applications/Julia-0.5.app/Contents/Resources/julia/lib/julia/sys.dylib --compile=yes --depwarn=yes --bind-to 127.0.0.1 --worker pjVTa5WBhFbDB7vs`; dir="/Users/chlin/Dropbox/Projects/StackOverflow"): no such file or directory (ENOENT)Traceback (most recent call last):
  File "testJulia_addproc.py", line 5, in <module>
    j.eval("test_addproc()")
  File "//anaconda/lib/python2.7/site-packages/julia-0.1.1-py2.7.egg/julia/core.py", line 436, in eval
    ans = self._call(src)
  File "//anaconda/lib/python2.7/site-packages/julia-0.1.1-py2.7.egg/julia/core.py", line 399, in _call
    self.check_exception(src)
  File "//anaconda/lib/python2.7/site-packages/julia-0.1.1-py2.7.egg/julia/core.py", line 419, in check_exception
    .format(exception_type, src))
julia.core.JuliaError: Exception 'UVError' occurred while calling julia code:
test_addproc()

基于 this post ，我发现 pyjulia 中的处理器数量需要与 Julia 中的数量匹配(@Isaiah:纯 Julia 中的 nprocs() 必须匹配 j.nprocs())。

我尝试在 pyjulia 中添加 procs。

import julia
j = julia.Julia()
j.addprocs(2)

出现错误:

ERROR (unhandled task failure): could not spawn setenv(`/anaconda/lib/python2.7/site-packages/julia-0.1.1-py2.7.egg/julia/../fake-julia/julia -Ccore2 -J/Applications/Julia-0.5.app/Contents/Resources/julia/lib/julia/sys.dylib --compile=yes --depwarn=yes --bind-to 127.0.0.1 --worker 8WpA7O0EArLzXh25`; dir="/Applications/Julia-0.5.app/Contents/Resources/julia/bin"): no such file or directory (ENOENT)
 in _jl_spawn(::String, ::Array{String,1}, ::Ptr{Void}, ::Base.Process, ::Base.DevNullStream, ::Base.PipeEndpoint, ::Base.TTY) at ./process.jl:321
 in #424 at ./process.jl:478 [inlined]
 in setup_stdio(::Base.##424#425{Cmd,Ptr{Void},Base.Process}, ::Tuple{Base.DevNullStream,Pipe,Base.TTY}) at ./process.jl:466
 in #spawn#423(::Nullable{Base.ProcessChain}, ::Function, ::Cmd, ::Tuple{Base.DevNullStream,Pipe,Base.TTY}, ::Bool, ::Bool) at ./process.jl:477
 in (::Base.#kw##spawn)(::Array{Any,1}, ::Base.#spawn, ::Cmd, ::Tuple{Base.DevNullStream,Pipe,Base.TTY}, ::Bool, ::Bool) at ./<missing>:0
 in #spawn#420(::Nullable{Base.ProcessChain}, ::Function, ::Base.CmdRedirect, ::Tuple{Base.DevNullStream,Pipe,Base.TTY}, ::Bool, ::Bool) at ./process.jl:359
 in (::Base.#kw##spawn)(::Array{Any,1}, ::Base.#spawn, ::Base.CmdRedirect, ::Tuple{Base.DevNullStream,Pipe,Base.TTY}, ::Bool, ::Bool) at ./<missing>:0
 in open(::Base.CmdRedirect, ::String, ::Base.DevNullStream) at ./process.jl:539
 in launch(::Base.LocalManager, ::Dict{Any,Any}, ::Array{WorkerConfig,1}, ::Condition) at ./managers.jl:318
 in (::Base.##666#670{Base.LocalManager,Dict{Any,Any},Array{WorkerConfig,1},Condition})() at ./event.jl:68
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-3-bfa063d91dec> in <module>()
----> 1 j.addprocs(2)

RuntimeError: Julia exception: could not spawn setenv(`/anaconda/lib/python2.7/site-packages/julia-0.1.1-py2.7.egg/julia/../fake-julia/julia -Ccore2 -J/Applications/Julia-0.5.app/Contents/Resources/julia/lib/julia/sys.dylib --compile=yes --depwarn=yes --bind-to 127.0.0.1 --worker 8WpA7O0EArLzXh25`; dir="/Applications/Julia-0.5.app/Contents/Resources/julia/bin"): no such file or directory (ENOENT)

基于 this post ，我发现有一种方法可以在 pyjulia 中添加处理器，但没有返回 Julia 函数的值。

# One way to add procs in pyjulia
import ctypes
JLPATH=b"/Applications/Julia-0.5.app/Contents/Resources/julia"
jl = ctypes.PyDLL(JLPATH+b"/lib/libjulia.dylib", ctypes.RTLD_GLOBAL)
jl.jl_init(JLPATH+b"/bin/")
jl.jl_eval_string(" include(test_addproc.jl) ")
a = jl.jl_eval_string(" test_addproc()) ")
print a
# 0
# It should be 2

我的 MacPro 版本是 OS X Yosemite 10.10.5
我的 Python 版本是 Python 2.7.12::Anaconda 4.2.0 (x86_64)。
我的 Julia 版本是

Julia Version 0.5.2
Commit f4c6c9d4bb (2017-05-06 16:34 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin13.4.0)
  CPU: Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Sandybridge)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.7.1 (ORCJIT, ivybridge)

欢迎任何建议或想法。谢谢。

Answer 1

第一条错误信息说，

错误(未处理的任务失败):无法产生 setenv(`/anaconda/lib/python2.7/site-packages/julia-0.1.1-py2.7.egg/julia/../fake -julia/julia -Ccore2 -J/Applications/Julia-0.5.app/Contents/Resources/julia/lib/julia/sys.dylib --compile=yes --depwarn=yes --bind-to 127.0.0.1 -- worker pjVTa5WBhFbDB7vs`; dir="/Users/chlin/Dropbox/Projects/StackOverflow"): 没有这样的文件或目录 (ENOENT)


我发现python包文件夹下没有fake-julia目录。
所以我将 fake-julia 目录从 PyJulia 复制到 /anaconda/lib/python2.7/site-packages/julia-0.1.1-py2.7.egg/ 。
现在我可以通过 j.addprocs(1) 添加处理器并从 testJulia_addproc.py 中得到结果 2 而不会出错。

到目前为止，这是一种解决方法。
我还在 PyJulia GitHub 上发布问题，希望在未来的版本中修复它。