python 多处理悲情 - AuthenticationError : digest sent was rejected - Windows

Question

请帮忙...

我有一个简单的脚本，可以在 Linux (Ubuntu) 上运行，没有任何问题，但在 Windows 上给出 AuthenticationError: 摘要发送被拒绝，我知道它与 multiprocessing.Manager 有关，但还不知道如何修复它:

这是代码:

(test)root@sharknado:/tmp# cat test3.py
import datetime

import multiprocessing
from multiprocessing import Manager
from pathos.multiprocessing import ProcessingPool
import time

class Test(object):
    def __init__(self):
        manager = Manager()
        self.logs = manager.list()

    def mysql_test(self, device):
        self.logs.append('mysql-device connected: {0} at {1}'.format(device, time.asctime()))

    def soap_test(self, device):
        self.logs.append('soap_test-device connected: {0} at {1}'.format(device, time.asctime()))

    def all_tests(self, device):
        self.mysql_test(device)
        self.soap_test(device)

    def test_multiprocessing(self):
        devices = range(5)

        time_start = time.time()
        pool = ProcessingPool(multiprocessing.cpu_count())
        pool.map(self.all_tests, devices)
        time_stop = time.time()

        print 'prgm-time: {0}'.format(str(datetime.timedelta(seconds=time_stop-time_start)))
        print '....................self.logs................'
        for i in self.logs:
            print i



def main():

    test = Test()
    test.test_multiprocessing()

if __name__ == '__main__':
    main()

在 Linux 上，

(test)root@sharknado:/tmp# pip freeze
PyMySQL==0.6.2
dill==0.2.1
pathos==0.2a1.dev
pox==0.2
pp==1.6.4.2
processing==0.52-pathos
pyodbc==3.0.7
pyre==0.8.2.0-pathos
six==1.8.0
suds==0.4
wsgiref==0.1.2
(test)root@sharknado:/tmp# uname -a
Linux sharknado 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:12 UTC 2014 i686 i686 i686 GNU/Linux
(test)root@sharknado:/tmp# python test3.py
prgm-time: 0:00:00.019835
....................self.logs................
mysql-device connected: 0 at Mon Oct 27 22:35:58 2014
soap_test-device connected: 0 at Mon Oct 27 22:35:58 2014
mysql-device connected: 1 at Mon Oct 27 22:35:58 2014
soap_test-device connected: 1 at Mon Oct 27 22:35:58 2014
mysql-device connected: 2 at Mon Oct 27 22:35:58 2014
soap_test-device connected: 2 at Mon Oct 27 22:35:58 2014
mysql-device connected: 3 at Mon Oct 27 22:35:58 2014
soap_test-device connected: 3 at Mon Oct 27 22:35:58 2014
mysql-device connected: 4 at Mon Oct 27 22:35:58 2014
soap_test-device connected: 4 at Mon Oct 27 22:35:58 2014
(test)root@sharknado:/tmp# python -version
(test)root@sharknado:/tmp# python --version
Python 2.7.8 :: Continuum Analytics, Inc.
(test)root@sharknado:/tmp#

Windows...

[test] C:\Temp>python --version
Python 2.7.8 :: Continuum Analytics, Inc.

[test] C:\Temp>pip freeze
PyMySQL==0.6.2
argparse==1.2.1
ciscoconfparse==1.1.4
dill==0.2.1
netaddr==0.7.12
pathos==0.2a1.dev
pox==0.2
pp==1.6.4.2
processing==0.52-pathos
pyodbc==3.0.7
pyre==0.8.2.0-pathos
python-dateutil==2.2
pywin32==219
six==1.8.0
suds==0.4
wsgiref==0.1.2

[test] C:\Temp>systeminfo | findstr /B /C:"OS Name" /C:"OS Version"
OS Name:                   Microsoft Windows 7 Enterprise
OS Version:                6.1.7601 Service Pack 1 Build 7601

[test] C:\Temp> echo windows 32-bit
windows 32-bit
[test] C:\Temp>
[test] C:\Temp>python C:\Temp\tests\simple_test_pathos_multiprocessing.py
Process PoolWorker-4:
Traceback (most recent call last):
  File "C:\Users\foo\AppData\Local\Continuum\Miniconda\envs\test\lib\site-packages\processing\process.py", line 227, in _bootstrap
    self.run()
  File "C:\Users\foo\AppData\Local\Continuum\Miniconda\envs\test\lib\site-packages\processing\process.py", line 85, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\foo\AppData\Local\Continuum\Miniconda\envs\test\lib\site-packages\processing\pool.py", line 54, in worker
    for job, i, func, args, kwds in iter(inqueue.get, None):
  File "C:\Users\foo\AppData\Local\Continuum\Miniconda\envs\test\lib\site-packages\processing\queue.py", line 327, in get
    return recv()
  File "C:\Users\foo\AppData\Local\Continuum\Miniconda\envs\test\lib\site-packages\dill\dill.py", line 160, in loads
    return load(file)
  File "C:\Users\foo\AppData\Local\Continuum\Miniconda\envs\test\lib\site-packages\dill\dill.py", line 150, in load
    obj = pik.load()
  File "C:\Users\foo\AppData\Local\Continuum\Miniconda\envs\test\lib\pickle.py", line 858, in load
    dispatch[key](self)
  File "C:\Users\foo\AppData\Local\Continuum\Miniconda\envs\test\lib\pickle.py", line 1133, in load_reduce
Process PoolWorker-1:
 T   value = func(*args)
  File "C:\Users\foo\AppData\Local\Continuum\Miniconda\envs\test\lib\multiprocessing\managers.py", line 879, in RebuildProxy
raceback (most recent call last):
  File "C:\Users\foo\AppData\Local\Continuum\Miniconda\envs\test\lib\site-packages\processing\process.py", line 227, in _bootstrap
    self.run()
     return func(token, serializer, incref=incref, **kwds)
 File "C:\Users\foo\AppData\Local\Continuum\Miniconda\envs\test\lib\site-packages\processing\process.py", line 85, in run
  File "C:\Users\foo\AppData\Local\Continuum\Miniconda\envs\test\lib\multiprocessing\managers.py", line 733, in __init__
    self._target(*self._args, **self._kwargs)
     self._incref()
 File "C:\Users\foo\AppData\Local\Continuum\Miniconda\envs\test\lib\site-packages\processing\pool.py", line 54, in worker
  File "C:\Users\foo\AppData\Local\Continuum\Miniconda\envs\test\lib\multiprocessing\managers.py", line 783, in _incref
    for job, i, func, args, kwds in iter(inqueue.get, None):
     conn = self._Client(self._token.address, authkey=self._authkey)
 File "C:\Users\foo\AppData\Local\Continuum\Miniconda\envs\test\lib\site-packages\processing\queue.py", line 327, in get
  File "C:\Users\foo\AppData\Local\Continuum\Miniconda\envs\test\lib\multiprocessing\connection.py", line 175, in Client
     return recv()
  File "C:\Users\foo\AppData\Local\Continuum\Miniconda\envs\test\lib\site-packages\dill\dill.py", line 160, in loads
    answer_challenge(c, authkey)
   return load(file)
   File "C:\Users\foo\AppData\Local\Continuum\Miniconda\envs\test\lib\multiprocessing\connection.py", line 435, in answer_challenge
 File "C:\Users\foo\AppData\Local\Continuum\Miniconda\envs\test\lib\site-packages\dill\dill.py", line 150, in load
    raise AuthenticationError('digest sent was rejected')
    obj = pik.load()
A uthenticationError: digest sent was rejected

Answer 1

最新版本 (git master) 应该可以解决此问题，因为我已将 pathos.multiprocessing 分支更新为 multiprocessing 的最新版本。

但是，在 Windows 上，我目前必须使用 freeze_support，而在非 Windows 系统上，您甚至不必在 __main__ 中运行。