gpt4 book ai didi

python - 为什么我收到多处理的递归错误?

转载 作者:行者123 更新时间:2023-12-04 01:23:04 25 4
gpt4 key购买 nike

我希望使用多处理对大量地址进行地理编码。我有以下代码:

import multiprocessing
import geocoder

addresses = ['New York City, NY','Austin, TX', 'Los Angeles, CA', 'Boston, MA'] # and on and on

def geocode_worker(address):
return geocoder.arcgis(address)

def main_process():
pool = multiprocessing.Pool(processes=multiprocessing.cpu_count())
return pool.map(geocode_worker, addresses)

if __name__ == '__main__':
main_process()

但它给了我这个错误:

Traceback (most recent call last):
File "/opt/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/opt/anaconda3/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/opt/anaconda3/lib/python3.7/multiprocessing/pool.py", line 470, in _handle_results
task = get()
File "/opt/anaconda3/lib/python3.7/multiprocessing/connection.py", line 251, in recv
return _ForkingPickler.loads(buf.getbuffer())
File "/opt/anaconda3/lib/python3.7/site-packages/geocoder/base.py", line 599, in __getattr__
if not self.ok:
File "/opt/anaconda3/lib/python3.7/site-packages/geocoder/base.py", line 536, in ok
return len(self) > 0
File "/opt/anaconda3/lib/python3.7/site-packages/geocoder/base.py", line 422, in __len__
return len(self._list)

错误的最后3行一遍又一遍的重复,然后traceback的最后一行是:

RecursionError: maximum recursion depth exceeded while calling a Python object

谁能帮我弄清楚为什么?

最佳答案

问题是 geocoder 返回的 ArcgisQuery 对象不可 picklable - 或者更确切地说,它不可 unpicklable。 unpickle 进程由于使用 __getattr__ 而进入无限循环,它在内部尝试访问 self.ok,最终依赖于 self._list 待定义,unpickling 时未定义,因为它仅在 __init__ 中定义,并且 __init__ is not called while unpickling .由于它未定义,它会尝试使用 __getattr__ 找到它,这会尝试再次访问 self.ok,并创建无限循环。

您可以通过不在工作进程和主进程之间传递 ArcgisQuery 对象本身来解决此问题,而是仅传递其底层 __dict__。然后,在主进程中重建 ArcgisQuery 对象:

import multiprocessing
import geocoder
from geocoder.arcgis import ArcgisQuery

addresses = ['New York City, NY','Austin, TX', 'Los Angeles, CA', 'Boston, MA'] # and on and on

def geocode_worker(address):
out = geocoder.arcgis(address)
return out.__dict__ # Only return the object's __dict__

def main_process():
pool = multiprocessing.Pool(processes=multiprocessing.cpu_count())
l = pool.map(geocode_worker, addresses)
out = []
for d in l:
q = ArcgisQuery(d['location']) # location is a required constructor arg
q.__dict__.update(d) # Load the rest of our state into the new object
out.append(q)
return out

if __name__ == '__main__':
print(main_process())

如果您实际上不需要整个 ArcgisQuery 对象,而只需要它的某些部分,您也可以只从工作进程中返回这些对象,以避免需要这种 hack。

就其值(value)而言,看起来 geocoder 可以通过在 ArcgisQuery 或其基类上实现 __getstate____setstate__ 来解决其酸洗问题,像这样:

    def __getstate__(self):
return self.__dict__

def __setstate__(self, state):
self.__dict__.update(state)

关于python - 为什么我收到多处理的递归错误?,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/62331047/

25 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com