python多处理struct.error

Question

我正在遍历一组大文件，并使用多处理进行操作/写入。我从我的数据框中创建了一个可迭代对象，并将其传递给多处理的 map 函数。对于较小的文件，处理没问题，但是当我遇到较大的文件 (~10g) 时，出现错误:

python struct.error: 'i' format requires -2147483648 <= number <= 2147483647

代码:

    data = np.array_split(data, 10)        
    with mp.Pool(processes=5, maxtasksperchild=1) as pool1:
                    pool1.map(write_in_parallel, data)
                    pool1.close()
                    pool1.join()

基于 this answer我认为问题是我传递给 map 的文件太大了。因此，我首先尝试将数据帧拆分为 1.5g block ，然后将每个 block 独立传递给映射，但我仍然收到相同的错误。

完整回溯:

Traceback (most recent call last):
  File "_FNMA_LLP_dataprep_final.py", line 51, in <module>
    write_files()
  File "_FNMA_LLP_dataprep_final.py", line 29, in write_files
    '.txt')
  File "/DATAPREP/appl/FNMA_LLP/code/FNMA_LLP_functions.py", line 116, in write_dynamic_columns_fannie
    pool1.map(write_in_parallel, first)
  File "/opt/Python364/lib/python3.6/multiprocessing/pool.py", line 266, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/Python364/lib/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
  File "/opt/Python364/lib/python3.6/multiprocessing/pool.py", line 424, in _handle_tasks
    put(task)
  File "/opt/Python364/lib/python3.6/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/opt/Python364/lib/python3.6/multiprocessing/connection.py", line 393, in _send_bytes
    header = struct.pack("!i", n)
struct.error: 'i' format requires -2147483648 <= number <= 2147483647

Answer 1

在answer你提到的也是另一个要点:数据应该由子函数加载。在您的例子中，它是 write_in_parallel 函数。我建议您通过以下方式更改子函数:

def write_in_parallel('/path/to/your/data'):
    """ We'll make an assumption that your data is stored in csv file""" 

    data = pd.read_csv('/path/to/your/data')
    ...

那么你的“池代码”应该是这样的:

with mp.Pool(processes=(mp.cpu_count() - 1)) as pool:
    chunks = pool.map(write_in_parallel, ('/path/to/your/data',))
df = pd.concat(chunks)

希望对你有所帮助