python - 从一个来源计算多个 dask.dataframe.from_delayed()

Question

如何从一个 delayed 序列中并行计算 .from_delayed()？

def foo():
    df1, df2 = ...  # prepare two pd.DataFrame() in one foo() call
    return df1, df2

dds = [dask.delayed(foo)() for _ in range(5)]  # 5 delayed pairs (df1, df2)...
df1 = dd.from_delayed([d[0] for d in dds], meta=...)
df2 = dd.from_delayed([d[1] for d in dds], meta=...)
client.compute([
    df1.to_parquet(file1, write_index=True, engine='fastparquet', compute=False),
    df2.to_parquet(file2, write_index=True, engine='fastparquet', compute=False)
], sync=True)

这里 foo() 将被调用 10 次。是否可以创建图表，使其仅被调用 5 次？

谢谢

Answer 1

感谢您提供清晰的示例。原则上你是正确的 foo 应该只被调用五次。我的猜测是优化在这里行为不端。短期内，我建议尝试最近发布的以下内容:

dask.config.set({"optimization.fuse.active": False})

... your code follows