python - 使用 Pyarrow 将 .parquet 文件转换为 CSV

Question

我有一个 .parquet 文件，我正在使用 PyArrow。我使用以下代码将 .parquet 文件转换为表格:

import pyarrow.parquet as pq
import pandas as pd
filepath = "xxx"  # This contains the exact location of the file on the server
from pandas import Series, DataFrame
table = pq.read_table(filepath)

执行 table.shape 返回 (39014 行，19 列)。

表的架构是:

col1: int64 not null
col2: string not null
col3: string not null
col4: int64 not null
col5: string not null
col6: string not null
col7: int64 not null
col8: int64 not null
col9: string not null
col10: string not null
col11: string not null
col12: string not null
col13: string not null
col14: string not null
col15: string not null
col16: int64 not null
col17: int64 not null
col18: int64 not null
col19: string not null

执行 p = table.to_pandas() 时出现以下错误:

ImportError: cannot import name RangeIndex

如何将此 Parquet 文件转换为数据框，然后再转换为 CSV？请帮忙。谢谢。

Answer 1

尝试以下操作:

import pyarrow as pa
import pyarrow.parquet as pq
import pandas as pd
import pyodbc

def read_pyarrow(path, nthreads=1):
    return pq.read_table(path, nthreads=nthreads).to_pandas()

path = './test.parquet'
df1 = read_pyarrow(path)

df1.to_csv(
    './test.csv',
    sep='|',
    index=False,
    mode='w',
    line_terminator='\n',
    encoding='utf-8')