gpt4 book ai didi

python - 从数据框中转换的 apache 箭头文件在使用 arrow.js 读取时给出 null

转载 作者:行者123 更新时间:2023-12-04 21:01:48 24 4
gpt4 key购买 nike

我将一个示例数据帧转换为 .arrow文件使用 pyarrow

import numpy as np
import pandas as pd
import pyarrow as pa

df = pd.DataFrame({"a": [10, 2, 3]})
df['a'] = pd.to_numeric(df['a'],errors='coerce')
table = pa.Table.from_pandas(df)
writer = pa.RecordBatchFileWriter('test.arrow', table.schema)
writer.write_table(table)
writer.close()

这将创建一个文件 test.arrow
df.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3 entries, 0 to 2
Data columns (total 1 columns):
a 3 non-null int64
dtypes: int64(1)
memory usage: 104.0 bytes

然后在 NodeJS 中我用 arrowJS 加载文件。
https://arrow.apache.org/docs/js/
const fs = require('fs');
const arrow = require('apache-arrow');

const data = fs.readFileSync('test.arrow');
const table = arrow.Table.from(data);

console.log(table.schema.fields.map(f => f.name));
console.log(table.count());
console.log(table.get(0));

这打印像
[ 'a' ]
0
null

我期待这张表的长度为 3 和 table.get(0)给出第一行而不是 null .

这是表的样子 console.log(table._schema)
[ Int_ [Int] { isSigned: true, bitWidth: 16 } ]
Schema {
fields:
[ Field { name: 'a', type: [Int_], nullable: true, metadata: Map {} } ],
metadata:
Map {
'pandas' => '{"index_columns": [{"kind": "range", "name": null, "start": 0, "stop": 5, "step": 1}], "column_indexes": [{"name": null, "field_name": null, "pandas_type": "unicode", "numpy_type": "object", "metadata": {"encoding": "UTF-8"}}], "columns": [{"name": "a", "field_name": "a", "pandas_type": "int16", "numpy_type": "int16", "metadata": null}], "creator": {"library": "pyarrow", "version": "0.15.0"}, "pandas_version": "0.22.0"}' },
dictionaries: Map {} }

知道为什么它没有按预期获取数据吗?

最佳答案

这是由于 Arrow 0.15 中的格式更改,如 mentioned by Wes在 Apache JIRA 上。这意味着在将 IPC 文件发送到旧版本的 Arrow 时,所有 Arrow 库,而不仅仅是 PyArrow,都会出现这个问题。修复是将 ArrowJS 升级到 0.15.0,以便您可以在其他 Arrow 库和 JS 库之间往返。如果由于某种原因无法更新,则可以改用以下解决方法之一:

通行证use_legacy_format=True作为 kwarg 到 RecordBatchFileWriter :

with pa.RecordBatchFileWriter('file.arrow', table.schema, use_legacy_format=True) as writer:
writer.write_table(table)

设置环境变量 ARROW_PRE_0_15_IPC_FORMAT到 1:

$ export ARROW_PRE_0_15_IPC_FORMAT = 1
$ python
>>> import pyarrow as pa
>>> table = pa.Table.from_pydict( {"a": [1, 2, 3], "b": [4, 5, 6]} )
>>> with pa.RecordBatchFileWriter('file.arrow', table.schema) as writer:
... writer.write_table(table)
...

或者将 PyArrow 降级为 0.14.x :

$ conda install -c conda-forge pyarrow=0.14.1

关于python - 从数据框中转换的 apache 箭头文件在使用 arrow.js 读取时给出 null,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/58313254/

24 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com