个人中心
文章发布
退出
作者热门文章
- android - RelativeLayout 背景可绘制重叠内容
- android - 如何链接 cpufeatures lib 以获取 native android 库？
- java - OnItemClickListener 不起作用，但 OnLongItemClickListener 在自定义 ListView 中起作用
- java - Android 文件转字符串
28
4
我正在尝试使用 Python 的 parquet 模块来读取从本地 MapR 实例编写的一些 Parquet 文件。
我用来输出这些parquet文件的命令是:
df.sqlContext.sql("SQL HERE").write.format("parquet").option("mergeSchema", "true").save("/path/to/parquet/test.parquet")
这是文件在我的 Linux 主机上的样子:
drwxr-xr-x 2 mapr mapr 403 Oct 5 13:56 igayfvpwrs.parquet
不幸的是，当我在这里使用 Python (https://pypi.python.org/pypi/parquet) - 我收到以下异常:
IOError: [Errno 21] Is a directory: '/mnt/mapr/saw/user/mapr/igayfvpwrs.parquet'
有什么想法吗？这些文件在 MapR 中运行良好。
编辑 2:
我想通了。由于原始的 .parquet"file"是一个目录，只需使用 glob 遍历所有内部 .parquet 文件的目录 - Python-parquet 的原始代码在那里工作。
for filename in glob.glob("/mnt/mapr/saw/user/mapr/{0}.parquet/*.parquet".format(tempTableID)):
with open(filename) as foo:
for row in parquet.DictReader(foo, columns=["column"]):
print(json.dumps(row))
编辑:这是 Parquet 文件中的内容:
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 _common_metadata
-rwxr-xr-x 1 mapr mapr 2.4K Oct 5 13:58 _metadata
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00000-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00001-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00002-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00003-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00004-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00005-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00006-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00007-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00008-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00009-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00010-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00011-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00012-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00013-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00014-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00015-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00016-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00017-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00018-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00019-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00020-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00021-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00022-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00023-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00024-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00025-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00026-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00027-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00028-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00029-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00030-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00031-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00032-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00033-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00034-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00035-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00036-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00037-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00038-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00039-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00040-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00041-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00042-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00043-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00044-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00045-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00046-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00047-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00048-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00049-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00050-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00051-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00052-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 1.2K Oct 5 13:58 part-r-00053-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00054-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00055-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00056-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00057-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00058-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00059-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00060-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00061-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00062-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00063-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00064-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00065-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00066-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00067-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00068-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00069-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00070-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00071-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00072-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00073-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00074-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00075-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00076-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00077-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00078-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00079-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00080-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00081-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00082-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00083-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00084-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00085-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00086-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00087-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00088-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00089-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00090-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00091-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00092-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00093-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00094-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00095-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00096-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00097-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00098-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00099-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00100-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00101-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00102-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00103-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00104-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 1.2K Oct 5 13:58 part-r-00105-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00106-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00107-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00108-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00109-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00110-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00111-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00112-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00113-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00114-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00115-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00116-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00117-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00118-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00119-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00120-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00121-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00122-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00123-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00124-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 1.2K Oct 5 13:58 part-r-00125-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00126-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00127-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00128-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00129-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00130-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00131-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00132-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00133-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00134-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00135-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00136-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00137-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00138-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00139-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00140-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00141-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00142-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00143-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00144-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00145-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00146-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00147-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00148-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00149-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00150-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00151-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00152-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00153-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00154-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00155-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00156-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00157-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00158-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00159-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00160-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00161-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00162-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00163-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00164-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00165-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00166-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00167-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00168-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00169-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00170-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00171-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00172-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00173-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00174-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00175-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00176-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00177-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00178-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00179-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00180-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00181-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00182-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00183-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00184-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00185-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00186-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00187-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00188-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00189-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00190-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00191-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00192-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00193-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00194-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00195-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00196-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00197-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00198-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00199-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00200-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00201-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00202-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00203-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00204-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00205-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00206-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00207-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00208-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00209-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00210-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00211-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00212-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00213-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00214-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00215-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00216-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00217-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00218-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00219-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00220-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00221-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00222-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00223-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00224-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00225-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00226-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00227-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00228-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00229-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00230-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00231-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00232-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00233-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00234-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00235-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00236-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00237-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00238-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00239-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00240-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00241-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00242-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00243-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00244-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00245-02f0b80b-2993-46f8-
[truncated]
-rwxr-xr-x 1 mapr mapr 0 Oct 5 13:58 _SUCCESS
最佳答案
spark 上的 parquet 没有问题。 DataFrameWriter 将 parquet 格式写入目录，并根据正在写入的 DataFrame 的分区数对输出进行分区。
你得到的是绝对正常的。
关于python - 来自 Spark 的 Parquet 文件被检测为 Linux 中的目录，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/39881051/
28
4
0
是否可以对 parquet 格式执行分布式并发写入？ 是否可以在写入拼花文件时读取它们？ 如果有并发读/写的方法，我有兴趣了解。 提前感谢您的帮助。 最佳答案 我最终得到了 Parquet 开发人员的
如何从命令行检查 Parquet 文件的内容？ 我现在看到的唯一选择是 $ hadoop fs -get my-path local-file $ parquet-tools head local-f
我正在使用基于 Java(1.8) 的应用程序使用库创建 Parquet 文件 org.apache.avro.Schema 和 org.apache.parquet.hadoop.ParquetWr
我已经使用 pyspark 创建了多个 parquet 文件，现在我正在尝试将所有 parquet 文件合并为 1 个。我能够合并这些文件，但是在读取生成的文件时，我遇到了错误。以前有人遇到过这个问题
我创建了一个数据框，如下所示: expanded_1 = pd.DataFrame({"Point": [random.choice(points) for x in range(30000000)]
当我在 R 和 Python 中保存 Parquet 文件(使用 pyarrow)时，我得到一个保存在元数据中的箭头模式字符串。 如何读取元数据？它是 Flatbuffer 编码数据吗？架构的定义在哪
例如，pandas 的 read_csv有一个 chunk_size允许 read_csv 的参数在 CSV 文件上返回一个迭代器，以便我们可以分块读取它。 Parquet 格式以块的形式存储数据，但
我正在尝试运行最新版本的 Parquet 工具，但遇到了一些问题。出于某种原因org.apache.hadoop.conf.Configuration不在阴影的 jar 里。 (我对 v1.6.0 也
我正在使用 Parquet 框架来编写 Parquet 文件。 我使用此构造函数创建了 Parquet 作家- public class ParquetBaseWriter extends Parqu
使用 spark 和钻头，我可以查询本地 Parquet 文件。 presto 是否提供相同的功能？ 换句话说，是否可以使用 presto 查询本地 Parquet 文件 - 无需通过 HDFS 或
我有一个加密的 parquet 数据文件，它被读取为一个输入流。我想从此输入流中提取单个 Parquet 记录。有什么办法可以做到这一点吗？在 avro 中，使用 DatumReader 是可能的。我
我知道 Apache Arrow Parquet 可以读取符合规范的 Delta 编码文件，但不能将它们写出。我想知道是否有任何常用的开源 C++/Python 库可以写出符合 Parquet 规范的
背景: DuckDB 允许直接查询 parquet 文件。例如con.execute("从'Hierarchy.parquet'中选择 *) Parquet 允许按列值对文件进行分区。当一个 Parq
有没有办法将一个巨大的 parquet 文件分成较小的文件(使用 Python)？保留所有列并划分行？谢谢 最佳答案 你可以用 dask 来做. import dask.dataframe as dd
我的 Parquet 文件为 800K 行 x 8.7K 列。我将其加载到 dask 数据框中: import dask.dataframe as dd dask_train_df = dd.read
我有数百个用 PyArrow 创建的 Parquet 文件。然而，其中一些文件的字段/列的名称(我们称其为 Orange)与原始列(称其为 Sporange)略有不同，因为其中一个使用了查询的变体。否
我正在尝试在配置单元中创建 Parquet 表。我可以创建它，但是当我运行 analyze table mytable compute statistics 时；我得到这个结果: numfiles=8
我知道 hdfs 会将文件拆分成大约 64mb 的 block 。我们有流式传输的数据，我们可以将它们存储到大文件或中等大小的文件中。列式文件存储的最佳大小是多少？如果我可以将文件存储到最小列为 64
我想使用 Apache 的 parquet-mr 项目通过 Java 以编程方式读取/写入 Parquet 文件。我似乎找不到任何有关如何使用此 API 的文档(除了查看源代码并查看它的使用方式)——
我在 Impala 中移动数据，而不是我的设计，我丢失了一些数据。我需要将数据从 Parquet 表复制回它们原来的非 Parquet 表。最初，开发人员使用脚本中的一个简单的一行来完成此操作。由于我
我是一名优秀的程序员,十分优秀！