gpt4 book ai didi

apache-spark - Apache Spark拼花地板: Cannot build an empty group

转载 作者:行者123 更新时间:2023-12-04 14:40:14 26 4
gpt4 key购买 nike

我使用的是Apache Spark 2.1.1(使用的是2.1.0,并且今天仍然相同)。
我有一个数据集:

root
|-- muons: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- reco::Candidate: struct (nullable = true)
| | |-- qx3_: integer (nullable = true)
| | |-- pt_: float (nullable = true)
| | |-- eta_: float (nullable = true)
| | |-- phi_: float (nullable = true)
| | |-- mass_: float (nullable = true)
| | |-- vertex_: struct (nullable = true)
| | | |-- fCoordinates: struct (nullable = true)
| | | | |-- fX: float (nullable = true)
| | | | |-- fY: float (nullable = true)
| | | | |-- fZ: float (nullable = true)
| | |-- pdgId_: integer (nullable = true)
| | |-- status_: integer (nullable = true)
| | |-- cachePolarFixed_: struct (nullable = true)
| | |-- cacheCartesianFixed_: struct (nullable = true)

如您所见,此架构中有3个空结构。我知道100%我可以阅读/操作/做任何事情。但是,当我尝试以拼花形式写入磁盘时,出现以下异常:
dsReduced.write.format("parquet").save(outputPathName):

java.lang.IllegalStateException: Cannot build an empty group
at org.apache.parquet.Preconditions.checkState(Preconditions.java:91)
at org.apache.parquet.schema.Types$BaseGroupBuilder.build(Types.java:622)
at org.apache.parquet.schema.Types$BaseGroupBuilder.build(Types.java:497)
at org.apache.parquet.schema.Types$Builder.named(Types.java:286)
at org.apache.spark.sql.execution.datasources.parquet.ParquetSchemaConverter.convertField(ParquetSchemaConverter.scala:535)
at org.apache.spark.sql.execution.datasources.parquet.ParquetSchemaConverter.convertField(ParquetSchemaConverter.scala:321)
at org.apache.spark.sql.execution.datasources.parquet.ParquetSchemaConverter$$anonfun$convertField$1.apply(ParquetSchemaConverter.scala:534)
at org.apache.spark.sql.execution.datasources.parquet.ParquetSchemaConverter$$anonfun$convertField$1.apply(ParquetSchemaConverter.scala:533)

所以,基本上,我想了解它是错误还是预期的行为???我还假定它与空结构有关。任何帮助将非常感激!

更新:我已经快速创建了剥离版本,并且该版本可以正常运行!任何见解都将真正有帮助!

VK

最佳答案

Parquet不写空结构:

有关更多信息,请参见https://issues.apache.org/jira/browse/SPARK-20593

VK

关于apache-spark - Apache Spark拼花地板: Cannot build an empty group,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/43767358/

26 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com