java - Spark : Named_struct requires at least one argument

Question

我正在将行数据集映射到自定义类的数据集。

Dataset<Row> rows= sparkSession.read().orc(path);
Dataset<customClass> dataset =
  rows.map(I'm parsing row to map it to customClass, 
    Encoders.bean(customClass.class));

我收到了这个 AnalysisException:

AnalysisException: cannot resolve 'named_struct()' due to data type mismatch: input to function named_struct requires at least one argument;

我正在使用 Spark 2.3.0 并使用 javaBeans 对我的自定义类进行编码。

我检查了编码器是否有效推断了架构，情况确实如此。因此，从技术上讲， map 操作应该可以工作。

有人遇到过这个异常消息吗？ name_struct 函数有什么作用？没有找到与Spark相关的相关资料...

root
 |-- field1: struct (nullable = true)
 |    |-- value: string (nullable = true)
 |-- field2: string (nullable = true)
 |-- field3: integer (nullable = true)
 |-- field4: double (nullable = true)
 |-- field5: struct (nullable = true)
 |    |-- value: double (nullable = true)
 |-- field6: struct (nullable = true)
 |    |-- field61: double (nullable = true)
 |    |-- field62: string (nullable = true)
 |    |-- field63: integer (nullable = true)
 |    |-- field64: struct (nullable = true)
 |    |    |-- value: string (nullable = true)
 |-- field7: struct (nullable = true)
 |    |-- value: double (nullable = true)
 |-- field8: struct (nullable = true)
 |    |-- value: double (nullable = true)
 |-- field9: struct (nullable = true)
 |    |-- field91: map (nullable = true)
 |    |    |-- key: struct
 |    |    |-- value: struct (valueContainsNull = true)
 |    |    |    |-- value: string (nullable = true)
 |    |    |    |-- field911: struct (nullable = true)
 |    |    |    |    |-- value: double (nullable = true)
 |    |    |    |-- field912: struct (nullable = true)
 |    |    |    |    |-- value: double (nullable = true)
 |    |    |    |-- field913: map (nullable = true)
 |    |    |    |    |-- key: struct
 |    |    |    |    |-- value: struct (valueContainsNull = true)
 |    |    |    |    |    |-- value: integer (nullable = false)
 |    |    |    |    |    |-- field9131: struct (nullable = true)
 |    |    |    |    |    |    |-- value: double (nullable = true)
 |    |    |    |    |    |-- field9131: struct (nullable = true)
 |    |    |    |    |    |    |-- value: double (nullable = true)
 |    |    |    |-- field914: struct (nullable = true)
 |    |    |    |    |-- value: double (nullable = true)
 |    |    |    |-- field915: string (nullable = true)
 |-- field10: string (nullable = true)
 |-- field11: struct (nullable = true)
 |    |-- field111: map (nullable = true)
 |    |    |-- key: struct
 |    |    |-- value: struct (valueContainsNull = true)
 |    |    |    |-- value: integer (nullable = false)
 |    |    |    |-- field1111: struct (nullable = true)
 |    |    |    |    |-- value: double (nullable = true)
 |    |    |    |-- field1112: struct (nullable = true)
 |    |    |    |    |-- value: double (nullable = true)
 |-- field12: boolean (nullable = true)
 |-- field13: struct (nullable = true)
 |    |-- field131: integer (nullable = false)
 |    |-- field132: integer (nullable = false)
 |-- field14: struct (nullable = true)
 |    |-- field141: string (nullable = true)

Answer 1

我终于找到了为什么会出现这个 named_struct 错误:我使用的字段之一被声明为 final，这意味着它没有 setter。这违反了 JavaBean 契约。