scala - 出现类似需要结构类型的错误，但在 spark scala 中得到了简单结构类型的字符串

Question

这是我的架构

root
 |-- DataPartition: string (nullable = true)
 |-- TimeStamp: string (nullable = true)
 |-- PeriodId: long (nullable = true)
 |-- FinancialAsReportedLineItemName: struct (nullable = true)
 |    |-- _VALUE: string (nullable = true)
 |    |-- _languageId: long (nullable = true)
 |-- FinancialLineItemSource: long (nullable = true)
 |-- FinancialStatementLineItemSequence: long (nullable = true)
 |-- FinancialStatementLineItemValue: double (nullable = true)
 |-- FiscalYear: long (nullable = true)
 |-- IsAnnual: boolean (nullable = true)
 |-- IsAsReportedCurrencySetManually: boolean (nullable = true)
 |-- IsCombinedItem: boolean (nullable = true)
 |-- IsDerived: boolean (nullable = true)
 |-- IsExcludedFromStandardization: boolean (nullable = true)
 |-- IsFinal: boolean (nullable = true)
 |-- IsTotal: boolean (nullable = true)
 |-- ParentLineItemId: long (nullable = true)
 |-- PeriodPermId: struct (nullable = true)
 |    |-- _VALUE: long (nullable = true)
 |    |-- _objectTypeId: long (nullable = true)
 |-- ReportedCurrencyId: long (nullable = true)

根据上面的模式，我正在尝试这样做

val temp = tempNew1
      .withColumn("FinancialAsReportedLineItemName", $"FinancialAsReportedLineItemName._VALUE")
      .withColumn("FinancialAsReportedLineItemName_languageId", $"FinancialAsReportedLineItemName._languageId")
      .withColumn("PeriodPermId", $"PeriodPermId._VALUE")
      .withColumn("PeriodPermId_objectTypeId", $"PeriodPermId._objectTypeId").drop($"AsReportedItem").drop($"AsReportedItem")

我不知道我在这里错过了什么。我遇到以下错误

Exception in thread "main" org.apache.spark.sql.AnalysisException: Can't extract value from FinancialAsReportedLineItemName#2262: need struct type but got string;

Answer 1

问题是当 FinancialAsReportedLineItemName 列已被 FinancialAsReportedLineItemName._VALUE 替换时，您正在尝试访问 FinancialAsReportedLineItemName._languageId

你应该改变下面两行

.withColumn("FinancialAsReportedLineItemName", $"FinancialAsReportedLineItemName._VALUE")
.withColumn("FinancialAsReportedLineItemName_languageId", $"FinancialAsReportedLineItemName._languageId")

到

.withColumn("FinancialAsReportedLineItemName_value", $"FinancialAsReportedLineItemName._VALUE")
.withColumn("FinancialAsReportedLineItemName_languageId", $"FinancialAsReportedLineItemName._languageId")

如果 FinancialAsReportedLineItemName_value 列名称应该是 FinancialAsReportedLineItemName 那么您应该将 withColumns 交换为

.withColumn("FinancialAsReportedLineItemName_languageId", $"FinancialAsReportedLineItemName._languageId")    
.withColumn("FinancialAsReportedLineItemName", $"FinancialAsReportedLineItemName._VALUE")