apache-spark - 使用ES-Spark连接器创建嵌套对象数组

Question

我有一个模式的Spark DataFrame:

 |-- ROW_ID: string (nullable = true)
 |-- SUBJECT_ID: string (nullable = true)
 |-- HADM_ID: string (nullable = true)
 |-- CHARTDATE: string (nullable = true)
 |-- CHARTTIME: string (nullable = true)
 |-- STORETIME: string (nullable = true)
 |-- CATEGORY: string (nullable = true)
 |-- DESCRIPTION: string (nullable = true)
 |-- CGID: string (nullable = true)
 |-- ISERROR: string (nullable = true)
 |-- TEXT: string (nullable = true)
 |-- annotations: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- text: string (nullable = true)
 |    |    |-- subject: string (nullable = true)
 |    |    |-- polarity: integer (nullable = false)
 |    |    |-- confidence: float (nullable = false)
 |    |    |-- historyOf: integer (nullable = false)
 |    |    |-- ontologyMappings: array (nullable = true)
 |    |    |    |-- element: struct (containsNull = true)
 |    |    |    |    |-- preferredText: string (nullable = true)
 |    |    |    |    |-- codingScheme: string (nullable = true)
 |    |    |    |    |-- code: string (nullable = true)
 |    |    |    |    |-- cui: string (nullable = true)
 |    |    |    |    |-- tui: string (nullable = true)

我正在ElasticSearch中为整个结构建立索引，但是注解字段(StructTypes的数组)和ontologyMappings字段都没有显示为嵌套模式。例如，ontologyMappings映射如下所示:

"ontologyMappings": {
                "properties": {
                  "code": {
                    "type": "text",
                    "fields": {
                      "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                      }
                    }
                  },
                  "codingScheme": {
                    "type": "text",
                    "fields": {
                      "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                      }
                    }
                  },
                  "cui": {
                    "type": "text",
                    "fields": {
                      "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                      }
                    }
                  },
                  "preferredText": {
                    "type": "text",
                    "fields": {
                      "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                      }
                    }
                  },
                  "code": {
                    "type": "text",
                    "fields": {
                      "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                      }
                    }
                  },

有没有一种方法可以强制将它们编写为嵌套类型，而不仅仅是具有属性字段的对象？我希望能够运行查询来查找包含一个实例的文档，该实例中的代码是特定的字符串，并且关联的极性为1(在onologyMappings下)。没有嵌套，这种关联是不可能的。

Answer 1

定义嵌套哪些字段的PUT请求是必要的。有效负载如下所示。

"""{"mappings":{
      "data":{
         "properties":{
            "annotations":{
               "type":"nested",
               "properties":{
                  "ontologyMappings":{
                     "type":"nested",
                     "properties":{
                        "code":{
                           "type":"text",
                           "fields":{
                              "keyword":{
                                 "type":"keyword"
                              }
                           }
                        },
                        "codingScheme":{
                           "type":"text",
                           "fields":{
                              "keyword":{
                                 "type":"keyword"
                              }
                           }
                        },
                        "cui":{
                           "type":"text",
                           "fields":{
                              "keyword":{
                                 "type":"keyword"
                              }
                           }
                        },
                        "preferredText":{
                           "type":"text",
                           "fields":{
                              "keyword":{
                                 "type":"keyword"
                              }
                           }
                        },
                        "tui":{
                           "type":"text",
                           "fields":{
                              "keyword":{
                                 "type":"keyword"
                              }
                           }
                        }
                     }
                  }
               }
            }
         }
      }
   }
}
    """