apache-kafka - 带有可选记录的 AVRO 模式

Question

大家好，我需要为以下示例创建 AVRO 模式；

{ "Car" : { "Make" : "Ford" , "Year": 1990 , "Engine" : "V8" , "VIN" : "123123123" , "Plate" : "XXTT9O", 
"Accident" : { "Date" :"2020/02/02" , "Location" : "NJ" , "Driver" : "Joe" } ,
"Owner" :  { "Name" : "Joe" , "LastName" : "Doe" } } 

Accident 和 Owner 是可选对象，创建的 schema 还需要验证以下子集消息；

{ "Car" : { "Make" : "Tesla" , "Year": 2020 , "Engine" : "4ELEC" , "VIN" : "54545426" , "Plate" : "TESLA" }

我阅读了 AVRO 规范并看到了很多可选属性和数组示例，但没有一个适用于记录。如何将记录定义为可选的？谢谢。

没有任何可选属性的以下架构正在运行。

{
  "name": "MyClass",  "type": "record",  "namespace": "com.acme.avro",  "fields": [
    {
      "name": "Car",   "type": {
        "name": "Car","type": "record","fields": [
          { "name": "Make",    "type": "string"   },
          { "name": "Year",    "type": "int"      },
          { "name": "Engine",    "type": "string" },
          { "name": "VIN",    "type": "string"    },
          { "name": "Plate",    "type": "string"  },
          { "name": "Accident",
                "type":
                { "name": "Accident",
                  "type": "record",
                  "fields": [
                       { "name": "Date","type": "string" },
                       { "name": "Location","type": "string" },
                       { "name": "Driver", "type": "string" }
                    ]
                 }
          },

          { "name": "Owner",
            "type":
                {"name": "Owner",
                 "type": "record",
                 "fields": [
                {"name": "Name",  "type": "string" },
                {"name": "LastName", "type": "string" }
              ]
            }
          }
        ]
      }
    }
  ]
}

当我按照建议更改 Owner 对象时，avro-tool 返回错误。

      { "name": "Owner",
        "type": [
            "null",
            "record" : {
              "name": "Owner",
              "fields": [
                 {"name": "Name",  "type": "string" },
                 {"name": "LastName", "type": "string" }
               ]
             }
          ] , "default": null  }
       ]
   }
}

]

测试:

Projects/avro_test$ java -jar avro-tools-1.8.2.jar fromjson --schema-file CarStackOver.avsc Car.json > o2
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "main" org.apache.avro.SchemaParseException: org.codehaus.jackson.JsonParseException: Unexpected character (':' (code 58)): was expecting comma to separate ARRAY entries
 at [Source: org.apache.hadoop.fs.ChecksumFileSystem$FSDataBoundedInputStream@4034c28c; line: 26, column: 13]
        at org.apache.avro.Schema$Parser.parse(Schema.java:1034)
        at org.apache.avro.Schema$Parser.parse(Schema.java:1004)
        at org.apache.avro.tool.Util.parseSchemaFromFS(Util.java:165)
        at org.apache.avro.tool.DataFileWriteTool.run(DataFileWriteTool.java:83)
        at org.apache.avro.tool.Main.run(Main.java:87)
        at org.apache.avro.tool.Main.main(Main.java:76)
Caused by: org.codehaus.jackson.JsonParseException: Unexpected character (':' (code 58)): was expecting comma to separate ARRAY entries
 at [Source: org.apache.hadoop.fs.ChecksumFileSystem$FSDataBoundedInputStream@4034c28c; line: 26, column: 13]
        at org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1433)
        at org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:521)
        at org.codehaus.jackson.impl.JsonParserMinimalBase._reportUnexpectedChar(JsonParserMinimalBase.java:442)
        at org.codehaus.jackson.impl.Utf8StreamParser.nextToken(Utf8StreamParser.java:482)
        at org.codehaus.jackson.map.deser.std.BaseNodeDeserializer.deserializeArray(JsonNodeDeserializer.java:222)
        at org.codehaus.jackson.map.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:200)
        at org.codehaus.jackson.map.deser.std.BaseNodeDeserializer.deserializeArray(JsonNodeDeserializer.java:224)
        at org.codehaus.jackson.map.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:200)
        at org.codehaus.jackson.map.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:197)
        at org.codehaus.jackson.map.deser.std.BaseNodeDeserializer.deserializeArray(JsonNodeDeserializer.java:224)
        at org.codehaus.jackson.map.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:200)
        at org.codehaus.jackson.map.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:58)
        at org.codehaus.jackson.map.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:15)
        at org.codehaus.jackson.map.ObjectMapper._readValue(ObjectMapper.java:2704)
        at org.codehaus.jackson.map.ObjectMapper.readTree(ObjectMapper.java:1344)
        at org.apache.avro.Schema$Parser.parse(Schema.java:1032)

Answer 1

您可以通过与 null 联合来使记录成为可选的。

像这样:

  {
    "name": "Owner",
    "type": [
      "null",
      {
        "name": "Owner",
        "type": "record",
        "fields": [
          { "name": "Name", type": "string" },
          { "name": "LastName", type": "string" },
        ]

      }
    ],
    "default": null
  },