azure-data-lake - 使用 Azure Data Lake Analytics 处理事件中心捕获 AVRO 文件

Question

我正在尝试从事件中心捕获生成的 AVRO 文件中提取数据。在大多数情况下，这可以完美地工作。但是某些文件给我带来了问题。当我运行以下 U-SQL 作业时，出现错误:

USE DATABASE Metrics;
USE SCHEMA dbo;

REFERENCE ASSEMBLY [Newtonsoft.Json];
REFERENCE ASSEMBLY [Microsoft.Analytics.Samples.Formats];
REFERENCE ASSEMBLY [Avro];
REFERENCE ASSEMBLY [log4net];

USING Microsoft.Analytics.Samples.Formats.ApacheAvro;
USING Microsoft.Analytics.Samples.Formats.Json;
USING System.Text;

//DECLARE @input string = "adl://mydatalakestore.azuredatalakestore.net/event-hub-capture/v3/{date:yyyy}/{date:MM}/{date:dd}/{date:HH}/{filename}";
DECLARE @input string = "adl://mydatalakestore.azuredatalakestore.net/event-hub-capture/v3/2018/01/16/19/rcpt-metrics-us-es-eh-metrics-v3-us-0-35-36.avro";


@eventHubArchiveRecords =
    EXTRACT Body byte[], 
            date DateTime, 
            filename System.String
    FROM @input
    USING new AvroExtractor(@"
        {
            ""type"":""record"",
            ""name"":""EventData"",
            ""namespace"":""Microsoft.ServiceBus.Messaging"",
            ""fields"":[
                {""name"":""SequenceNumber"",""type"":""long""},
                {""name"":""Offset"",""type"":""string""},
                {""name"":""EnqueuedTimeUtc"",""type"":""string""},
                {""name"":""SystemProperties"",""type"":{""type"":""map"",""values"":[""long"",""double"",""string"",""bytes""]}},
                {""name"":""Properties"",""type"":{""type"":""map"",""values"":[""long"",""double"",""string"",""bytes""]}},
                {""name"":""Body"",""type"":[""null"",""bytes""]}
            ]
        }
    ");

@json =
    SELECT Encoding.UTF8.GetString(Body) AS json
    FROM @eventHubArchiveRecords;

OUTPUT @json
TO "/outputs/Avro/testjson.csv"
USING Outputters.Csv(outputHeader : true, quoting : true);

我收到以下错误:

来自用户代码的未处理异常:“字典中不存在给定的键。”

在用户类型“Microsoft.Analytics.Samples.Formats.ApacheAvro.AvroExtractor”上调用方法“Extract”时，报告了来自用户代码的未处理异常

我假设问题出在 Event Hub Capture 生成的 AVRO 文件中是否正确，或者我的代码有问题吗？

Answer 1

Key Not Present 错误指的是提取语句中的字段。它没有找到数据和文件名字段。我删除了这些字段，您的脚本在我的 ADLA 实例中正确运行。