azure - 使用 Azure 表存储的 Databricks Scala 应用程序出现错误

Question

首先感谢您抽出时间回答下一个问题:)

我正在尝试将 Databricks Scala 应用程序与 Azure 表存储连接，但出现以下错误:

Azure 表 Scala APP

Error:

NoSuchMethodError:reactor.netty.http.client.HttpClient.resolver(Lio/netty/resolver/AddressResolverGroup;)Lreactor/netty/transport/ClientTransport;

atcom.azure.core.http.netty.NettyAsyncHttpClientBuilder.build(NettyAsyncHttpClientBuilder.java:94)atcom.azure.core.http.netty.NettyAsyncHttpClientProvider.createInstance(NettyAsyncHttpClientProvider.java:18)atcom.azure.core.implementation.http.HttpClientProviders.createInstance(HttpClientProviders.java:58)at com.azure.core.http.HttpClient.createDefault(HttpClient.java:50) atcom.azure.core.http.HttpClient.createDefault(HttpClient.java:40) atcom.azure.core.http.HttpPipelineBuilder.build(HttpPipelineBuilder.java:62)atcom.azure.data.tables.BuilderHelper.buildPipeline(BuilderHelper.java:122)atcom.azure.data.tables.TableServiceClientBuilder.buildAsyncClient(TableServiceClientBuilder.java:161)atcom.azure.data.tables.TableServiceClientBuilder.buildClient(TableServiceClientBuilder.java:93)

我附上代码:

val clientCredential: ClientSecretCredential  = new ClientSecretCredentialBuilder()
.tenantId(tenantID)
.clientId(client_san_Id)
.clientSecret(client_san_Secret)
.build()


val tableService = new TableServiceClientBuilder()
.endpoint("https://<Resource-Table>.table.core.windows.net")
.credential(clientCredential)
.buildClient()

非常感谢您抽出时间!

Answer 1

首先，您需要在 Azure databricks 上安装存储。

然后使用下面的代码来挂载表存储。

dbutils.fs.mount(
  source = "wasbs://<container-name>@<storage-account-name>.blob.core.windows.net/<directory-name>",
  mountPoint = "/mnt/<mount-name>",
  extraConfigs = Map("<conf-key>" -> dbutils.secrets.get(scope = "<scope-name>", key = "<key-name>")))

使用以下代码访问表存储:

// scala
val df = spark.read.text("/mnt/<mount-name>/...")
val df = spark.read.text("dbfs:/<mount-name>/...")

可以引用这个notebook

另请参阅此 article作者:高里·马哈詹