scala - 使用azure databricks scala将数据从blob存储加载到sql数据仓库

Question

我正在尝试使用 azure databricks scala 将数据从 blob 存储加载到 SQL 数据仓库中。

spark.conf.set("spark.sql.parquet.writeLegacyFormat","true")    
df.write.format("com.databricks.spark.sqldw")
.option("url",sqlDwUrlSmall)
.option("dbtable", "Person")        
.option("forward_spark_azure_storage_credentials","True")
.option("tempdir",tempDir).mode("overwrite").save()

我收到此错误

Underlying SQLException(s): - com.microsoft.sqlserver.jdbc.SQLServerException: External file access failed due to internal error: 'Error occurred while accessing HDFS: Java exception raised on call to HdfsBridge_IsDirExist. Java exception message: HdfsBridge::isDirExist - Unexpected error encountered checking whether directory exists or not: StorageException: This request is not authorized to perform this operation.' [ErrorCode = 105019] [SQLState = S0001]

Answer 1

使用azure databricks scala成功将数据从blob存储加载到sql数据仓库。

• 确保通过正确的路径。

• 确保传递“tempDir”，如下格式所示。

tempDir = "wasbs://" + blobContainer + "@" + blobStorage +"/tempDirs"

引用: Load data into Azure SQL Data Warehouse

希望这有帮助。