使用 token 凭据的 Azure Synapse 身份验证

Question

我正在考虑在 azure synapse 上运行 Spark 批处理作业。我目前可以使用 az cli for synapse 来测试运行。在生产中，我需要通过外部应用程序(完美的流程)触发这些 Spark 提交。

为了提交 Spark 作业，我正在考虑使用 azure synapse sdk ( https://learn.microsoft.com/en-us/python/api/azure-mgmt-synapse/azure.mgmt.synapse.synapsemanagementclient?view=azure-python )

如何传递此处提到的 TokenCredential https://learn.microsoft.com/en-us/python/api/azure-mgmt-synapse/azure.mgmt.synapse.synapsemanagementclient?view=azure-python#constructor

Answer 1

这相当简单，我忽略了文档。

短代码如下

from azure.identity import ClientSecretCredential
from azure.synapse.spark import SparkClient
from azure.synapse.spark.models import SparkBatchJobOptions

def run(self, job_name: str, job_args):

   credential = ClientSecretCredential(self.tenant_id, self.client_id,self.client_secret)

   options = SparkBatchJobOptions.from_dict({
                "tags": None,
                "artifactId": None,
                "name": f"{job_name}",
                "file": f"{job_name}.py",
                "className": None,
                "args": job_args,
                "jars": [],
                "files": [],
                "archives": [],
                "conf": None,
                "driverMemory": "4g",
                "driverCores": 4,
                "executorMemory": "2g",
                "executorCores": 2,
                "numExecutors": 2,
            }

    })

    job = spark_client.spark_batch.create_spark_batch_job(options, detailed=False)