azure - 如何使用 Python 将数据引入 Azure 集群

Question

我有一组数据想要通过 Azure 数据资源管理器使用 KQL 进行查询。我每隔几秒就有一个连续的传入数据源。我想将此数据放入 Azure 集群中以运行查询。

我使用 python 库探索了一些选项，但它只支持使用文件或 blob。

https://learn.microsoft.com/en-us/azure/data-explorer/python-ingest-data

如何使用 python 将单个记录放入其中，以便我可以使用 Azure 数据资源管理器进行查询。

Answer 1

这里有使用 azure-kusto-ingest 库从数据帧(包含一条或多条记录)提取的示例代码 here

cluster = "https://ingest-{cluster_name}.kusto.windows.net/"

# In case you want to authenticate with AAD application.
client_id = "<insert here your AAD application id>"
client_secret = "<insert here your AAD application key>"

# read more at https://docs.microsoft.com/en-us/onedrive/find-your-office-365-tenant-id
authority_id = "<insert here your tenant id>"

kcsb = KustoConnectionStringBuilder.with_aad_application_key_authentication(cluster, client_id, client_secret, authority_id)

client = QueuedIngestClient(kcsb)

# there are a lot of useful properties, make sure to go over docs and check them out
ingestion_props = IngestionProperties(
    database="{database_name}",
    table="{table_name}",
    data_format=DataFormat.CSV,
    # in case status update for success are also required (remember to import ReportLevel from azure.kusto.ingest)
    # report_level=ReportLevel.FailuresAndSuccesses,
    # in case a mapping is required (remember to import IngestionMappingKind from azure.kusto.data.data_format)
    # ingestion_mapping_reference="{json_mapping_that_already_exists_on_table}",
    # ingestion_mapping_kind= IngestionMappingKind.JSON,
)

###########################
## ingest from dataframe ##
###########################

import pandas

fields = ["id", "name", "value"]
rows = [[1, "abc", 15.3], [2, "cde", 99.9]]

df = pandas.DataFrame(data=rows, columns=fields)

client.ingest_from_dataframe(df, ingestion_properties=ingestion_props)