个人中心
gpt4 book ai didi

azure - 如何一次性配置Azure流分析作业？

转载 作者：行者123 更新时间：2023-12-03 02:13:26 25 4
gpt4 key购买 nike

我需要为其作业的系统数据预配连接到存储帐户的 Azure 流分析作业。这要求其托管标识有权访问此存储帐户。这是通过角色分配来完成的，这只能在存在托管标识时完成。

微软提供的azure-streamanalytics-cicd cli，生成了arm模板和参数文件，下面显然无法直接部署。

这可以在一次部署中完成吗？这似乎是一个先有鸡还是先有蛋的问题:需要托管身份来执行角色分配，但角色分配必须在部署完整作业之前就位。

每当我在没有角色分配的情况下部署完整作业时，都会收到错误无法使用作业存储帐户进行身份验证。

看来我需要将部署分成三个步骤:

  1. 配置已配置存储帐户的骨架作业，但没有实际工作负载。这不会产生错误，但会点亮托管标识。
  2. 对存储帐户执行托管身份的角色分配。
  3. 规定工作工作量。

ARM 模板:

{
  "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "ASAApiVersion": {
      "type": "string"
    },
    "StreamAnalyticsJobName": {
      "type": "string",
      "minLength": 3,
      "maxLength": 63,
      "metadata": {
        "description": "Stream Analytics Job Name, can contain alphanumeric characters and hypen and must be 3-63 characters long"
      }
    },
    "Location": {
      "type": "string"
    },
    "OutputStartMode": {
      "type": "string",
      "allowedValues": [
        "JobStartTime",
        "CustomTime",
        "LastOutputEventTime"
      ]
    },
    "OutputStartTime": {
      "type": "string"
    },
    "DataLocale": {
      "type": "string"
    },
    "OutputErrorPolicy": {
      "type": "string",
      "allowedValues": [
        "Drop",
        "Stop"
      ]
    },
    "EventsLateArrivalMaxDelayInSeconds": {
      "type": "int"
    },
    "EventsOutOfOrderMaxDelayInSeconds": {
      "type": "int"
    },
    "EventsOutOfOrderPolicy": {
      "type": "string",
      "allowedValues": [
        "Adjust",
        "Drop"
      ]
    },
    "StreamingUnits": {
      "type": "int",
      "minValue": 1,
      "maxValue": 396,
      "metadata": {
        "description": "Number of Streaming Units"
      },
      "allowedValues": [
        1,
        3,
        6,
        12,
        18,
        24,
        30,
        36,
        42,
        48,
        54,
        60,
        66,
        72,
        78,
        84,
        90,
        96,
        102,
        108,
        114,
        120,
        126,
        132,
        138,
        144,
        150,
        156,
        162,
        168,
        174,
        180,
        186,
        192,
        198,
        204,
        210,
        216,
        222,
        228,
        234,
        240,
        246,
        252,
        258,
        264,
        270,
        276,
        282,
        288,
        294,
        300,
        306,
        312,
        318,
        324,
        330,
        336,
        342,
        348,
        354,
        360,
        366,
        372,
        378,
        384,
        390,
        396
      ]
    },
    "CompatibilityLevel": {
      "type": "string",
      "allowedValues": [
        "1.0",
        "1.1",
        "1.2"
      ]
    },
    "ContentStoragePolicy": {
      "type": "string",
      "allowedValues": [
        "SystemAccount",
        "JobStorageAccount"
      ]
    },
    "JobStorageAccountName": {
      "type": "string"
    },
    "JobStorageAuthMode": {
      "type": "string",
      "allowedValues": [
        "ConnectionString",
        "Msi"
      ]
    },
    "CustomCodeStorageAccountName": {
      "type": "string"
    },
    "CustomCodeStorageAccountKey": {
      "type": "string"
    },
    "CustomCodeContainer": {
      "type": "string"
    },
    "CustomCodePath": {
      "type": "string"
    },
    "Input_InputIoTHub_iotHubNamespace": {
      "type": "string"
    },
    "Input_InputIoTHub_consumerGroupName": {
      "type": "string"
    },
    "Input_InputIoTHub_endpoint": {
      "type": "string"
    },
    "Input_InputIoTHub_sharedAccessPolicyName": {
      "type": "string"
    },
    "Input_InputIoTHub_sharedAccessPolicyKey": {
      "type": "string"
    },
    "Output_outputmsgunfilteredcosmos_accountId": {
      "type": "string"
    },
    "Output_outputmsgunfilteredcosmos_accountKey": {
      "type": "string"
    },
    "Output_outputmsgunfilteredcosmos_database": {
      "type": "string"
    },
    "Output_outputmsgunfilteredcosmos_collectionNamePattern": {
      "type": "string"
    },
    "Output_outputmsgunfilteredcosmos_documentId": {
      "type": "string"
    }
  },
  "resources": [
    {
      "type": "Microsoft.StreamAnalytics/StreamingJobs",
      "apiVersion": "[parameters('ASAApiVersion')]",
      "name": "[parameters('StreamAnalyticsJobName')]",
      "location": "[parameters('Location')]",
      "identity": {
        "type": "SystemAssigned"
      },
      "properties": {
        "outputStartMode": "[parameters('OutputStartMode')]",
        "outputStartTime": "[if(equals(parameters('OutputStartMode'),'CustomTime'), parameters('OutputStartTime'), json('null'))]",
        "sku": {
          "name": "standard"
        },
        "jobType": "Cloud",
        "eventsOutOfOrderPolicy": "[parameters('EventsOutOfOrderPolicy')]",
        "outputErrorPolicy": "[parameters('OutputErrorPolicy')]",
        "eventsOutOfOrderMaxDelayInSeconds": "[parameters('EventsOutOfOrderMaxDelayInSeconds')]",
        "eventsLateArrivalMaxDelayInSeconds": "[parameters('EventsLateArrivalMaxDelayInSeconds')]",
        "dataLocale": "[parameters('DataLocale')]",
        "compatibilityLevel": "[parameters('CompatibilityLevel')]",
        "jobStorageAccount": {
          "accountName": "[parameters('JobStorageAccountName')]",
          "authenticationMode": "[parameters('JobStorageAuthMode')]"
        },
        "contentStoragePolicy": "[parameters('ContentStoragePolicy')]",
        "externals": {
          "storageAccount": {
            "accountName": "[parameters('CustomCodeStorageAccountName')]",
            "accountKey": "[parameters('CustomCodeStorageAccountKey')]"
          },
          "container": "[parameters('CustomCodeContainer')]",
          "path": "[parameters('CustomCodePath')]"
        },
        "transformation": {
          "name": "Transformation",
          "properties": {
            "streamingUnits": "[parameters('StreamingUnits')]",
            "query": "SELECT\r\n    GetMetadataPropertyValue(InputIoTHub, '[EventId]') AS Id,\r\n    GetMetadataPropertyValue(InputIoTHub, '[IotHub].[ConnectionDeviceId]') AS deviceId,\r\n    GetMetadataPropertyValue(InputIoTHub, '[IoTHub].[EnqueuedTime]') AS timeStamp,\r\n    InputIoTHub.*\r\nINTO\r\n    outputmsgunfilteredcosmos\r\nFROM\r\n    InputIoTHub\r\n"
          }
        },
        "inputs": [
          {
            "name": "InputIoTHub",
            "properties": {
              "type": "Stream",
              "datasource": {
                "type": "Microsoft.Devices/IotHubs",
                "properties": {
                  "iotHubNamespace": "[parameters('Input_InputIoTHub_iotHubNamespace')]",
                  "consumerGroupName": "[parameters('Input_InputIoTHub_consumerGroupName')]",
                  "endpoint": "[parameters('Input_InputIoTHub_endpoint')]",
                  "sharedAccessPolicyName": "[parameters('Input_InputIoTHub_sharedAccessPolicyName')]",
                  "sharedAccessPolicyKey": "[parameters('Input_InputIoTHub_sharedAccessPolicyKey')]"
                }
              },
              "compression": {
                "type": "None"
              },
              "serialization": {
                "type": "Json",
                "properties": {
                  "encoding": "UTF8"
                }
              }
            }
          }
        ],
        "outputs": [
          {
            "name": "outputmsgunfilteredcosmos",
            "properties": {
              "datasource": {
                "type": "Microsoft.Storage/DocumentDB",
                "properties": {
                  "accountId": "[parameters('Output_outputmsgunfilteredcosmos_accountId')]",
                  "accountKey": "[parameters('Output_outputmsgunfilteredcosmos_accountKey')]",
                  "database": "[parameters('Output_outputmsgunfilteredcosmos_database')]",
                  "collectionNamePattern": "[parameters('Output_outputmsgunfilteredcosmos_collectionNamePattern')]",
                  "partitionKey": null,
                  "documentId": "[parameters('Output_outputmsgunfilteredcosmos_documentId')]"
                }
              }
            }
          }
        ]
      }
    }
  ]
}

Arm模板参数:

{
  "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "ASAApiVersion": {
      "value": "2017-04-01-preview"
    },
    "StreamAnalyticsJobName": {
      "value": "myasajob"
    },
    "Location": {
      "value": "Central US"
    },
    "OutputStartMode": {
      "value": "JobStartTime"
    },
    "OutputStartTime": {
      "value": "2019-01-01T00:00:00Z"
    },
    "DataLocale": {
      "value": "nl-NL"
    },
    "OutputErrorPolicy": {
      "value": "Stop"
    },
    "EventsLateArrivalMaxDelayInSeconds": {
      "value": 5
    },
    "EventsOutOfOrderMaxDelayInSeconds": {
      "value": 0
    },
    "EventsOutOfOrderPolicy": {
      "value": "Adjust"
    },
    "StreamingUnits": {
      "value": 1
    },
    "CompatibilityLevel": {
      "value": "1.2"
    },
    "ContentStoragePolicy": {
      "value": "JobStorageAccount"
    },
    "JobStorageAccountName": {
      "value": "mystorageaccount"
    },
    "JobStorageAuthMode": {
      "value": "Msi"
    },
    "CustomCodeStorageAccountName": {
      "value": "mystorageaccount"
    },
    "CustomCodeStorageAccountKey": {
      "value": null
    },
    "CustomCodeContainer": {
      "value": "43803218-0998-487b-9d49-4eb00ef41ca5"
    },
    "CustomCodePath": {
      "value": "UserCustomCode.zip"
    },
    "Input_InputIoTHub_iotHubNamespace": {
      "value": "myiothub"
    },
    "Input_InputIoTHub_consumerGroupName": {
      "value": "$Default"
    },
    "Input_InputIoTHub_endpoint": {
      "value": "messages/events"
    },
    "Input_InputIoTHub_sharedAccessPolicyName": {
      "value": "DPSRegistry"
    },
    "Input_InputIoTHub_sharedAccessPolicyKey": {
      "value": null
    },
    "Output_outputmsgunfilteredcosmos_accountId": {
      "value": "mycosmos"
    },
    "Output_outputmsgunfilteredcosmos_accountKey": {
      "value": null
    },
    "Output_outputmsgunfilteredcosmos_database": {
      "value": "mycosmosdb"
    },
    "Output_outputmsgunfilteredcosmos_collectionNamePattern": {
      "value": "unfiltered"
    },
    "Output_outputmsgunfilteredcosmos_documentId": {
      "value": ""
    }
  }
}

最佳答案

这是先有鸡还是先有蛋的问题。目前，在使用系统分配的 MSI 时，您无法一次性使用该配置配置作业。由于该身份不会被授予访问存储帐户的权限 - 它之前不存在，您无法执行此操作。

对于系统分配的 MSI，您的部署步骤是正确的，您需要:

  • 在没有任何子资源(输入、输出、转换...)的情况下创建作业
  • 授予对存储帐户的访问权限
  • 使用正确的配置创建子资源

或者，您可以使用连接字符串创建作业，然后尝试切换到 MSI 以解决先有鸡还是先有蛋的问题，但这仍然超过 1 个步骤，并且需要额外的凭据，因此并不能真正解决太多问题。

但是如果你使用user assigned MSI ，它应该正常工作，因为您可以在创建作业之前授予对存储帐户的访问权限。

关于azure - 如何一次性配置Azure流分析作业？，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/72415746/

25 4 0
文章推荐： azure - 有没有办法使用 Azure B2C 以编程方式使用部署在 Azure APIM 中的 REST API？
文章推荐： syntax-highlighting - vim 匹配删除其他匹配
文章推荐： asp.net-core - ASP.NET Core/MVC6 中的 WebTelemetryInitializerBase
文章推荐： python - 安装 cx_Oracle 库并依赖于 Azure 函数应用
行者123
个人简介

我是一名优秀的程序员,十分优秀！

滴滴打车优惠券免费领取
滴滴打车优惠券
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com