azure - 如何使用数据工厂创建 Azure 按需 HD Insight Spark 集群

Question

我正在尝试使用 Azure 数据工厂使用 Hdi 版本 3.5 创建按需 HD Insight Spark 集群。数据工厂拒绝创建并显示错误消息

HdiVersion:'3.5' is not supported

如果目前无法创建按需 HD Insight Spark 集群，那么其他明智的选择是什么？我觉得很奇怪为什么微软没有在 Azure 数据工厂中添加按需 HD 洞察 Spark 集群。

Answer 1

这是一个完整的解决方案，它使用 ADF 在 C# 中安排自定义 .NET 事件，而 C# 事件又使用 ARM 模板和 SSH.NET 来执行运行 R 脚本的命令。

因此，ADF 用于调度 .NET Activity，Batch 服务用于运行 dll 中的代码，然后 HDInsight 集群的 json 模板文件存储在 blob 中，可以根据需要进行配置。

完整的描述在文章“Automating Azure: Creating an On-Demand HDInsight Cluster”中，但这里是 C# 代码，它是自动化的本质(其他一切只是设置位的管理工作):

using System;
  using System.Collections.Generic;
using Microsoft.Azure.Management.DataFactories.Models;
using Microsoft.Azure.Management.DataFactories.Runtime;

using Microsoft.Azure.Management.ResourceManager.Fluent;

using Microsoft.Azure.Management.ResourceManager.Fluent.Core;

 using Renci.SshNet;

 namespace VM

 {
  public class StartVM : IDotNetActivity
  {
      private IActivityLogger _logger;
      public IDictionary<string, string> Execute(
          IEnumerable<LinkedService> linkedServices,
          IEnumerable<Dataset> datasets,
          Activity activity,
          IActivityLogger logger)
      {
          _logger = logger;
          _logger.Write("Starting execution...");
          var credentials = SdkContext.AzureCredentialsFactory.FromServicePrincipal(
               "" // enter clientId here, this is the ApplicationID
               , "" // this is the Application secret key
               , "" // this is the tenant id 
               , AzureEnvironment.AzureGlobalCloud);
          var azure = Microsoft.Azure.Management.Fluent.Azure
              .Configure()
              .WithLogLevel(HttpLoggingDelegatingHandler.Level.Basic)
              .Authenticate(credentials)
              .WithDefaultSubscription();
          var groupName = "myResourceGroup";
          var location = Region.EuropeNorth;
          // create the resource group
          var resourceGroup = azure.ResourceGroups.Define(groupName)
              .WithRegion(location)
              .Create();
          // deploy the template
          var templatePath = "https://myblob.blob.core.windows.net/blobcontainer/myHDI_template.JSON";
          var paramPath = "https:// myblob.blob.core.windows.net/blobcontainer /myHDI_parameters.JSON";
          var deployment = azure.Deployments.Define("myDeployment")
              .WithExistingResourceGroup(groupName)
              .WithTemplateLink(templatePath, "0.9.0.0") // make sure it matches the file
              .WithParametersLink(paramPath, "1.0.0.0") // make sure it matches the file
              .WithMode(Microsoft.Azure.Management.ResourceManager.Fluent.Models.DeploymentMode.Incremental)
              .Create();
    _logger.Write("The cluster is ready...");
          executeSSHCommand();
          _logger.Write("The SSH command was executed...");
          _logger.Write("Deleting the cluster...");
          // delete the resource group
          azure.ResourceGroups.DeleteByName(groupName);

          return new Dictionary<string, string>();
      }
      private void executeSSHCommand()
      {
          ConnectionInfo ConnNfo = new ConnectionInfo("myhdi-ssh.azurehdinsight.net", "sshuser",
              new AuthenticationMethod[]{
              // Pasword based Authentication
              new PasswordAuthenticationMethod("sshuser","Addso@1234523123"),
              }
          );
          // Execute a (SHELL) Command - prepare upload directory
          using (var sshclient = new SshClient(ConnNfo))
          {
              sshclient.Connect();
              using (var cmd = sshclient.CreateCommand(
                  "hdfs dfs -copyToLocal \"wasbs:///rscript/test.R\";env -i R CMD BATCH --no-save --no-restore \"test.R\"; hdfs dfs -copyFromLocal -f \"test-output.txt\" \"wasbs:///rscript/test-output.txt\" "))
              {
                  cmd.Execute();

              }
              sshclient.Disconnect();
          }
      }
  }

  }

祝你好运!

费奥多