java - 如何使用 java sdk 将文件从我的语言环境上传和下载到 azure adls？

Question

     String clientId ="***********";
     String authTokenEndpoint = "***";
     String clientKey = "****";
     AccessTokenProvider provider = new ClientCredsTokenProvider(authTokenEndpoint, clientId, 
           clientKey);
     String accountFQDN = "******";  // 
     ADLStoreClient client = ADLStoreClient.createClient(accountFQDN, provider);

1.如何使用ADLStoreClient obj上传文件？有没有类似的东西

s3.putObject(BucketName, FileName, new File(Srcfileloc))

在 azure SDK 中。

2.文档说明创建文件并写入文件。如果我的数据类似于图像或大 tar 文件怎么办!

3.对于 adls gen 1 和 adls gen2，我是否必须遵循两种单独的方法？相同的代码可以重复使用吗？

Answer 1

Q1: How to upload file to Azure data lake

根据我的测试，我们可以使用以下代码将文件从本地上传到Azure数据湖。我使用 800 MB tar.gz 文件进行测试

1. SDK

 <dependency>
        <groupId>com.microsoft.azure</groupId>
        <artifactId>azure-data-lake-store-sdk</artifactId>
        <version>2.2.3</version>
      </dependency>
      <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-nop</artifactId>
        <version>1.7.21</version>
      </dependency>

代码

AccessTokenProvider provider = new ClientCredsTokenProvider(authTokenEndpoint, clientId, clientKey);
        ADLStoreClient client = ADLStoreClient.createClient(accountFQDN, provider);

        String filepath="D:\\download\\ideaIU-2019.3.3.tar.gz";
        Path path = Paths.get(filepath);

        String filename = "test/" +path.getFileName();

        try{
            FileInputStream in = new FileInputStream(filepath);
            ADLFileOutputStream out =client.createFile(filename,IfExists.OVERWRITE);
            int bufSize = 4 * 1000 * 1000;
            out.setBufferSize(bufSize);
            byte[] buffer = new byte[bufSize];
            int n;

            while ((n = in.read(buffer)) != -1) {
                out.write(buffer, 0, n);

            }
            out.close();
            in.close();
        }
        catch (Exception e){
           // process exception


        }

更多详情请引用sample

Q2: How to upload file to Azure data lake Gen2

根据我的测试，我们可以使用下面的代码

1. SDK

<dependency>
      <groupId>com.azure</groupId>
      <artifactId>azure-storage-file-datalake</artifactId>
      <version>12.0.1</version>
    </dependency>
    <dependency>
      <groupId>com.fasterxml.jackson.core</groupId>
      <artifactId>jackson-core</artifactId>
      <version>2.10.3</version>
    </dependency>

代码

try {
          StorageSharedKeyCredential sharedKeyCredential =
                  new StorageSharedKeyCredential(accountName, accountKey);
          DataLakeServiceClientBuilder builder = new DataLakeServiceClientBuilder();
          builder.credential(sharedKeyCredential);
          builder.endpoint("https://" + accountName + ".dfs.core.windows.net");
          DataLakeServiceClient client = builder.buildClient();
          String fileSystem = "test";
          DataLakeFileSystemClient fileSystemClient = client.getFileSystemClient(fileSystem);
          DataLakeDirectoryClient directoryClient =fileSystemClient.getDirectoryClient("testFolder");
          String filepath="D:\\download\\ideaIU-2019.3.3.tar.gz";
          Path path = Paths.get(filepath);
          DataLakeFileClient fileClient = directoryClient.createFile(String.valueOf(path.getFileName()));
          fileClient.uploadFromFile(filepath, true);

      }catch(Exception ex){

          System.out.println(ex.getMessage());
      }

更多详情请引用docuemnt和 sample