java - 是否有任何现有的 API 可以从 JAVA 读取和写入 HDFS

Question

是否存在可从 HDFS 读取/写入的现有 API，以及如何实现它的最佳实践。

Answer 1

下面是代码片段

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;
import org.apache.hadoop.util.Progressable;



String Dest = "/user/pkumar/test.xml";

 Configuration conf = new Configuration();
       FileSystem fs = FileSystem.get(URI.create(Dest),conf);
        Path path = new Path(Dest);

    if(!fs.exists(path)){
        OutputStream out = fs.create(path, new Progressable(){
            public void progress(){
                System.out.print(".");
            }
        });
        System.out.println();
        IOUtils.copyBytes(connect, out, 4096, true);
    }

在您的 pom.xml 中使用以下依赖项

<dependencies>
<dependency>
     <groupId>org.apache.hadoop</groupId>
     <artifactId>hadoop-common</artifactId>
     <version>2.3.0-cdh5.1.3</version>
</dependency>
<dependency>
     <groupId>org.apache.hadoop</groupId>
     <artifactId>hadoop-client</artifactId>
     <version>2.3.0-cdh5.1.3</version>
</dependency>
<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpcore</artifactId>
    <version>4.3.2</version>
</dependency>
<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpclient</artifactId>
    <version>4.3.2</version>
</dependency>

</dependencies>