java - 编译具有其他依赖项的Hadoop Java程序

Question

我正在尝试构建一个Hadoop程序，其目的是基于this tutorial来将我先前上传到HDFS的文件cat编码，该程序如下所示:

import java.io.*;
import java.net.URI;
import org.apache.hadoop.fs.*;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;

public class ReadHDFS {
    public static void main(String[] args) throws IOException {

        String uri = args[0];

        Configuration conf = new Configuration();
        FileSystem fs = FileSystem.get(URI.create(uri), conf);
        FSDataInputStream in = null ;

        try
        {
            in = fs.open(new Path(uri));
            IOUtils.copyBytes(in, System.out, 4096, false);
        }
        finally
        {
            IOUtils.closeStream(in);
        }   
    }
}

在我看来，该教程是有缺陷的，因为-根据我的理解-IOUtils是 apache.commons库的一部分。但是，尽管我在尝试部署的程序中添加了以下行:

import org.apache.commons.compress.utils.IOUtils;

我仍然遇到以下错误:



那是:

FileSystemCat.java:37: error: cannot find symbol
        IOUtils.copyBytes(in, System.out, 4096, false);
               ^
  symbol:   method copyBytes(InputStream,PrintStream,int,boolean)
  location: class IOUtils
FileSystemCat.java:40: error: cannot find symbol
        IOUtils.closeStream(in);
                            ^
  symbol:   variable in
  location: class FileSystemCat
2 errors

我正在使用以下命令在NameNode上执行它:

javac -cp /usr/local/hadoop/share/hadoop/common/hadoop-common-2.8.1.jar:/home/ubuntu/job_program/commons-io-2.5/commons-io-2.5.jar FileSystemCat.java 

Answer 1

~/.bashrc的必要附录:

# Classpath for Java

# export HADOOP_CLASSPATH=$($HADOOP_HOME/bin/hadoop classpath)
export HADOOP_CLASSPATH=$($HADOOP_HOME/bin/hadoop classpath)

如何在底部编译程序:

javac -cp ${HADOOP_CLASSPATH}:commons-io-2.5.jar ReaderHDFS.java

如何为该程序生成jar文件:

jar cf rhdfs.jar ReaderHDFS*.class

运行命令:

$HADOOP_HOME/bin/hadoop jar rhdfs.jar ReaderHDFS hdfs://master:9000/input_1/codes.txt

这是程序:

import org.apache.hadoop.io.IOUtils;
//import org.apache.commons.io.IOUtils;
import java.io.*;
import java.net.URI;
import org.apache.hadoop.fs.*;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;

public class ReaderHDFS {
    public static void main(String[] args) throws IOException {

        String uri = args[0];

        Configuration conf = new Configuration();
        FileSystem fs = FileSystem.get(URI.create(uri), conf);
        FSDataInputStream in = null ;

        try
        {
            in = fs.open(new Path(uri));
            IOUtils.copyBytes(in, System.out, 4096, false);
        }
        finally
        {
            IOUtils.closeStream(in);
        }   
    }
}