hadoop - 集群节点上的Hbase映射器/缩减库要求

Question

我有一个分布式的hadoop集群，在其hdfs上运行了hbase。为了使用hbase构建一个map / reduce作业，我包括以下依赖项:

    <dependencies>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-core</artifactId>
            <version>1.2.1</version>
            <scope>provided</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-client</artifactId>
            <version>1.2.3</version>
            <scope>provided</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-server</artifactId>
            <version>1.2.3</version>
            <scope>provided</scope>
        </dependency>
    </dependencies>

我正在尝试弄清楚如何正确部署所有这些库。 hadoop是否应该在其类路径中仅包含$ HBASE_HOME / lib / *？有很多重叠和版本冲突。似乎我只需要一些子集，但是hbase文档仅提供了一些帮助:

Replace the Hadoop Bundled With HBase! Because HBase depends on Hadoop, it bundles an instance of the Hadoop jar under its lib directory. The bundled jar is ONLY for use in standalone mode. In distributed mode, it is critical that the version of Hadoop that is out on your cluster match what is under HBase. Replace the hadoop jar found in the HBase lib directory with the hadoop jar you are running on your cluster to avoid version mismatch issues. Make sure you replace the jar in HBase everywhere on your cluster. Hadoop version mismatch issues have various manifestations but often all looks like its hung up.

我找不到在哪儿可以肯定地告诉您需要向hadoop的计算节点添加哪些hbase库。

Answer 1

我试图通过实验回答这个问题。我似乎需要使其工作的最少项目集是:

hbase-client-1.2.3.jar -> ../../../../hbase/lib/hbase-client-1.2.3.jar
hbase-common-1.2.3.jar -> ../../../../hbase/lib/hbase-common-1.2.3.jar
hbase-hadoop2-compat-1.2.3.jar -> ../../../../hbase/lib/hbase-hadoop2-compat-1.2.3.jar
hbase-hadoop-compat-1.2.3.jar -> ../../../../hbase/lib/hbase-hadoop-compat-1.2.3.jar
hbase-prefix-tree-1.2.3.jar -> ../../../../hbase/lib/hbase-prefix-tree-1.2.3.jar
base-protocol-1.2.3.jar -> ../../../../hbase/lib/hbase-protocol-1.2.3.jar
hbase-server-1.2.3.jar -> ../../../../hbase/lib/hbase-server-1.2.3.jar
metrics-core-2.2.0.jar -> ../../../../hbase/lib/metrics-core-2.2.0.jar

稍微解释一下，我的hadoop安装位于/ home / hadoop中，而hbase安装位于/ home / hbase中。只需将apache hadoop和hbase压缩包分别解压缩到“hadoop”和“hbase”用户的主目录中，即可“安装”这些文件。 reducer 为空的TableReducer。

它似乎可以运行-或至少它没有与hbase相关的ClassNotFound异常。

我不确定这是否是真实的集合...我只能说我的空TableReducer确实需要这些东西。