maven - Cloudera VM 上的 Apache Giraph - org.apache.hadoop 的 POM :hadoop-core:jar:2. 6.0 缺失，无依赖信息

Question

我是 Hadoop/Giraph 和 Java 的新手。作为任务的一部分，我在其上下载了 Cloudera Quickstart VM 和 Giraph。我正在使用这本书，名为“使用 Apache Giraph 进行实用图形分析；作者:Shaposhnik、Roman、Martella、Claudio、Logothetis、Dionysios”，我尝试从中运行第 111 页上的第一个示例(Twitter Followership Graph)。

尝试在集群 2.6.0-mr1-cdh5.12.0 上使用 hadoop 版本运行更改后的 pom.xml 文件时，请发现以下错误

`[cloudera@quickstart first]$ mvn clean install
[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building book-examples 1.0.0
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ book-examples ---
[INFO] Deleting /home/cloudera/workspace/first/target
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ book-examples ---
[debug] execute contextualize
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 0 resource
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ book-examples ---
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[INFO] Compiling 1 source file to /home/cloudera/workspace/first/target/classes
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] /home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[5,27] error: package org.apache.hadoop.io does not exist
[ERROR] /home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[6,27] error: package org.apache.hadoop.io does not exist
[ERROR] /home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[7,29] error: cannot find symbol
[ERROR]  package org.apache.hadoop.util
/home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[14,17] error: cannot find symbol
[ERROR]  class IntWritable
/home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[14,30] error: cannot find symbol
[ERROR]  class IntWritable
/home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[15,0] error: cannot find symbol
[ERROR]  class NullWritable
/home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[15,14] error: cannot find symbol
[ERROR]  class NullWritable
/home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[17,28] error: cannot find symbol
[ERROR]  class GiraphHelloWorld
/home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[18,3] error: cannot find symbol
[ERROR]  class GiraphHelloWorld
/home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[18,16] error: cannot find symbol
[ERROR]  class GiraphHelloWorld
/home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[19,12] error: cannot find symbol
[ERROR]  class GiraphHelloWorld
/home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[24,12] error: cannot find symbol
[ERROR]  class GiraphHelloWorld
/home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[24,25] error: cannot find symbol
[ERROR]  class GiraphHelloWorld
/home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[34,14] error: cannot find symbol
[INFO] 14 errors 
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 6.495s
[INFO] Finished at: Fri Dec 08 14:57:01 PST 2017
[INFO] Final Memory: 18M/57M

`

我根据其他 Stack overflow 响应添加了 Cloudera Repository。请找到适用于上述错误的更新的 pom xml:

`<?xml version="1.0" encoding="UTF-8"?>
<project>
<modelVersion>4.0.0</modelVersion>

<groupId>giraph</groupId>
<artifactId>book-examples</artifactId>
<version>1.0.0</version>

<dependencies>
<dependency>
<groupId>org.apache.giraph</groupId>
<artifactId>giraph-core</artifactId>
<version>1.1.0</version>
</dependency>

<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>2.6.0-mr1-cdh5.12.0</version>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>2.4</version>
                <executions>
                    <execution>
                        <id>create-jar-bundle</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                        <configuration>
                            <descriptorRefs>
                                <descriptorRef>jar-with-dependencies</descriptorRef>
                            </descriptorRefs>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
</plugins>
</build>

<repositories>
        <repository>
            <id>cloudera</id>
            <url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>
    </repositories>

</project>`

书上的hadoop版本是1.2.1。两个版本之间的书籍依赖性存在问题。

如果有人能帮助我理解如何处理这个错误，那就太好了。

提前致谢。

Answer 1

您书本中的 pom.xml 已过时。改用这个。来源: book examples repository在 Github 上。

编辑:

您想使用最新版本的hadoop-core，但最新版本Maven Central Repository (默认存储库)提供的是 1.2.1 .您将需要使用 Cloudera Repository获取最新版本的库。为此，只需将存储库添加到您的 pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
<project>
    <modelVersion>4.0.0</modelVersion>
    <groupId>giraph</groupId>
    <artifactId>book-examples</artifactId>
    <version>1.0.0</version>

    <dependencies>
        <dependency>
            <groupId>org.apache.giraph</groupId>
            <artifactId>giraph-core</artifactId>
            <version>1.1.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-core</artifactId>
            <version>2.6.0-mr1-cdh5.12.0</version>
        </dependency>
    </dependencies>

    <build>
    </build>

    <repositories>
        <repository>
            <id>cloudera</id>
            <url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>
    </repositories>
</project>

您现在应该看到 Maven 首先尝试在 Cloudera 中找到 jar，然后返回到 Central:

$ mvn clean install
[INFO] Scanning for projects...
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] Building book-examples 1.0.0
[INFO] ------------------------------------------------------------------------
...
Downloading: https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/hadoop/hadoop-core/2.6.0-mr1-cdh5.12.0/hadoop-core-2.6.0-mr1-cdh5.12.0.pom
Downloaded: https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/hadoop/hadoop-core/2.6.0-mr1-cdh5.12.0/hadoop-core-2.6.0-mr1-cdh5.12.0.pom (6.4 kB at 2.9 kB/s)
...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.661 s
[INFO] Finished at: 2017-12-08T22:56:04+01:00
[INFO] Final Memory: 15M/224M
[INFO] ------------------------------------------------------------------------

编辑 2:

好吧，我终于明白了。从版本 2 开始，Hadoop 更改了它的包装，因此您应该使用 hadoop-client，而不是声明对 hadoop-core 的依赖，它是一个聚合了所有必要依赖的元数据包为你。从 pom.xml 中删除 hadoop-core 依赖项并添加:

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-client</artifactId>
    <version>2.9.0</version>
</dependency>