hadoop - 在 cloudera CDH5.1.0 上运行 giraph 时出错

Question

所以，我在集群上安装了 Hadoop 2.3.0-cdh5.1.0……我通过以下命令编译了 giraph……mvn clean package -DskipTests -Dhadoop=non_secure -Phadoop_2.0.0所以，编译工作得很好，然后 pagerank 基准测试也成功了..
所以，我猜 giraph 编译得很好..

现在，我正在尝试运行我的代码..

hadoop jar graphAnalytics-0.0.1-SNAPSHOT-jar-with-dependencies.jar org.apache.giraph.GiraphRunner  -Dgiraph.zkList=srv-110-07:2181,srv-110-08:2181,srv-210-08:2181 org.examples.graphAnalytics.PageRankVertex -w 1 -vif org.examples.graphAnalytics.PageRankInputFormat

And I am getting this error:
Exception in thread "main" java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected
    at org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
    at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:458)
    at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:343)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1295)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1292)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at....

我知道，它是一个不完整的命令，但它已经在提示.. :-/

我的 POM 文件如下所示:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

  <modelVersion>4.0.0</modelVersion>

  <groupId>org.rdio.graphanalytics</groupId>

  <artifactId>graphAnalytics</artifactId>

  <version>0.0.1-SNAPSHOT</version>

  <name>Graph Analytics Algorithms</name>

  <description>Package for Graph Analytics Algorithms

</description>

<repositories>

    <repository>

      <id>cloudera</id>

      <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>

    </repository>

  </repositories>

  <dependencies>

  <dependency>

  <groupId>org.apache.giraph</groupId>

  <artifactId>giraph-core</artifactId>

  <version>1.0.0</version>

  </dependency>

    <dependency>

            <groupId>junit</groupId>

            <artifactId>junit-dep</artifactId>

            <version>4.8.2</version>

     </dependency>

    <dependency>

    <groupId>org.apache.hadoop</groupId>

    <artifactId>hadoop-client</artifactId>

    <version>2.3.0-cdh5.1.0</version>

    </dependency>

  </dependencies>

  <build>

    <plugins>

    <plugin>

            <groupId>org.apache.maven.plugins</groupId>

            <artifactId>maven-compiler-plugin</artifactId>

            <version>2.1</version>

            <configuration>

                <source>1.6</source>

                <target>1.6</target>

            </configuration>

        </plugin>

        <plugin>

            <groupId>org.apache.maven.plugins</groupId>

            <artifactId>maven-assembly-plugin</artifactId>

            <version>2.4</version>

            <configuration>

                <descriptorRefs>

                    <descriptorRef>jar-with-dependencies</descriptorRef>

                </descriptorRefs>



            </configuration>

            <executions>

                <execution>

                    <id>assemble-all</id>

                    <phase>package</phase>

                    <goals>

                        <goal>single</goal>

                    </goals>

                </execution>

            </executions>

        </plugin>

    </plugins>

  </build>

</project>

我该如何解决这个问题？
谢谢

Answer 1

我认为 Giraph 不支持 Hadoop 2.2 及更高版本:http://giraph.apache.org/build.html

CDH 5.1 是 Hadoop 2.3 + 补丁。

您可能会成功地为更高版本的 Hadoop 修改和重建 Giraph。但是您看到的错误是针对比编译时更高版本的 Hadoop 运行的典型症状。