java - 使用 Hadoop 构建 Giraph

Question

我正在尝试使用 Hadoop 2.7.1 设置 Giraph尽我所能，它似乎不起作用。

我试过以下链接，

• http://giraph.apache.org/build.html
• https://blog.cloudera.com/blog/2014/02/how-to-write-and-run-giraph-jobs-on-hadoop/
• http://lab.hypotheses.org/1207

当我尝试使用 mvn -Phadoop_yarn -Dhadoop.version=2.7.1 -DskipTests package 构建时会发生什么

hadoopuser:/usr/local/giraph$ mvn -Phadoop_yarn -Dhadoop.version=2.7.1 -DskipTests package
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO] 
[INFO] Apache Giraph Parent
[INFO] Apache Giraph Core
[INFO] Apache Giraph Examples
[INFO] Apache Giraph Distribution
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Apache Giraph Parent 1.1.0
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- mavanagaiata:0.5.0:branch (git-commit) @ giraph-parent ---
[INFO] 
[INFO] --- mavanagaiata:0.5.0:commit (git-commit) @ giraph-parent ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.2:enforce (default) @ giraph-parent ---
[INFO] Skipping Rule Enforcement.
[INFO] 
[INFO] --- maven-dependency-versions-check-plugin:2.0.2:check (default) @ giraph-parent ---
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.4:process (default) @ giraph-parent ---
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:analyze-only (default) @ giraph-parent ---
[INFO] Skipping plugin execution
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:analyze-duplicate (default) @ giraph-parent ---
[INFO] Skipping plugin execution
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:analyze-dep-mgt (default) @ giraph-parent ---
[INFO] Skipping plugin execution
[INFO] 
[INFO] --- maven-duplicate-finder-plugin:1.0.4:check (default) @ giraph-parent ---
[INFO] 
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ giraph-parent ---
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Apache Giraph Core 1.1.0
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- mavanagaiata:0.5.0:branch (git-commit) @ giraph-core ---
[INFO] 
[INFO] --- mavanagaiata:0.5.0:commit (git-commit) @ giraph-core ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.2:enforce (default) @ giraph-core ---
[INFO] Skipping Rule Enforcement.
[INFO] 
[INFO] --- maven-dependency-versions-check-plugin:2.0.2:check (default) @ giraph-core ---
[INFO] 
[INFO] --- munge-maven-plugin:1.0:munge (munge) @ giraph-core ---
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.4:process (default) @ giraph-core ---
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ giraph-core ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.0:compile (default-compile) @ giraph-core ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 711 source files to /usr/local/giraph/giraph-core/target/munged/classes
[INFO] -------------------------------------------------------------
[WARNING] COMPILATION WARNING : 
[INFO] -------------------------------------------------------------
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeByteArrayOutputStream.java:[42,29] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeByteArrayOutputStream.java:[44,25] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeByteArrayOutputStream.java:[57,32] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeArrayReads.java:[39,32] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeArrayReads.java:[42,29] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeArrayReads.java:[44,25] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/DistributedCacheUtils.java: Some input files use or override a deprecated API.
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/DistributedCacheUtils.java: Recompile with -Xlint:deprecation for details.
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/TestGraph.java: Some input files use unchecked or unsafe operations.
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/TestGraph.java: Recompile with -Xlint:unchecked for details.
[INFO] 10 warnings 
[INFO] -------------------------------------------------------------
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyClient.java:[84,68] cannot find symbol
  symbol:   variable SASL_PROPS
  location: class org.apache.hadoop.security.SaslRpcServer
[ERROR] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyServer.java:[105,62] cannot find symbol
  symbol:   variable SASL_PROPS
  location: class org.apache.hadoop.security.SaslRpcServer
[INFO] 2 errors 
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Giraph Parent ............................... SUCCESS [  4.374 s]
[INFO] Apache Giraph Core ................................. FAILURE [  8.469 s]
[INFO] Apache Giraph Examples ............................. SKIPPED
[INFO] Apache Giraph Distribution ......................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 13.136 s
[INFO] Finished at: 2015-11-16T10:19:07+05:30
[INFO] Final Memory: 45M/335M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.0:compile (default-compile) on project giraph-core: Compilation failure: Compilation failure:
[ERROR] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyClient.java:[84,68] cannot find symbol
[ERROR] symbol:   variable SASL_PROPS
[ERROR] location: class org.apache.hadoop.security.SaslRpcServer
[ERROR] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyServer.java:[105,62] cannot find symbol
[ERROR] symbol:   variable SASL_PROPS
[ERROR] location: class org.apache.hadoop.security.SaslRpcServer
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :giraph-cor

---

当我尝试使用 mvn -Phadoop_yarn -Dhadoop.version=2.7.1 -DskipTests install 构建时会发生什么

hadoopuser:/usr/local/giraph$ mvn -Phadoop_yarn -Dhadoop.version=2.7.1 -DskipTests install
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO] 
[INFO] Apache Giraph Parent
[INFO] Apache Giraph Core
[INFO] Apache Giraph Examples
[INFO] Apache Giraph Distribution
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Apache Giraph Parent 1.1.0
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- mavanagaiata:0.5.0:branch (git-commit) @ giraph-parent ---
[INFO] 
[INFO] --- mavanagaiata:0.5.0:commit (git-commit) @ giraph-parent ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.2:enforce (default) @ giraph-parent ---
[INFO] Skipping Rule Enforcement.
[INFO] 
[INFO] --- maven-dependency-versions-check-plugin:2.0.2:check (default) @ giraph-parent ---
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.4:process (default) @ giraph-parent ---
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:analyze-only (default) @ giraph-parent ---
[INFO] Skipping plugin execution
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:analyze-duplicate (default) @ giraph-parent ---
[INFO] Skipping plugin execution
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:analyze-dep-mgt (default) @ giraph-parent ---
[INFO] Skipping plugin execution
[INFO] 
[INFO] --- maven-duplicate-finder-plugin:1.0.4:check (default) @ giraph-parent ---
[INFO] 
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ giraph-parent ---
[INFO] 
[INFO] --- maven-install-plugin:2.3.1:install (default-install) @ giraph-parent ---
[INFO] Installing /usr/local/giraph/pom.xml to /home/hadoopuser/.m2/repository/org/apache/giraph/giraph-parent/1.1.0/giraph-parent-1.1.0.pom
[INFO] Installing /usr/local/giraph/target/giraph-parent-1.1.0-site.xml to /home/hadoopuser/.m2/repository/org/apache/giraph/giraph-parent/1.1.0/giraph-parent-1.1.0-site.xml
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Apache Giraph Core 1.1.0
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- mavanagaiata:0.5.0:branch (git-commit) @ giraph-core ---
[INFO] 
[INFO] --- mavanagaiata:0.5.0:commit (git-commit) @ giraph-core ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.2:enforce (default) @ giraph-core ---
[INFO] Skipping Rule Enforcement.
[INFO] 
[INFO] --- maven-dependency-versions-check-plugin:2.0.2:check (default) @ giraph-core ---
[INFO] 
[INFO] --- munge-maven-plugin:1.0:munge (munge) @ giraph-core ---
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.4:process (default) @ giraph-core ---
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ giraph-core ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.0:compile (default-compile) @ giraph-core ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 711 source files to /usr/local/giraph/giraph-core/target/munged/classes
[INFO] -------------------------------------------------------------
[WARNING] COMPILATION WARNING : 
[INFO] -------------------------------------------------------------
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeByteArrayOutputStream.java:[42,29] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeByteArrayOutputStream.java:[44,25] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeByteArrayOutputStream.java:[57,32] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeArrayReads.java:[39,32] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeArrayReads.java:[42,29] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeArrayReads.java:[44,25] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/DistributedCacheUtils.java: Some input files use or override a deprecated API.
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/DistributedCacheUtils.java: Recompile with -Xlint:deprecation for details.
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/TestGraph.java: Some input files use unchecked or unsafe operations.
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/TestGraph.java: Recompile with -Xlint:unchecked for details.
[INFO] 10 warnings 
[INFO] -------------------------------------------------------------
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyClient.java:[84,68] cannot find symbol
  symbol:   variable SASL_PROPS
  location: class org.apache.hadoop.security.SaslRpcServer
[ERROR] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyServer.java:[105,62] cannot find symbol
  symbol:   variable SASL_PROPS
  location: class org.apache.hadoop.security.SaslRpcServer
[INFO] 2 errors 
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Giraph Parent ............................... SUCCESS [  4.360 s]
[INFO] Apache Giraph Core ................................. FAILURE [  7.473 s]
[INFO] Apache Giraph Examples ............................. SKIPPED
[INFO] Apache Giraph Distribution ......................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 12.122 s
[INFO] Finished at: 2015-11-16T10:26:04+05:30
[INFO] Final Memory: 46M/374M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.0:compile (default-compile) on project giraph-core: Compilation failure: Compilation failure:
[ERROR] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyClient.java:[84,68] cannot find symbol
[ERROR] symbol:   variable SASL_PROPS
[ERROR] location: class org.apache.hadoop.security.SaslRpcServer
[ERROR] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyServer.java:[105,62] cannot find symbol
[ERROR] symbol:   variable SASL_PROPS
[ERROR] location: class org.apache.hadoop.security.SaslRpcServer
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :giraph-core







---
Version information follows,

Hadoop:  

    hadoopuser:~$ hadoop version
    Hadoop 2.7.1
    Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r 15ecc87ccf4a0228f35af08fc56de536e6ce657a
    Compiled by jenkins on 2015-06-29T06:04Z
    Compiled with protoc 2.5.0
    From source with checksum fc0a1a23fc1868e4d5ee7fa2b28a58a
    This command was run using /usr/local/hadoop/share/hadoop/common/hadoop-common-2.7.1.jar

Giraph is version 1.1.0 downloaded from here : http://archive.apache.org/dist/giraph/giraph-1.1.0/giraph-dist-1.1.0-src.tar.bz2

Java

    hadoopuser:~$ java -version
    java version "1.8.0_05"
    Java(TM) SE Runtime Environment (build 1.8.0_05-b13)
    Java HotSpot(TM) 64-Bit Server VM (build 25.5-b02, mixed mode)

Maven 

    hadoopuser:~$ mvn -version
    Apache Maven 3.3.3 (7994120775791599e205a5524ec3e0dfe41d4a06; 2015-04-22T17:27:37+05:30)
    Maven home: /opt/apache-maven-3.3.3
    Java version: 1.8.0_05, vendor: Oracle Corporation
    Java home: /usr/lib/jvm/jdk1.8.0/jre
    Default locale: en_IN, platform encoding: UTF-8
    OS name: "linux", version: "3.13.0-68-generic", arch: "amd64", family: "unix"

---

编辑:

正如 Steve Kaeser 在他的 answer 中所建议的那样我更改了 pom.xlm，更改部分如下所示。

  <properties>
<hadoop.version>SET_HADOOP_VERSION_USING_MVN_DASH_D_OPTION</hadoop.version>
    <munge.symbols>PURE_YARN</munge.symbols>
    <!-- TODO: add these checks eventually -->
    <project.enforcer.skip>true</project.enforcer.skip>
    <giraph.maven.dependency.plugin.skip>true</giraph.maven.dependency.plugin.skip>
    <giraph.maven.duplicate.finder.skip>true</giraph.maven.duplicate.finder.skip>
  </properties>

执行此操作后，当我运行 mvn -Phadoop_yarn -Dhadoop.version=2.7.1 -DskipTests install 时，我得到了很多输出，但它失败了。部分输出如下所示，

[INFO] 
[INFO] --- apache-rat-plugin:0.8:check (default) @ giraph-core ---
[INFO] Exclude: CODE_CONVENTIONS
[INFO] Exclude: **/target/**
[INFO] Exclude: /_bsp/**
[INFO] Exclude: .checkstyle
[INFO] Exclude: .reviewboardrc
[INFO] Exclude: .git/**
[INFO] Exclude: .gitignore
[INFO] Exclude: .arcconfig
[INFO] Exclude: *.patch
[INFO] Exclude: .idea/**
[INFO] Exclude: **/*.iml
[INFO] Exclude: **/*.ipr
[INFO] Exclude: **/test/resources/**
[INFO] Exclude: giraph-gora/conf/*
[INFO] Exclude: giraph-rexster/giraph-kibble/src/main/resources/META-INF/services/com.tinkerpop.rexster.extension.RexsterExtension
[INFO] Exclude: src/site/resources/images/*.svg
[INFO] 
[INFO] >>> findbugs-maven-plugin:2.5.1:check (default) > :findbugs @ giraph-core >>>
[INFO] 
[INFO] --- findbugs-maven-plugin:2.5.1:findbugs (findbugs) @ giraph-core ---
[INFO] Fork Value is true
     [java] The following errors occurred during analysis:
     [java] Nov 17, 2015 3:16:36 PM edu.umd.cs.findbugs.TextUIBugReporter reportAnalysisError
     [java] SEVERE: Error scanning java/util/Comparator for referenced classes
     [java] java.lang.ArrayIndexOutOfBoundsException: 51966
     [java]     at org.objectweb.asm.ClassReader.readUTF8(Unknown Source)
     [java]     at org.objectweb.asm.ClassReader.readClass(Unknown Source)

.
.
.
.
.
.
.
.
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Giraph Parent ............................... SUCCESS [  4.140 s]
[INFO] Apache Giraph Core ................................. FAILURE [ 37.541 s]
[INFO] Apache Giraph Examples ............................. SKIPPED
[INFO] Apache Giraph Distribution ......................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 41.972 s
[INFO] Finished at: 2015-11-17T15:16:37+05:30
[INFO] Final Memory: 68M/537M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:findbugs-maven-plugin:2.5.1:findbugs (findbugs) on project giraph-core: Execution findbugs of goal org.codehaus.mojo:findbugs-maven-plugin:2.5.1:findbugs failed: Java returned: 1 -> [Help 1]
[ERROR] 

Answer 1

我今天遇到了这个问题。最后我需要两个补丁来修复它:

1. 打开基本目录中的 pom.xml 并替换
   <munge.symbols>PURE_YARN,STATIC_SASL_SYMBOL</munge.symbols>与:
   <munge.symbols>PURE_YARN</munge.symbols>

2. 应用在此处找到的补丁:GIRAPH-1110.02.patch

3. 使用 clean 构建Maven 标志:
   mvn -Phadoop_yarn -Dhadoop.version=2.7.3 clean package -DskipTests