java - 一个简单的 Spark 应用程序，存在 MojoExecutionException 问题

Question

我正在研究 Apache Spark 并尝试在 Java 程序中运行 Spark，但我遇到了一些问题。
我使用 Maven 构建程序并创建了一个新的 Maven 项目。这是 pom.xml

  <groupId>spark</groupId>
  <artifactId>testspark</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <packaging>jar</packaging>
  <name>testspark</name>
  <url>http://maven.apache.org</url>
  <properties>
   <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>
  <repositories>
     <repository>
       <id>Akka repository</id>
       <url>http://repo.akka.io/releases</url>
     </repository>
  </repositories>
  <dependencies>
    <dependency>
     <groupId>junit</groupId>
     <artifactId>junit</artifactId>
     <version>3.8.1</version>
     <scope>test</scope>
   </dependency>
   <dependency> 
     <groupId>org.apache.spark</groupId>
     <artifactId>spark-core_2.10</artifactId>
     <version>0.9.0-incubating</version>
     </dependency>
  </dependencies>

我的 Java Spark 程序在这里:

public class SimpleApp{ 
    public static void main(String[] args) { 
            String logFile = "/usr/Java/spark-0.9.0/README.md"; 
            JavaSparkContext sc = new JavaSparkContext("local", "Simple App","/usr/Java/spark-0.9.0",new String[] { "target/testspark-0.0.1-SNAPSHOT.jar" }); 
            JavaRDD<String> logData = sc.textFile(logFile).cache(); 
            long numAs = logData.filter(new Function<String, Boolean>() { 
                    public Boolean call(String s) { 
                            return s.contains("a"); 
                    } 
            }).count(); 

            long numBs = logData.filter(new Function<String, Boolean>() { 
                    public Boolean call(String s) { 
                            return s.contains("b"); 
                    } 
            }).count(); 

            System.out.println("Lines with a: " + numAs + ", lines with b: " + numBs); 
    } 

}

我使用 mvn package 和 mvn -X exec:java -Dexec.mainClass="spark.testspark.SimpleApp" 来运行程序。我可以得到程序结果:Lines with a: 70, lines with b: 44，但是之后出现如下问题:

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3:java (default-cli) on project testspark: An exception occured while executing the Java class. null: InterruptedException -> [Help 1] 
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3:java (default-cli) on project testspark: An exception occured while executing the Java class. null 
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:216) 
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) 
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) 
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:108) 
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:76) 
        at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51) 
        at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:116) 
        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:361) 
        at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155) 
        at org.apache.maven.cli.MavenCli.execute(MavenCli.java:584) 
        at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:213) 
        at org.apache.maven.cli.MavenCli.main(MavenCli.java:157) 
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
        at java.lang.reflect.Method.invoke(Method.java:606) 
        at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289) 
        at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229) 
        at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415) 
        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356) 
Caused by: org.apache.maven.plugin.MojoExecutionException: An exception occured while executing the Java class. null 
        at org.codehaus.mojo.exec.ExecJavaMojo.execute(ExecJavaMojo.java:345) 
        at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:133) 
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208) 
        ... 19 more 
Caused by: java.lang.InterruptedException 
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) 
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2052) 
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) 
        at org.apache.spark.scheduler.SparkListenerBus$$anon$2.run(SparkListenerBus.scala:42) 
[ERROR] 
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles: 
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
root@ubuntu:~/workspace/testspark# mvn -X exec:java -Dexec.mainClass="spark.testspark.SimpleApp"

我被这些问题弄糊涂了。任何想法？谢谢你!

Answer 1

我想你忘了在最后调用 JavaSparkContext stop() 方法。以下也会给你错误吗？

public class SimpleApp{ 
public static void main(String[] args) { 
        String logFile = "/usr/Java/spark-0.9.0/README.md"; 
        JavaSparkContext sc = new JavaSparkContext("local", "Simple App","/usr/Java/spark-0.9.0",new String[] { "target/testspark-0.0.1-SNAPSHOT.jar" }); 
        JavaRDD<String> logData = sc.textFile(logFile).cache(); 
        long numAs = logData.filter(new Function<String, Boolean>() { 
                public Boolean call(String s) { 
                        return s.contains("a"); 
                } 
        }).count(); 

        long numBs = logData.filter(new Function<String, Boolean>() { 
                public Boolean call(String s) { 
                        return s.contains("b"); 
                } 
        }).count(); 

        System.out.println("Lines with a: " + numAs + ", lines with b: " + numBs);
        sc.stop();
  }
}