java - 如何使用 JUNit 测试 Java-Spark？

Question

我不知道如何对 java spark 应用程序进行单元测试...

我这里有一个简单的测试，但由于 Jackson 异常而失败。我也试过将主人设置为“本地”

    private JavaSparkContext sparkCtx;

    @Before
    public void init() throws IllegalArgumentException, IOException {
        //ctxtBuilder = new ContextBuilder(tempFolder);
        SparkConf conf = new SparkConf();
        conf.setMaster("local[2]");
        conf.setAppName("junit");
        sparkCtx = new JavaSparkContext(conf);     
    }

    @Test
    public void test() {
        final List<Integer> nums = new ArrayList<Integer>();
        nums.add(3);
        nums.add(4);
        nums.add(2);
        JavaRDD<Integer> rdd = sparkCtx.parallelize(nums,1);
        assertEquals(3, rdd.count());

    }

---

异常

java.lang.VerifyError: class com.fasterxml.jackson.module.scala.ser.ScalaIteratorSerializer overrides final method withResolved.(Lcom/fasterxml/jackson/databind/BeanProperty;Lcom/fasterxml/jackson/databind/jsontype/TypeSerializer;Lcom/fasterxml/jackson/databind/JsonSerializer;)Lcom/fasterxml/jackson/databind/ser/std/AsArraySerializerBase;
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at com.fasterxml.jackson.module.scala.ser.IteratorSerializerModule$class.$init$(IteratorSerializerModule.scala:70)
    at com.fasterxml.jackson.module.scala.DefaultScalaModule.<init>(DefaultScalaModule.scala:19)
    at com.fasterxml.jackson.module.scala.DefaultScalaModule$.<init>(DefaultScalaModule.scala:35)
    at com.fasterxml.jackson.module.scala.DefaultScalaModule$.<clinit>(DefaultScalaModule.scala)
    at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:78)
    at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)
    at org.apache.spark.SparkContext.withScope(SparkContext.scala:700)
    at org.apache.spark.SparkContext.parallelize(SparkContext.scala:714)
    at org.apache.spark.api.java.JavaSparkContext.parallelize(JavaSparkContext.scala:145)
    at sal_mr.ingestion.storage.IngestionOutputFormatTest.test(IngestionOutputFormatTest.java:38)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
    at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)

---

控制台输出:

2016-03-23 09:56:58 INFO  SparkContext:59 - Running Spark version 1.5.0
2016-03-23 09:56:59 INFO  Utils:59 - Successfully started service 'sparkDriver' on port 38028.
2016-03-23 09:57:00 INFO  Utils:59 - Successfully started service 'SparkUI' on port 4040.
2016-03-23 09:57:00 INFO  Executor:59 - Starting executor ID driver on host localhost
2016-03-23 09:57:00 INFO  Utils:59 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 35071.
2016-03-23 09:57:00 INFO  BlockManagerMasterEndpoint:59 - Registering block manager localhost:35071 with 1918.1 MB RAM, BlockManagerId(driver, localhost, 35071)
2016-03-23 09:57:00 INFO  BlockManagerMaster:59 - Registered BlockManager
2016-03-23 09:57:00 INFO  SparkContext:59 - Invoking stop() from shutdown hook

Answer 1

好的，我想我遇到了 jar 不匹配的问题。

这thread带我去做。

在我的 pom 中，我有以下依赖项:

    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-databind</artifactId>
        <version>2.7.1</version>        
    </dependency>

查看 jars spark pulls 我看到:jackson-module-scala_2.10-2.2.3.jar

现在测试通过了，我移除了 jackson 依赖!