java - 星火 2.0.1 java.lang.NegativeArraySizeException

Question

我开始玩 Spark 2.0.1。新数据集 API 非常干净，但我在执行非常简单的操作时遇到了问题。

也许我遗漏了什么，希望有人能提供帮助。

这些说明

SparkConf conf = new SparkConf().setAppName("myapp").setMaster("local[*]");
SparkSession spark = SparkSession
        .builder()
        .config(conf)
        .getOrCreate();

Dataset<Info> infos = spark.read().json("data.json").as(Encoders.bean(Info.class));

System.out.println(infos.rdd().count());

产生一个

 java.lang.NegativeArraySizeException

以及 JVM (1.8) 检测到的 fatal error 。

使用数据集 API 处理数据(即选择、计算信息对象)效果很好。

如何在 Dataset 和 RDD 之间切换？

Answer 1

一般来说这个错误是when an application tries to create an array with negative size.请参见下面的示例。

它的一般 java 错误。在你的情况下，我怀疑这是由

 Dataset<Info> infos = spark.read().json("data.json").as(Encoders.bean(Info.class));

System.out.println(infos.rdd().count());

您可以通过打印完整的堆栈跟踪来查看此代码在哪种情况下进行了消极初始化。

~/
❯ cat StackTest.java
import java.util.*;
import java.io.*;

public class StackTest {
    public static void main(String args[])throws IOException {
        int c[]=new int[-2];
        Scanner in=new Scanner(new InputStreamReader(System.in));
        int b=in.nextInt();
        int a[]=new int[b];
    }
}

~/
❯ javac StackTest.java && java StackTest
Exception in thread "main" java.lang.NegativeArraySizeException: -2
        at StackTest.main(StackTest.java:6)

---

Note: One of the use case is using Kryo serialization along withapache spark... when it can happen/fix is like below...

Very large object graphs

Reference limits

Kryo stores references in a map that is based on an int array.Since Java array indices are limited to Integer.MAX_VALUE,serializing large (> 1 billion) objects may result in ajava.lang.NegativeArraySizeException.

A workaround for this issue is disabling Kryo's reference tracking asindicated below:

  Kryo kryo = new Kryo();
  kryo.setReferences(false);

或者 spark-default.conf 或 sparkConf 中的 spark.kryo.referenceTrackingEnabled=false 属性> 对象，如果你想以编程方式设置它..

Spark docs says that

spark.kryo.referenceTracking 默认值 true

Whether to track references to the same object when serializing datawith Kryo, which is necessary if your object graphs have loops anduseful for efficiency if they contain multiple copies of the sameobject. Can be disabled to improve performance if you know this is notthe case.