apache-spark - 如何使指针为四个字节而不是八个字节

Question

我正在阅读 spark 文档，我可以设置指针大小。
我必须使用 JVM 标志:-XX:+UseCompressedOops ，但我不清楚如何将它插入 spark-env.sh文件。我可以通过代码设置这个标志吗？

Answer 1

Can I set this flag by code?

不。因为 jvm -xx 选项是启动时指向 jvm 的指针。你不能通过代码设置

这是您可以引用使用的示例 shell 脚本。

    #!/usr/bin/env bash
# Set Spark environment variables for your site in this file. Some useful
# variables to set are:
# - MESOS_HOME, to point to your Mesos installation
# - SCALA_HOME, to point to your Scala installation
# - SPARK_CLASSPATH, to add elements to Spark's classpath
# - SPARK_JAVA_OPTS, to add JVM options
# - SPARK_MEM, to change the amount of memory used per node (this should
# be in the same format as the JVM's -Xmx option, e.g. 300m or 1g).
# - SPARK_LIBRARY_PATH, to add extra search paths for native libraries.
export SPARK_MEM="5000m"
BASEDIR=/root
export JAVA_HOME=/usr/lib/jvm/java-6-sun
export SCALA_HOME=$BASEDIR/scala-2.8.1.final
export MESOS_HOME=$BASEDIR/mesos
JARS=
JARS+=$BASEDIR/spark/conf/
JARS+=:/nfs/spark/alignerscala_2.8.1-1.0.jar
JARS+=:$SCALA_HOME/lib/scala-library.jar:$SCALA_HOME/lib/scala-compiler.jar
export SPARK_CLASSPATH=$JARS
#export SPARK_JAVA_OPTS="-Dspark.default.parallelism=40 -Dspark.cache.class=spark.SerializingCache -Dspark.boundedMemoryCache.memoryFraction=0.50 -XX:+DoEscapeAnalysis -Dspark.task.cpus=1 -XX:NewRatio=7 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC"
#export SPARK_JAVA_OPTS="-Dspark.default.parallelism=40 -Dspark.cache.class=spark.SerializingCache -Dspark.boundedMemoryCache.memoryFraction=0.50 -XX:+DoEscapeAnalysis -Dspark.task.cpus=1 -XX:+UseCompressedOops"
#export SPARK_JAVA_OPTS="-Dspark.default.parallelism=40 -XX:+DoEscapeAnalysis -Dspark.task.cpus=2 -XX:+UseCompressedOops"
#export SPARK_JAVA_OPTS="-Dspark.default.parallelism=40 -XX:+DoEscapeAnalysis -Dspark.task.cpus=1 -XX:+UseCompressedOops -Dspark.serialization=spark.KryoSerialization -Dspark.cache.class=spark.SerializingCache -Dspark.boundedMemoryCache.memoryFraction=0.8 -Dspark.locality.wait=10000 -Dspark.kryo.registrator=SparseVectorOriginalRegistrator"
export SPARK_JAVA_OPTS="-Dspark.default.parallelism=4 -XX:+DoEscapeAnalysis -Dspark.task.cpus=1 -XX:+UseCompressedOops -Dspark.boundedMemoryCache.memoryFraction=0.8 -Dspark.locality.wait=10000"
#export SPARK_JAVA_OPTS="-Dspark.default.parallelism=40 -XX:+DoEscapeAnalysis -Dspark.task.cpus=2 -XX:+UseCompressedOops -Dspark.serialization=spark.KryoSerialization -Dspark.locality.wait=10000"