scala - 如何读取压缩的 Spark eventLog？

Question

当我尝试读取用 lz4 压缩的 Spark 2.4.4 eventLog 时，我得到一个空的 DataFrame:

cd /opt/spark-2.4.4-bin-hadoop2.7
bin/spark-shell --master=local --conf spark.eventLog.enabled=true --conf spark.eventLog.compress=true --conf spark.io.compression.codec=lz4 --driver-memory 4G --driver-library-path=/opt/hadoop-2.7.1/lib/native/

// Trying to read an event log from a previous session
spark.read.option("compression", "lz4").json(s"file:///tmp/spark-events/local-1589202668377.lz4")

// res0: org.apache.spark.sql.DataFrame = []                                       

然而，当我读取未压缩的事件日志时它工作正常:

bin/spark-shell --master=local --conf spark.eventLog.enabled=true --conf spark.eventLog.compress=false
spark.read.json(s"file:///tmp/spark-events/${sc.applicationId}.inprogress").printSchema

//root
// |-- App ID: string (nullable = true)
// |-- App Name: string (nullable = true)
// |-- Block Manager ID: struct (nullable = true)
// |    |-- Executor ID: string (nullable = true)

我还尝试读取一个用 snappy 压缩的 eventLog，结果相同。

Answer 1

尝试做

spark.read.json("dbfs:/tmp/compress/part-00000.lz4")
spark.conf.set("spark.io.compression.codec","org.apache.spark.io.LZ4CompressionCodec")

如果它不起作用，您的 lz4 很可能与 org.apache.hadoop.io.compress.Lz4Codec 不兼容下面是相同的开放问题链接 lz4 incompatibility between OS and Hadoop