apache-spark - Spark 程序的 S3 减速异常

Question

这个问题在这里已经有了答案:





S3 SlowDown error in Spark on EMR

(1 个回答)


3年前关闭。




我在 EMR 集群中运行了简单的 spark 程序，试图将 60 GB 的 CSV 文件转换为 Parquet 。当我提交工作时，我得到以下异常。

391, ip-172-31-36-116.us-west-2.compute.internal, executor 96): org.apache.spark.SparkException: Task failed while writing rows.
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:285)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:197)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:196)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)Caused by: com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.model.AmazonS3Exception: Slow Down (Service: Amazon S3; Status Code: 503; Error Code: 503 Slow Down; Request ID: D13A3F4D7DD970FA; S3 Extended Request ID: gj3cPalkkOwtaf9XN/P+sb3jX0CNHu/QF9WTabkgP2ISuXcXdbvYO1Irg0O54OCvKlLz8WoR8E4=), S3 Extended Request ID: gj3cPalkkOwtaf9XN/P+sb3jX0CNHu/QF9WTabkgP2ISuXcXdbvYO1Irg0O54OCvKlLz8WoR8E4=
at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1639)

Answer 1

503 Slow Down是当您每秒执行过多请求时来自 AWS 服务的通用响应。

可能的解决方案:

首先将您的文件复制到 HDFS。

您有一个 60 Gb 的文件还是许多文件总和为 60 Gb？如果您有很多小文件，请尝试先将它们组合起来。

如果可以，请尝试减少 Parquet 输出中的分区数。df.repartition(100)

尝试使用更少的 Spark worker 。val spark = SparkSession.builder.appName("Simple Application").master("local[1]").getOrCreate()