gpt4 book ai didi

scala - Spark 错误 - 退出状态 : 143. 诊断:根据请求终止容器

转载 作者:行者123 更新时间:2023-12-05 06:10:32 26 4
gpt4 key购买 nike

我收到以下错误:

Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 653 in stage 7.0 failed 4 times, most recent failure: Lost task 653.3 in stage 7.0 (TID 27294, ip-10-0-57-16.ec2.internal, executor 34): ExecutorLostFailure (executor 34 exited caused by one of the running tasks) Reason: Container marked as failed: container_1602898457220_0001_01_000370 on host: ip-10-0-57-16.ec2.internal. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

我的数据集是80GB我所做的操作是创建一些方形的交互功能,因此可能会加倍列数。

我正在使用 20 个 m4.16xlarge(64CPU,256GB,https://aws.amazon.com/ec2/instance-types/)实例spark.yarn.executor.memoryOverhead = '16384'

我可以做些什么来解决这个问题吗?以及为什么即使我的数据集比我的实例数小得多也会出现 OOM 错误。

最佳答案

我增加了以下两个参数并避免了错误:

spark.default.parallelism = '128'
spark.executor.cores = '16'

关于scala - Spark 错误 - 退出状态 : 143. 诊断:根据请求终止容器,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/64399505/

26 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com