作者热门文章
- html - 出于某种原因,IE8 对我的 Sass 文件中继承的 html5 CSS 不友好?
- JMeter 在响应断言中使用 span 标签的问题
- html - 在 :hover and :active? 上具有不同效果的 CSS 动画
- html - 相对于居中的 html 内容固定的 CSS 重复背景?
我在 aws spark 上运行以下句子
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
import sqlContext.implicits._
case class Wiki(project: String, title: String, count: Int, byte_size: String)
val data = sc.textFile("s3n://+++/").map(_.split(" ")).filter(_.size ==4 ).map(p => Wiki(p(0), p(1), p(2).trim.toInt, p(3)))
val df = data.toDF()
df.printSchema()
val en_agg_df = df.filter("project = 'en'").select("title","count").groupBy("title").sum().collect()
WARN ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkYarnAM@172.31.14.190:42514] has failed, address is now gated for [5000] ms. Reason: [Disassociated]
15/10/15 17:38:36 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster has disassociated: 172.31.14.190:42514
15/10/15 17:38:36 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster has disassociated: 172.31.14.190:42514
15/10/15 17:38:36 WARN ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkExecutor@ip-172-31-14-190.ap-northeast-1.compute.internal:43340] has failed, address is now gated for [5000] ms. Reason: [Disassociated]
15/10/15 17:38:36 ERROR YarnScheduler: Lost executor 1 on ip-172-31-14-190.ap-northeast-1.compute.internal: remote Rpc client disassociated
15/10/15 17:38:36 INFO TaskSetManager: Re-queueing tasks for 1 from TaskSet 0.0
15/10/15 17:38:36 WARN TaskSetManager: Lost task 4736.0 in stage 0.0 (TID 4736, ip-172-31-14-190.ap-northeast-1.compute.internal): ExecutorLostFailure (executor 1 lost)
15/10/15 17:38:36 INFO DAGScheduler: Executor lost: 1 (epoch 0)
15/10/15 17:38:36 INFO BlockManagerMasterEndpoint: Trying to remove executor 1 from BlockManagerMaster.
15/10/15 17:38:36 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(1, ip-172-31-14-190.ap-northeast-1.compute.internal, 58890)
15/10/15 17:38:36 INFO BlockManagerMaster: Removed 1 successfully in removeExecutor
15/10/15 17:38:36 ERROR YarnScheduler: Lost executor 2 on ip-172-31-14-190.ap-northeast-1.compute.internal: remote Rpc client disassociated
15/10/15 17:38:36 WARN ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkExecutor@ip-172-31-14-190.ap-northeast-1.compute.internal:60961] has failed, address is now gated for [5000] ms. Reason: [Disassociated]
15/10/15 17:38:36 INFO TaskSetManager: Re-queueing tasks for 2 from TaskSet 0.0
15/10/15 17:38:36 WARN TaskSetManager: Lost task 4735.0 in stage 0.0 (TID 4735, ip-172-31-14-190.ap-northeast-1.compute.internal): ExecutorLostFailure (executor 2 lost)
15/10/15 17:38:36 INFO DAGScheduler: Executor lost: 2 (epoch 0)
15/10/15 17:38:36 INFO BlockManagerMasterEndpoint: Trying to remove executor 2 from BlockManagerMaster.
15/10/15 17:38:36 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(2, ip-172-31-14-190.ap-northeast-1.compute.internal, 58811)
15/10/15 17:38:36 INFO BlockManagerMaster: Removed 2 successfully in removeExecutor
最佳答案
答案似乎已经在评论中提供了:
It seems to be a out-of-memory on executor, because it went well if I add more machine to the cluster
关于apache-spark - WARN ReliableDeliverySupervisor : Association with remote system has failed, 地址现在被门控 [5000] 毫秒。原因:[已解除关联],我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/33155563/
我在 aws spark 上运行以下句子 val sqlContext = new org.apache.spark.sql.SQLContext(sc) import sqlContext.impl
我是一名优秀的程序员,十分优秀!