scala - 如何在不排序的情况下获取spark中的top-k频繁单词？

Question

在spark中，我们可以很方便地使用map-reduce来统计单词出现的时间，并使用sort来获取top-k的频繁单词，

// Sort locally inside node, keep only top-k results,
// no network communication

val partialTopK = wordCount.mapPartitions(it => {
    val a = it.toArray
    a.sortBy(-_._2).take(10).iterator
}, true)


// Collect local top-k results, faster than the naive solution

val collectedTopK = partialTopK.collect
collectedTopK.size


// Compute global top-k at master,
// no communication, everything done on the master node

val topK = collectedTopK.sortBy(-_._2).take(10)

但我想知道是否有更好的解决方案完全避免排序？

Answer 1

我想你想要takeOrdered

Returns the first k (smallest) elements from this RDD as defined by the specified implicit Ordering[T] and maintains the ordering.

或top

Returns the top k (largest) elements from this RDD as defined by the specified implicit Ordering[T].

还有其他几个问题/答案似乎也至少部分重复