scala - Spark zipWithIndex 在并行实现中是否安全？

Question

如果我有一个文件，并且我每行做了一个 RDD zipWithIndex，

([row1, id1001, name, address], 0)
([row2, id1001, name, address], 1)
...
([row100000, id1001, name, address], 100000)

如果我重新加载文件，我能否获得相同的索引顺序？由于它并行运行，其他行可能会以不同方式分区？

Answer 1

RDD s 可以排序，所以也有顺序。此命令用于创建索引 .zipWithIndex() .

每次获得相同的顺序取决于之前的调用在您的程序中执行的操作。文档提到 .groupBy()可以破坏顺序或生成不同的顺序。可能还有其他调用也可以执行此操作。

我想你可以随时调用.sortBy()在调用 .zipWithIndex() 之前如果您需要保证特定的订购。

这在 .zipWithIndex() scala API docs 中有解释。

public RDD<scala.Tuple2<T,Object>> zipWithIndex() Zips this RDD with its element indices. The ordering is first based on the partition index and then the ordering of items within each partition. So the first item in the first partition gets index 0, and the last item in the last partition receives the largest index. This is similar to Scala's zipWithIndex but it uses Long instead of Int as the index type. This method needs to trigger a spark job when this RDD contains more than one partitions.

Note that some RDDs, such as those returned by groupBy(), do not guarantee order of elements in a partition. The index assigned to each element is therefore not guaranteed, and may even change if the RDD is reevaluated. If a fixed ordering is required to guarantee the same index assignments, you should sort the RDD with sortByKey() or save it to a file.