gpt4 book ai didi

python - Spark toDebugString 在 python 中不好用

转载 作者:太空狗 更新时间:2023-10-29 21:22:12 26 4
gpt4 key购买 nike

这是我在 scala 中使用 toDebugString 时得到的结果:

scala> val a  = sc.parallelize(Array(1,2,3)).distinct
a: org.apache.spark.rdd.RDD[Int] = MappedRDD[3] at distinct at <console>:12

scala> a.toDebugString
res0: String =
(4) MappedRDD[3] at distinct at <console>:12
| ShuffledRDD[2] at distinct at <console>:12
+-(4) MappedRDD[1] at distinct at <console>:12
| ParallelCollectionRDD[0] at parallelize at <console>:12

这在 python 中是等价的:

>>> a = sc.parallelize([1,2,3]).distinct()
>>> a.toDebugString()
'(4) PythonRDD[6] at RDD at PythonRDD.scala:43\n | MappedRDD[5] at values at NativeMethodAccessorImpl.java:-2\n | ShuffledRDD[4] at partitionBy at NativeMethodAccessorImpl.java:-2\n +-(4) PairwiseRDD[3] at RDD at PythonRDD.scala:261\n | PythonRDD[2] at RDD at PythonRDD.scala:43\n | ParallelCollectionRDD[0] at parallelize at PythonRDD.scala:315'

如您所见,python 中的输出不如 scala 中的输出好。有没有什么技巧可以让这个函数有更好的输出?

我正在使用 Spark 1.1.0。

最佳答案

尝试添加一个 print 语句,以便实际打印调试字符串,而不是显示其 __repr__:

>>> a = sc.parallelize([1,2,3]).distinct()
>>> print a.toDebugString()
(8) PythonRDD[27] at RDD at PythonRDD.scala:44 [Serialized 1x Replicated]
| MappedRDD[26] at values at NativeMethodAccessorImpl.java:-2 [Serialized 1x Replicated]
| ShuffledRDD[25] at partitionBy at NativeMethodAccessorImpl.java:-2 [Serialized 1x Replicated]
+-(8) PairwiseRDD[24] at distinct at <stdin>:1 [Serialized 1x Replicated]
| PythonRDD[23] at distinct at <stdin>:1 [Serialized 1x Replicated]
| ParallelCollectionRDD[21] at parallelize at PythonRDD.scala:358 [Serialized 1x Replicated]

关于python - Spark toDebugString 在 python 中不好用,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/26342132/

26 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com