java - 如何有效生成充满随机值的Spark数据集？

Question

我需要一个 Dataset<Double>填充随机值或生成值的任意大小。

似乎可以通过实现 RDD 来完成，并在 compute 中生成值方法。有更好的解决方案吗？

Answer 1

你可以尝试随机数据生成

• sql 函数生成用随机值填充的列
• 两种支持的分布:统一分布和正态分布
• 可用于随机算法、原型(prototype)设计和性能测试

import org.apache.spark.sql.functions.{rand, randn}

val dfr = sqlContext.range(0,10) // range can be what you want
val randomValues = dfr.select("id")
                      .withColumn("uniform", rand(10L))
                      .withColumn("normal", randn(10L))

randomValues.show(truncate = false)

输出

+---+-------------------+--------------------+
|id |uniform            |normal              |
+---+-------------------+--------------------+
|0  |0.41371264720975787|-0.5877482396744728 |
|1  |0.7311719281896606 |1.5746327759749246  |
|2  |0.1982919638208397 |-0.256535324205377  |
|3  |0.12714181165849525|-0.31703264334668824|
|4  |0.7604318153406678 |0.4977629425313746  |
|5  |0.12030715258495939|-0.506853671746243  |
|6  |0.12131363910425985|1.4250903895905769  |
|7  |0.44292918521277047|-0.1413699193557902 |
|8  |0.8898784253886249 |0.9657665088756656  |
|9  |0.03650707717266999|-0.5021009082343131 |
+---+-------------------+--------------------+