apache-spark - 将 Python 字典转换为 Spark DataFrame

Question

我有一个 Python 字典:

dic = {
       (u'aaa',u'bbb',u'ccc'):((0.3, 1.2, 1.3, 1.5), 1.4, 1),
       (u'kkk',u'ggg',u'ccc',u'sss'):((0.6, 1.2, 1.7, 1.5), 1.4, 2)
       }

我想将此字典转换为包含列的 Spark DataFrame:

['key', 'val_1', 'val_2', 'val_3', 'val_4', 'val_5', 'val_6']

示例行 (1):

键 | val_1 |val_2 | val_3 | val_4 | val_5| val_6|

u'aaa',u'bbb',u'ccc' | 0.3 |1.2 |1.3 |1.5 |1.4 |1 |

提前致谢

Answer 1

提取项目，将键转换为 list 并将所有内容组合到一个 tuple 中:

df = sc.parallelize([
    (list(k), ) + 
    v[0] + 
    v[1:] 
    for k, v in  dic.items()
]).toDF(['key', 'val_1', 'val_2', 'val_3', 'val_4', 'val_5', 'val_6'])

df.show()

## +--------------------+-----+-----+-----+-----+-----+-----+
## |                 key|val_1|val_2|val_3|val_4|val_5|val_6|
## +--------------------+-----+-----+-----+-----+-----+-----+
## |     [aaa, bbb, ccc]|  0.3|  1.2|  1.3|  1.5|  1.4|    1|
## |[kkk, ggg, ccc, sss]|  0.6|  1.2|  1.7|  1.5|  1.4|    2|
## +--------------------+-----+-----+-----+-----+-----+-----+