gpt4 book ai didi

scala - Spark : Create temporary table by executing sql query on temporary tables

转载 作者:行者123 更新时间:2023-12-01 03:06:06 29 4
gpt4 key购买 nike

我正在使用 Spark,我想知道:如何通过对表 A 和 B 执行 sql 查询来创建名为 C 的临时表?

sqlContext
.read.json(file_name_A)
.createOrReplaceTempView("A")

sqlContext
.read.json(file_name_B)
.createOrReplaceTempView("B")

val tableQuery = "(SELECT A.id, B.name FROM A INNER JOIN B ON A.id = B.fk_id) C"

sqlContext.read
.format(SQLUtils.FORMAT_JDBC)
.options(SQLUtils.CONFIG())
.option("dbtable", tableQuery)
.load()

最佳答案

您需要将结果保存为临时表

tableQuery .createOrReplaceTempView("dbtable")
外部表上的永久存储,您可以使用 JDBC
val prop = new java.util.Properties
prop.setProperty("driver", "com.mysql.jdbc.Driver")
prop.setProperty("user", "vaquar")
prop.setProperty("password", "khan")

//jdbc mysql url - destination database is named "temp"
val url = "jdbc:mysql://localhost:3306/temp"

//destination database table
val dbtable = "sample_data_table"

//write data from spark dataframe to database
df.write.mode("append").jdbc(url, dbtable, prop)
https://docs.databricks.com/spark/latest/data-sources/sql-databases.html
http://spark.apache.org/docs/latest/sql-programming-guide.html#saving-to-persistent-tables

关于scala - Spark : Create temporary table by executing sql query on temporary tables,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/50949806/

29 4 0