python - 重命名写入的 CSV 文件 Spark 抛出错误 "Path must be absolute"- Azure Data Lake

Question

我尝试了 Rename written CSV file Spark 中描述的解决方案但我收到以下错误“java.lang.IllegalArgumentException:路径必须是绝对的”。我怎么能修好呢？它可以是 Scala 或 Python 代码。谢谢 :)

import org.apache.hadoop.fs._
val fs = FileSystem.get(sc.hadoopConfiguration)

var table_name = dbutils.widgets.get("table_name")

val filePath = "mnt/datalake/" + table_name + "/"

print("file path: " + filePath)

val fileName = fs.globStatus(new Path(filePath+"part*"))(0).getPath.getName
print("file name: " + fileName)

fs.rename(new Path(filePath+fileName), new Path(filePath+"file.csv"))

输出:

file path: mnt/datalake/MyTable/
file name: part-00000-tid-9118XXX-c000.csv

错误

java.lang.IllegalArgumentException: Path must be absolute: mnt/datalake/MyTable/part-00000-tid-9118XXXXc000.csv

Answer 1

试试这个:

import org.apache.hadoop.fs._
import org.apache.hadoop.fs.{FileSystem, Path}
val fs = FileSystem.get(sc.hadoopConfiguration)
val filePath = "dbfs:/FileStore/tables/part_00000-6a99e/"
val fileName = fs.globStatus(new Path(filePath))(0).getPath.getName
fs.rename(new Path(filePath+fileName), new Path(filePath+"file.csv"))