python - 从 S3 将 CSV 加载到 PySpark

Question

使用 PySpark，我可以使用以下代码加载本地 CSV:

cd ./spark-1.6.0-bin-hadoop2.4/

./bin/pyspark --packages com.databricks:spark-csv_2.11:1.2.0 --driver-memory 4G

from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)

sqlContext.read.format('com.databricks.spark.csv').options(header='true').load('/my/local/folder/input_data.csv').write.save("/my/local/folder/input_data", format="parquet")

但是我无法让它与存储在 S3 上的(非公开的)CSV 一起工作，因为它超时了:

sqlContext.read.format('com.databricks.spark.csv').options(header='true').load('s3n://<AWS_ACCESS_KEY_ID>:<AWS_SECRET_ACCESS_KEY>@my.bucket/folder/input_data.csv').write.save("/my/local/folder/input_data", format="parquet")


py4j.protocol.Py4JJavaError: An error occurred while calling o25.load.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.net.SocketException: Operation timed out

这可能吗？如果是的话，有什么想法我做错了吗？提前致谢。

Answer 1

您是否尝试过从纯文本加载 rdd？ csv 文件很容易分割

sc.textFile('/my/local/folder/input_data.csv').map(lambda row:row.split(','))