amazon-s3 - Spark s3 读取给出 NullPointerException

Question

我正在尝试处理来自 S3 文件系统的文件。我已导出 AWS_ACCESS_KEY_ID 和 AWS_SECRET_ACCESS_KEY。我也设置了配置

hadoopConf.set("fs.s3.awsAccessKeyId","<key>")  
hadoopConf.set("fs.s3.awsSecretAccessKey","<secret>")

错误:

[error] (run-main-0) java.lang.NullPointerException
java.lang.NullPointerException
    at org.apache.hadoop.fs.s3native.NativeS3FileSystem.listStatus(NativeS3FileSystem.java:479)
    at org.apache.hadoop.fs.Globber.listStatus(Globber.java:69)
    at org.apache.hadoop.fs.Globber.glob(Globber.java:217)
    at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1642)
    at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:257)
    at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:228)
    at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:304)
    at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:207)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
    at scala.Option.getOrElse(Option.scala:120)
    at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
    at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)

build.sbt 我的 build.sbt 具有以下依赖项

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.1"

libraryDependencies += "org.apache.hadoop" % "hadoop-client" % "2.4.0"

libraryDependencies += "net.java.dev.jets3t" % "jets3t" % "0.9.3"

我使用 AWS IAM 作为访问 key 。我错过了什么吗？

任何帮助将不胜感激。

Answer 1

如果没有更多细节，我将盲目射击并说您的 URI 路径无效。