java Spark 1.6 csv 文件

Question

我正在使用spark 1.6读取csv文件，进行java编码

URL resource = Main.class.getResource("GlobalLandTemperaturesByCountry.csv");
    File filePath = Paths.get(resource.toURI()).toFile();

    JavaSparkContext jsc = new JavaSparkContext("local","Java Spark example");
    SQLContext sqlContext = new SQLContext(jsc);

    DataFrame dataFrame = sqlContext.read()
            .format("csv")
            .option("header", "true")
            .load(filePath.getAbsolutePath());
    dataFrame.show();

但是....线程“main”中出现异常 java.lang.ClassNotFoundException:找不到数据源:csv。请在 http://spark-packages.org 查找软件包

我做错了什么？对于我的版本没有 csv 解析器？路径正确请帮忙

Answer 1

将 format("csv") 更改为 .format("com.databricks.spark.csv") 并添加依赖项

<!-- https://mvnrepository.com/artifact/com.databricks/spark-csv -->
    <dependency>
        <groupId>com.databricks</groupId>
        <artifactId>spark-csv_2.11</artifactId>
        <version>1.5.0</version>
    </dependency>

结果代码:

 URL resource = Main.class.getResource("GlobalLandTemperaturesByCountry.csv");
    File filePath = Paths.get(resource.toURI()).toFile();

    JavaSparkContext jsc = new JavaSparkContext("local","Java Spark example");
    SQLContext sqlContext = new SQLContext(jsc);

    DataFrame dataFrame = sqlContext.read()
            .format("com.databricks.spark.csv")
            .option("inferSchema", "true")
            .option("header", "true")
            .load(filePath.getAbsolutePath());
    dataFrame.show();

成功了!