scala - 使用 Spark 分析推特数据

Question

其他任何人都可以帮助我了解如何根据我写的“键”分析推特数据。我找到了这段代码，但这给我一个错误。

import java.io.File
import com.google.gson.Gson
import org.apache.spark.streaming.twitter.TwitterUtils
import org.apache.spark.streaming.{Seconds, StreamingContext}
import org.apache.spark.{SparkConf, SparkContext}

/**
 * Collect at least the specified number of tweets into json text files.
 */
object Collect {
  private var numTweetsCollected = 0L
  private var partNum = 0
  private var gson = new Gson()

  def main(args: Array[String]) {
    // Process program arguments and set properties
    if (args.length < 3) {
      System.err.println("Usage: " + this.getClass.getSimpleName +
        "<outputDirectory> <numTweetsToCollect> <intervalInSeconds> <partitionsEachInterval>")
      System.exit(1)
    }
    val Array(outputDirectory, Utils.IntParam(numTweetsToCollect),  Utils.IntParam(intervalSecs), Utils.IntParam(partitionsEachInterval)) =
      Utils.parseCommandLineWithTwitterCredentials(args)
    val outputDir = new File(outputDirectory.toString)
    if (outputDir.exists()) {
      System.err.println("ERROR - %s already exists: delete or specify another directory".format(
        outputDirectory))
      System.exit(1)
    }
    outputDir.mkdirs()

    println("Initializing Streaming Spark Context...")
    val conf = new SparkConf().setAppName(this.getClass.getSimpleName)
    val sc = new SparkContext(conf)
    val ssc = new StreamingContext(sc, Seconds(intervalSecs))

    val tweetStream = TwitterUtils.createStream(ssc, Utils.getAuth)
      .map(gson.toJson(_))

    tweetStream.foreachRDD((rdd, time) => {
      val count = rdd.count()
      if (count > 0) {
        val outputRDD = rdd.repartition(partitionsEachInterval)
        outputRDD.saveAsTextFile(outputDirectory + "/tweets_" + time.milliseconds.toString)
        numTweetsCollected += count
        if (numTweetsCollected > numTweetsToCollect) {
          System.exit(0)
        }
      }
    })

    ssc.start()
    ssc.awaitTermination()
  }
}

错误是

object gson is not a member of package com.google

如果你知道关于它的任何链接或解决这个问题，你可以与我分享，因为我想用 spark 分析 twitter 数据。谢谢。:)

Answer 1

正如 Peter 所指出的，您缺少 gson 依赖项。因此，您需要将以下依赖项添加到您的 build.sbt 中:

libraryDependencies += "com.google.code.gson" % "gson" % "2.4"

您还可以执行以下操作以在一个序列中定义所有依赖项:

libraryDependencies ++= Seq(
    "com.google.code.gson" % "gson" % "2.4",
    "org.apache.spark" %% "spark-core" % "1.2.0",
    "org.apache.spark" %% "spark-streaming" % "1.2.0",
    "org.apache.spark" %% "spark-streaming-twitter" % "1.2.0"
)

奖励:如果缺少其他依赖项，您可以尝试在 http://mvnrepository.com/ 上搜索您的依赖项如果您需要查找给定类的关联 jar/依赖项，您还可以使用 findjar website