scala - Spark 流式 Elasticsearch 依赖项

Question

我正在尝试在 scala 中集成 Spark 和 Elasticsearch，如 Elasticsearch Guide 中所述

我在编译时遇到依赖问题:

[trace] Stack trace suppressed: run last *:update for the full output.
[error] (*:update) sbt.ResolveException: unresolved dependency: cascading#ing-local;2.5.6: not found
[error] unresolved dependency: clj-time#clj-time;0.4.1: not found
[error] unresolved dependency: compojure#compojure;1.1.3: not found
[error] unresolved dependency: hiccup#hiccup;0.3.6: not found
[error] unresolved dependency: ring#ring-devel;0.3.11: not found
[error] unresolved dependency: ring#ring-jetty-adapter;0.3.11: not found
[error] unresolved dependency: com.twitter#carbonite;1.4.0: not found
[error] unresolved dependency: cascading#cascading-hadoop;2.5.6: not found
[error] Total time: 86 s, completed 19 nov. 2014 08:42:58

我的 build.sbt 文件是这样的

name := "twitter-sparkstreaming-elasticsearch"

version := "0.0.1"

scalaVersion := "2.10.4"

// additional libraries
libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % "1.1.0",
  "org.apache.spark" %% "spark-streaming" % "1.1.0",
  "org.apache.spark" %% "spark-streaming-twitter" % "1.1.0",
  "org.elasticsearch" % "elasticsearch-hadoop" % "2.1.0"
)

帮助？谢谢。

Answer 1

Cascading 及其依赖项在 Maven central 中不可用，但在它们自己的 repo 中可用(es-hadoop 无法通过其 pom 指定)。

我使用 elasticsearch-spark_2.10 解决了这个问题

http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/master/install.html