apache-spark - 如何将处理程序附加到 sparkUI

Question

我正在尝试附加处理程序以使用此代码激发 UI:

 private def getSparkUI(sparkContext: SparkContext): SparkUI = {
       sparkContext.ui.getOrElse {
           throw new SparkException("Parent SparkUI to attach this tab to not found!")
       }

import org.apache.spark.ui.JettyUtils.createServletHandler
val ui = getSparkUI(sparkContext)
val graphHttpServlet = new GraphHttpServlet("/graph/", sessionUtils)
val handler = createServletHandler("/graph/", graphHttpServlet, "")
ui.attachHandler(handler)

此代码因错误而无法编译:

Error:(20, 19) Symbol 'type org.eclipse.jetty.servlet.ServletContextHandler' is missing from the classpath.
This symbol is required by 'method org.apache.spark.ui.JettyUtils.createServletHandler'.
Make sure that type ServletContextHandler is in your classpath and check for conflicting dependencies with `-Ylog-classpath`.
A full rebuild may help if 'JettyUtils.class' was compiled against an incompatible version of org.eclipse.jetty.servlet.
val handler = createServletHandler("/graph/", graphHttpServlet, "")

我知道 spark 将 jetty 依赖项隐藏到一个新包中:org.spark_project.jetty.servlet.ServletContextHandler

我还反编译了 jar 并检查 jar 是否具有阴影依赖项。

但是为什么编译器看不到呢？我如何附加处理程序？

Answer 1

此外，遇到类似的问题并在几次实验后弄明白了。为 org.eclipse.jetty 库添加阴影规则解决了这个问题。

这解决了以下异常:

• java.lang.NoSuchMethodError: 'void org.apache.spark.ui.WebUI.attachHandler(org.eclipse.jetty.servlet.ServletContextHandler)'

• java.lang.ClassCastException:类 org.sparkproject.jetty.servlet.ServletContextHandler 无法转换为类 org.eclipse.jetty.servlet.ServletContextHandler(org.sparkproject.jetty.servlet.ServletContextHandler 和org.eclipse.jetty.servlet.ServletContextHandler 位于加载程序“app”的未命名模块中)

sbt 规则

assemblyShadeRules in assembly := Seq(
    ShadeRule
      .rename("org.eclipse.jetty.**" -> "org.sparkproject.jetty.@1")
      .inAll
  )

Maven 规则

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <configuration>
        <shadedArtifactAttached>false</shadedArtifactAttached>
        <artifactSet>
        <includes>
            <include>org.eclipse.jetty:jetty-io</include>
            <include>org.eclipse.jetty:jetty-http</include>
            <include>org.eclipse.jetty:jetty-proxy</include>
            <include>org.eclipse.jetty:jetty-client</include>
            <include>org.eclipse.jetty:jetty-continuation</include>
            <include>org.eclipse.jetty:jetty-servlet</include>
            <include>org.eclipse.jetty:jetty-servlets</include>
            <include>org.eclipse.jetty:jetty-plus</include>
            <include>org.eclipse.jetty:jetty-security</include>
            <include>org.eclipse.jetty:jetty-util</include>
            <include>org.eclipse.jetty:jetty-server</include>
        </includes>
        </artifactSet>
        <relocations>
        <relocation>
            <pattern>org.eclipse.jetty</pattern>
            <shadedPattern>org.sparkproject.jetty</shadedPattern>
            <includes>
            <include>org.eclipse.jetty.**</include>
            </includes>
        </relocation>
        </relocations>
    </configuration>
    <executions>
        <execution>
        <phase>package</phase>
        <goals>
            <goal>shade</goal>
        </goals>
        </execution>
    </executions>
    </plugin>
</plugins>