java.lang.NoClassDefFoundError : org/apache/flink/streaming/connectors/rabbitmq/common/RMQConnectionConfig in flink job

Question

我正在将 flink(v1.10.0) 作业提交到我的集群，但显示此错误:

 System.out: (none)
System.err: (none)
    at org.apache.flink.client.program.OptimizerPlanEnvironment.generateException(OptimizerPlanEnvironment.java:149)
    at org.apache.flink.client.program.OptimizerPlanEnvironment.getPipeline(OptimizerPlanEnvironment.java:89)
    at org.apache.flink.client.program.PackagedProgramUtils.getPipelineFromProgram(PackagedProgramUtils.java:101)
    at org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:56)
    at org.apache.flink.runtime.webmonitor.handlers.utils.JarHandlerUtils$JarHandlerContext.toJobGraph(JarHandlerUtils.java:128)
    at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$getJobGraphAsync$6(JarRunHandler.java:138)
    at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: org/apache/flink/streaming/connectors/rabbitmq/common/RMQConnectionConfig$Builder
    at org.myorg.quickstart.StreamingJob.i(StreamingJob.java:123)
    at org.myorg.quickstart.StreamingJob.main(StreamingJob.java:55)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:321)
    at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:205)
    at org.apache.flink.client.program.OptimizerPlanEnvironment.getPipeline(OptimizerPlanEnvironment.java:79)
    ... 8 more
Caused by: java.lang.ClassNotFoundException: org.apache.flink.streaming.connectors.rabbitmq.common.RMQConnectionConfig$Builder
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
    at org.apache.flink.util.ChildFirstClassLoader.loadClass(ChildFirstClassLoader.java:60)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
    ... 17 more

似乎在运行时找不到RMQConnectionConfig，如何在系统范围内添加该jar并使我的任务jar又小又薄？这是我的 gradle.build 依赖项:

dependencies {

    // --------------------------------------------------------------
    // Compile-time dependencies that should NOT be part of the
    // shadow jar and are provided in the lib folder of Flink
    // --------------------------------------------------------------
    compile "org.apache.flink:flink-java:${flinkVersion}"
    compile "org.apache.flink:flink-streaming-java_${scalaBinaryVersion}:${flinkVersion}"

    // --------------------------------------------------------------
    // Dependencies that should be part of the shadow jar, e.g.
    // connectors. These must be in the flinkShadowJar configuration!
    // --------------------------------------------------------------
    //flinkShadowJar "org.apache.flink:flink-connector-kafka-0.11_${scalaBinaryVersion}:${flinkVersion}"

    compile "log4j:log4j:${log4jVersion}"
    compile "org.slf4j:slf4j-log4j12:${slf4jVersion}"

    compile "com.rabbitmq:amqp-client:4.2.0"
    compile "com.alibaba:fastjson:1.2.57"
    compile "org.apache.flink:flink-connector-rabbitmq_${scalaBinaryVersion}:1.10.0"

}

Answer 1

如果你想让jar包含到dist.jar文件中，你应该像这样编写依赖项:

 flinkShadowJar "com.rabbitmq:amqp-client:4.2.0"
 flinkShadowJar "com.alibaba:fastjson:1.2.57"
 flinkShadowJar "org.apache.flink:flink-connector-rabbitmq_${scalaBinaryVersion}:1.10.0"

它会将依赖项打包到 dist jar 文件并在运行时使用。