java - 强制 Storm 使用 fat jar 依赖而不是类路径定义的依赖

Question

我如何告诉 Storm 使用 fat jar 中包含的依赖项而不是 Storm 类路径中的依赖项？

这里是一些背景/细节:

• joda-time 是有问题的依赖项，2.0 包含在 storm classpath 中，2.7 包含在 fat jar 中。
• 使用 mvn compile exec:java -Dstorm.topology=ClassName 运行拓扑似乎使用 2.7 就好了。
• 使用 storm jar target/filename-jar-with-dependencies.jar ClassName 提交拓扑似乎使用 2.0 而不是 2.7。
• filename-jar-with-dependencies.jar 是使用 mvn package 创建的:
  • joda-time 2.7 在依赖部分下的 pom.xml 中
  • 当我执行 jar tvf target/filename-jar-with-dependencies.jar | 时，找到了 2.7 的正确 joda 时间戳 | grep 乔达

我什至注意到这一点的原因是我在通过 storm 命令提交拓扑时看到以下警告/错误:

WARN  com.amazonaws.services.s3.internal.S3MetadataResponseHandler - Unable to parse last modified date: Mon, 25 May 2015 13:23:29 GMT
java.lang.IllegalStateException: Joda-time 2.2 or later version is required, but found version: 2.0
    at com.amazonaws.util.DateUtils.handleException(DateUtils.java:156) ~[filename-jar-with-dependencies.jar:na]
    at com.amazonaws.util.DateUtils.parseRFC822Date(DateUtils.java:204) ~[filename-jar-with-dependencies.jar:na]
    at com.amazonaws.services.s3.internal.ServiceUtils.parseRfc822Date(ServiceUtils.java:78) ~[filename-jar-with-dependencies.jar:na]
    at com.amazonaws.services.s3.internal.AbstractS3ResponseHandler.populateObjectMetadata(AbstractS3ResponseHandler.java:115) ~[filename-jar-with-dependencies.jar:na]
    at com.amazonaws.services.s3.internal.S3ObjectResponseHandler.handle(S3ObjectResponseHandler.java:52) [filename-jar-with-dependencies.jar:na]
    at com.amazonaws.services.s3.internal.S3ObjectResponseHandler.handle(S3ObjectResponseHandler.java:30) [filename-jar-with-dependencies.jar:na]
    at com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:1050) [filename-jar-with-dependencies.jar:na]
    at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:724) [filename-jar-with-dependencies.jar:na]
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:467) [filename-jar-with-dependencies.jar:na]
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:302) [filename-jar-with-dependencies.jar:na]
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3672) [filename-jar-with-dependencies.jar:na]
    at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1160) [filename-jar-with-dependencies.jar:na]
Caused by: java.lang.IllegalArgumentException: Invalid format: "Mon, 25 May 2015 13:23:29 GMT" is malformed at "GMT"
    at org.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:747) ~[joda-time-2.0.jar:2.0]
    at com.amazonaws.util.DateUtils.parseRFC822Date(DateUtils.java:202) ~[filename-jar-with-dependencies.jar:na]
    ... 15 common frames omitted

Answer 1

我在阅读 this post 后发现的一种解决方法是用较新的版本替换 Storm 的 /lib 目录中的库(在我的例子中是 joda-time)，因为该目录中的所有 .jar 文件都显示在 默认情况下 Storm 类路径。

如果一个库的两个版本出于某种原因需要共存，我希望有更好的答案。