gpt4 book ai didi

java - 在 Spark 上运行 AWS S3 客户端时出现 NoSuchMethodError,而 javap 则显示其他错误

转载 作者:行者123 更新时间:2023-11-30 03:49:29 26 4
gpt4 key购买 nike

我在 Apache Spark 上运行的一段代码存在运行时问题。我依靠 AWS SDK 将文件上传到 S3 - 并且出现 NoSuchMethodError 错误。值得注意的是,我使用的是捆绑了 Spark 依赖项的 uber jar。运行代码时出错:

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.http.impl.conn.DefaultClientConnectionOperator.<init>(Lorg/apache/http/conn/scheme/SchemeRegistry;Lorg/apache/http/conn/DnsResolver;)V
at org.apache.http.impl.conn.PoolingClientConnectionManager.createConnectionOperator(PoolingClientConnectionManager.java:140)
at org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:114)
at org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:99)
at com.amazonaws.http.ConnectionManagerFactory.createPoolingClientConnManager(ConnectionManagerFactory.java:29)
at com.amazonaws.http.HttpClientFactory.createHttpClient(HttpClientFactory.java:97)
at com.amazonaws.http.AmazonHttpClient.<init>(AmazonHttpClient.java:165)
at com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:119)
at com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:103)
at com.amazonaws.services.s3.AmazonS3Client.<init>(AmazonS3Client.java:357)
at com.amazonaws.services.s3.AmazonS3Client.<init>(AmazonS3Client.java:339)

但是,当我检查 jar 中的方法签名时,我看到它清楚地列出:

vagrant@mesos:~/installs/spark-1.0.1-bin-hadoop2$ javap -classpath /tmp/rickshaw-spark-0.0.1-SNAPSHOT.jar org.apache.http.impl.conn.DefaultClientConnectionOperator
Compiled from "DefaultClientConnectionOperator.java"
public class org.apache.http.impl.conn.DefaultClientConnectionOperator implements org.apache.http.conn.ClientConnectionOperator {
protected final org.apache.http.conn.scheme.SchemeRegistry schemeRegistry;
protected final org.apache.http.conn.DnsResolver dnsResolver;
public org.apache.http.impl.conn.DefaultClientConnectionOperator(org.apache.http.conn.scheme.SchemeRegistry);
public org.apache.http.impl.conn.DefaultClientConnectionOperator(org.apache.http.conn.scheme.SchemeRegistry, org.apache.http.conn.DnsResolver); <-- it exists!
public org.apache.http.conn.OperatedClientConnection createConnection();
public void openConnection(org.apache.http.conn.OperatedClientConnection, org.apache.http.HttpHost, java.net.InetAddress, org.apache.http.protocol.HttpContext, org.apache.http.params.HttpParams) throws java.io.IOException;
public void updateSecureConnection(org.apache.http.conn.OperatedClientConnection, org.apache.http.HttpHost, org.apache.http.protocol.HttpContext, org.apache.http.params.HttpParams) throws java.io.IOException;
protected void prepareSocket(java.net.Socket, org.apache.http.protocol.HttpContext, org.apache.http.params.HttpParams) throws java.io.IOException;
protected java.net.InetAddress[] resolveHostname(java.lang.String) throws java.net.UnknownHostException;

}

我检查了 Spark 发行版中的一些其他 jar - 它们似乎没有这个特定的方法签名。所以我想知道 Spark 运行时检测到的是什么导致了这个问题。该 jar 是在 Maven 项目上构建的,我在其中排列了依赖项以确保正确的 aws java sdk 依赖项也被选取。

最佳答案

Spark 1.0.x 发行版已包含不兼容版本的 DefaultClientConnectionOperator,并且没有简单的方法来替换它。

我发现的唯一解决方法是包含 PoolingClientConnectionManager 的自定义实现,以避免使用缺少的构造函数。

替换:

return new DefaultClientConnectionOperator(schreg, this.dnsResolver);

对于:

return new DefaultClientConnectionOperator(schreg);

您需要确定,您的类(class)将被包括在内:

case PathList("org", "apache", "http", "impl", xs @ _*) => MergeStrategy.first

自定义 PoolingClientConnectionManager:https://gist.github.com/felixgborrego/568f3460d82d9c12e23c

关于java - 在 Spark 上运行 AWS S3 客户端时出现 NoSuchMethodError,而 javap 则显示其他错误,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/24788949/

26 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com