java - 使用构建器创建 spark session 时出现 NoSuchFieldException

Question

我正在尝试使用 :

创建一个 spark session

sparkSession = SparkSession.builder().appName(appName).master("local")
                .config("hive.metastore.uris", thriftURL).enableHiveSupport().getOrCreate();

但它给出了一个 NoSuchField 异常，如下所示:

2020-10-27 20:51:26.963  WARN 11206 --- [  restartedMain] org.apache.hadoop.util.NativeCodeLoader  : Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2020-10-27 20:51:27.053  INFO 11206 --- [  restartedMain] org.apache.spark.SparkContext            : Submitted application: HdfsPoc
2020-10-27 20:51:27.328 ERROR 11206 --- [  restartedMain] org.apache.spark.SparkContext            : Error initializing SparkContext.

java.lang.RuntimeException: java.lang.NoSuchFieldException: DEFAULT_TINY_CACHE_SIZE
    at org.apache.spark.network.util.NettyUtils.getPrivateStaticField(NettyUtils.java:131) ~[spark-network-common_2.11-2.2.1.jar:2.2.1]
    at org.apache.spark.network.util.NettyUtils.createPooledByteBufAllocator(NettyUtils.java:118) ~[spark-network-common_2.11-2.2.1.jar:2.2.1]
    at org.apache.spark.network.server.TransportServer.init(TransportServer.java:95) ~[spark-network-common_2.11-2.2.1.jar:2.2.1]
    at org.apache.spark.network.server.TransportServer.<init>(TransportServer.java:74) ~[spark-network-common_2.11-2.2.1.jar:2.2.1]
    at org.apache.spark.network.TransportContext.createServer(TransportContext.java:114) ~[spark-network-common_2.11-2.2.1.jar:2.2.1]
    at org.apache.spark.rpc.netty.NettyRpcEnv.startServer(NettyRpcEnv.scala:118) ~[spark-core_2.11-2.2.1.jar:2.2.1]
    at org.apache.spark.rpc.netty.NettyRpcEnvFactory$$anonfun$4.apply(NettyRpcEnv.scala:457) ~[spark-core_2.11-2.2.1.jar:2.2.1]
    at org.apache.spark.rpc.netty.NettyRpcEnvFactory$$anonfun$4.apply(NettyRpcEnv.scala:456) ~[spark-core_2.11-2.2.1.jar:2.2.1]
    at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:2231) ~[spark-core_2.11-2.2.1.jar:2.2.1]
    at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160) ~[scala-library-2.11.8.jar:na]
    at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:2223) ~[spark-core_2.11-2.2.1.jar:2.2.1]
    at org.apache.spark.rpc.netty.NettyRpcEnvFactory.create(NettyRpcEnv.scala:461) ~[spark-core_2.11-2.2.1.jar:2.2.1]
    at org.apache.spark.rpc.RpcEnv$.create(RpcEnv.scala:56) ~[spark-core_2.11-2.2.1.jar:2.2.1]
    at org.apache.spark.SparkEnv$.create(SparkEnv.scala:246) ~[spark-core_2.11-2.2.1.jar:2.2.1]
    at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:175) ~[spark-core_2.11-2.2.1.jar:2.2.1]
    at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:257) ~[spark-core_2.11-2.2.1.jar:2.2.1]
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:432) ~[spark-core_2.11-2.2.1.jar:2.2.1]
    at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2516) [spark-core_2.11-2.2.1.jar:2.2.1]
    at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:918) [spark-sql_2.11-2.2.1.jar:2.2.1]
    at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:910) [spark-sql_2.11-2.2.1.jar:2.2.1]
    at scala.Option.getOrElse(Option.scala:121) [scala-library-2.11.8.jar:na]
    at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:910) [spark-sql_2.11-2.2.1.jar:2.2.1]
    at com.csg.ipro.util.SparkCommonUtility.createSparkSessionWithHive(SparkCommonUtility.java:54) [classes/:na]
    at com.csg.ipro.util.SparkCommonUtility.getSparkSessionWithHive(SparkCommonUtility.java:28) [classes/:na]
    at com.csg.ipro.model.FileWatcher.createSession(FileWatcher.java:84) [classes/:na]
    at com.csg.ipro.service.WatcherService.startMonitoring(WatcherService.java:43) [classes/:na]
    at com.csg.ipro.IproStreamApplication.main(IproStreamApplication.java:21) [classes/:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_231]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_231]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_231]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_231]
    at org.springframework.boot.devtools.restart.RestartLauncher.run(RestartLauncher.java:49) [spring-boot-devtools-2.3.4.RELEASE.jar:2.3.4.RELEASE]
Caused by: java.lang.NoSuchFieldException: DEFAULT_TINY_CACHE_SIZE
    at java.lang.Class.getDeclaredField(Class.java:2070) ~[na:1.8.0_231]
    at org.apache.spark.network.util.NettyUtils.getPrivateStaticField(NettyUtils.java:127) ~[spark-network-common_2.11-2.2.1.jar:2.2.1]
    ... 31 common frames omitted

在pom.xml文件中添加的spark和hadoop依赖如下:

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>2.2.1</version>
            <exclusions>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-log4j12</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.11</artifactId>
            <version>2.2.1</version>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-hive_2.11</artifactId>
            <version>2.2.1</version>
        </dependency>

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>2.10.0</version>
            <exclusions>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-api</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-log4j12</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs-client</artifactId>
            <version>2.10.0</version>
            <scope>provided</scope>
        </dependency>

无论我在尝试寻找解决方案时读到什么，依赖版本都不匹配。但是我尝试了很多排列，但没有一个对我有用。我该如何解决这个错误？

Answer 1

这是因为添加了错误版本的netty。下面为我​​解决了这个问题:

<dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.11</artifactId>
            <version>2.3.0</version>
            <exclusions>
            <exclusion>
                <groupId>io.netty</groupId>
                <artifactId>netty-all</artifactId>
            </exclusion>
            </exclusions>
        </dependency>

<dependency>
            <groupId>io.netty</groupId>
            <artifactId>netty-all</artifactId>
            <version>4.1.17.Final</version>
        </dependency>