java - EMR ClassNotFoundException java 中的 Spark 提交-6ren

java - EMR ClassNotFoundException java 中的 Spark 提交

转载作者：行者123 更新时间：2023-11-30 10:04:44

我已经从 spring boot 应用程序中创建了 jar 包，我在其中创建了 spark session 来执行一些操作。

然后我将这个 jar 发送到 AWS EMR 并作为 spark-submit 运行或 java -jar无法运行的可执行文件。

这是我使用 jar -tf <jar name> 时的 jar 结构

META-INF/
META-INF/MANIFEST.MF
org/
org/springframework/
org/springframework/boot/
org/springframework/boot/loader/
org/springframework/boot/loader/data/
org/springframework/boot/loader/data/RandomAccessDataFile$DataInputStream.class
org/springframework/boot/loader/jar/
org/springframework/boot/loader/jar/CentralDirectoryEndRecord.class
org/springframework/boot/loader/data/RandomAccessDataFile$FileAccess.class
org/springframework/boot/loader/jar/CentralDirectoryParser.class
org/springframework/boot/loader/data/RandomAccessDataFile$1.class
org/springframework/boot/loader/jar/ZipInflaterInputStream.class
org/springframework/boot/loader/data/RandomAccessDataFile.class
org/springframework/boot/loader/jar/Handler.class
org/springframework/boot/loader/data/RandomAccessData.class
org/springframework/boot/loader/archive/
org/springframework/boot/loader/archive/Archive$EntryFilter.class
org/springframework/boot/loader/jar/Bytes.class
org/springframework/boot/loader/archive/Archive.class
org/springframework/boot/loader/archive/JarFileArchive.class
org/springframework/boot/loader/PropertiesLauncher$PrefixMatchingArchiveFilter.class
org/springframework/boot/loader/PropertiesLauncher$ArchiveEntryFilter.class
org/springframework/boot/loader/PropertiesLauncher$1.class
org/springframework/boot/loader/PropertiesLauncher.class
org/springframework/boot/loader/Launcher.class
org/springframework/boot/loader/MainMethodRunner.class
org/springframework/boot/loader/ExecutableArchiveLauncher.class
org/springframework/boot/loader/WarLauncher.class
org/springframework/boot/loader/JarLauncher.class
org/springframework/boot/loader/util/
org/springframework/boot/loader/util/SystemPropertyUtils.class
org/springframework/boot/loader/LaunchedURLClassLoader$UseFastConnectionExceptionsEnumeration.class
org/springframework/boot/loader/LaunchedURLClassLoader.class
org/springframework/boot/loader/jar/JarURLConnection$1.class
org/springframework/boot/loader/archive/JarFileArchive$EntryIterator.class
org/springframework/boot/loader/jar/JarURLConnection$JarEntryName.class
org/springframework/boot/loader/archive/ExplodedArchive$FileEntryIterator$EntryComparator.class
org/springframework/boot/loader/jar/JarURLConnection.class
org/springframework/boot/loader/jar/JarFile$1.class
org/springframework/boot/loader/archive/ExplodedArchive$FileEntryIterator.class
org/springframework/boot/loader/archive/ExplodedArchive$FileEntry.class
org/springframework/boot/loader/archive/ExplodedArchive$1.class
org/springframework/boot/loader/jar/JarFile$2.class
org/springframework/boot/loader/jar/JarFile$JarFileType.class
org/springframework/boot/loader/archive/ExplodedArchive.class
org/springframework/boot/loader/archive/JarFileArchive$JarFileEntry.class
org/springframework/boot/loader/jar/JarFile.class
org/springframework/boot/loader/jar/JarEntry.class
org/springframework/boot/loader/jar/FileHeader.class
org/springframework/boot/loader/jar/StringSequence.class
org/springframework/boot/loader/jar/AsciiBytes.class
org/springframework/boot/loader/jar/JarFileEntries$1.class
org/springframework/boot/loader/jar/JarFileEntries$EntryIterator.class
org/springframework/boot/loader/jar/JarFileEntries.class
org/springframework/boot/loader/jar/CentralDirectoryVisitor.class
org/springframework/boot/loader/jar/JarEntryFilter.class
org/springframework/boot/loader/jar/CentralDirectoryFileHeader.class
org/springframework/boot/loader/archive/Archive$Entry.class
BOOT-INF/
BOOT-INF/classes/
BOOT-INF/classes/com/
BOOT-INF/classes/com/ati/
BOOT-INF/classes/com/ati/media/
BOOT-INF/classes/com/ati/media/ref/
BOOT-INF/classes/com/ati/media/ref/scd/
BOOT-INF/classes/com/ati/media/ref/scd/config/
BOOT-INF/classes/com/ati/media/ref/scd/controllers/
BOOT-INF/classes/com/ati/media/ref/scd/error/
BOOT-INF/classes/com/ati/media/ref/scd/repositories/
BOOT-INF/classes/com/ati/media/ref/scd/services/
BOOT-INF/classes/com/ati/media/ref/scd/services/validator/
META-INF/maven/
META-INF/maven/com.ati.media.ref/
META-INF/maven/com.ati.media.ref/ref-svc-scd/
BOOT-INF/classes/application-local.yml
BOOT-INF/classes/bootstrap.yml
BOOT-INF/classes/com/ati/media/ref/scd/config/SwaggerConfig.class
BOOT-INF/classes/com/ati/media/ref/scd/error/ScheduleExceptionHandler.class
BOOT-INF/classes/com/ati/media/ref/scd/ScheduleApplication.class
BOOT-INF/classes/com/ati/media/ref/scd/services/ScheduleServiceImpl.class
META-INF/maven/com.ati.media.ref/ref-svc-scd/pom.xml
BOOT-INF/classes/bootstrap-docker.yml
BOOT-INF/classes/com/ati/media/ref/scd/config/ScheduleConstants.class
BOOT-INF/classes/com/ati/media/ref/scd/error/ScheduleCreationException.class
BOOT-INF/classes/com/ati/media/ref/scd/repositories/ScheduleMyBatisRepository.xml
BOOT-INF/classes/com/ati/media/ref/scd/services/ConfigurationService.class
BOOT-INF/classes/git.properties
META-INF/maven/com.ati.media.ref/ref-svc-scd/pom.properties
BOOT-INF/classes/application.yml
BOOT-INF/classes/com/ati/media/ref/scd/config/SparkConfig.class
BOOT-INF/classes/com/ati/media/ref/scd/error/ScheduleEditException.class
BOOT-INF/classes/com/ati/media/ref/scd/repositories/SparkAthenaRepository.class
BOOT-INF/classes/com/ati/media/ref/scd/services/ScheduleService.class
META-INF/build-info.properties
BOOT-INF/classes/bootstrap-local.yml
BOOT-INF/classes/com/ati/media/ref/scd/controllers/ScheduleController.class
BOOT-INF/classes/com/ati/media/ref/scd/repositories/ScheduleMyBatisRepository.class
BOOT-INF/classes/com/ati/media/ref/scd/services/AthenaQueryBuilder.class
BOOT-INF/classes/com/ati/media/ref/scd/services/validator/ScheduleValidator.class
META-INF/build.properties
BOOT-INF/lib/
BOOT-INF/lib/core-tech-starter-web-base-1.0.0.jar
BOOT-INF/lib/spring-boot-starter-web-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-boot-starter-json-2.0.3.RELEASE.jar
BOOT-INF/lib/jackson-datatype-jdk8-2.9.6.jar
BOOT-INF/lib/jackson-datatype-jsr310-2.9.6.jar
BOOT-INF/lib/jackson-module-parameter-names-2.9.6.jar
BOOT-INF/lib/spring-boot-starter-tomcat-2.0.3.RELEASE.jar
BOOT-INF/lib/tomcat-embed-core-8.5.31.jar
BOOT-INF/lib/tomcat-embed-el-8.5.31.jar
BOOT-INF/lib/tomcat-embed-websocket-8.5.31.jar
BOOT-INF/lib/hibernate-validator-6.0.10.Final.jar
BOOT-INF/lib/spring-web-5.0.7.RELEASE.jar
BOOT-INF/lib/spring-webmvc-5.0.7.RELEASE.jar
BOOT-INF/lib/spring-expression-5.0.7.RELEASE.jar
BOOT-INF/lib/spring-boot-starter-webflux-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-boot-starter-reactor-netty-2.0.3.RELEASE.jar
BOOT-INF/lib/reactor-netty-0.7.8.RELEASE.jar
BOOT-INF/lib/netty-codec-http-4.1.25.Final.jar
BOOT-INF/lib/netty-codec-4.1.25.Final.jar
BOOT-INF/lib/netty-handler-4.1.25.Final.jar
BOOT-INF/lib/netty-buffer-4.1.25.Final.jar
BOOT-INF/lib/netty-transport-4.1.25.Final.jar
BOOT-INF/lib/netty-resolver-4.1.25.Final.jar
BOOT-INF/lib/netty-handler-proxy-4.1.25.Final.jar
BOOT-INF/lib/netty-codec-socks-4.1.25.Final.jar
BOOT-INF/lib/netty-transport-native-epoll-4.1.25.Final-linux-x86_64.jar
BOOT-INF/lib/netty-common-4.1.25.Final.jar
BOOT-INF/lib/netty-transport-native-unix-common-4.1.25.Final.jar
BOOT-INF/lib/spring-webflux-5.0.7.RELEASE.jar
BOOT-INF/lib/reactor-core-3.1.8.RELEASE.jar
BOOT-INF/lib/nio-multipart-parser-1.1.0.jar
BOOT-INF/lib/nio-stream-storage-1.1.3.jar
BOOT-INF/lib/spring-boot-starter-actuator-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-boot-actuator-autoconfigure-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-boot-actuator-2.0.3.RELEASE.jar
BOOT-INF/lib/micrometer-core-1.0.5.jar
BOOT-INF/lib/HdrHistogram-2.1.10.jar
BOOT-INF/lib/LatencyUtils-2.0.3.jar
BOOT-INF/lib/spring-boot-starter-log4j2-2.0.3.RELEASE.jar
BOOT-INF/lib/log4j-slf4j-impl-2.10.0.jar
BOOT-INF/lib/log4j-core-2.10.0.jar
BOOT-INF/lib/log4j-jul-2.10.0.jar
BOOT-INF/lib/log4j-1.2-api-2.7.jar
BOOT-INF/lib/log4j-api-2.10.0.jar
BOOT-INF/lib/lombok-1.18.0.jar
BOOT-INF/lib/micrometer-registry-prometheus-1.0.3.jar
BOOT-INF/lib/simpleclient_common-0.3.0.jar
BOOT-INF/lib/simpleclient-0.3.0.jar
BOOT-INF/lib/spring-cloud-starter-zipkin-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-starter-sleuth-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-sleuth-core-2.0.0.RELEASE.jar
BOOT-INF/lib/aspectjrt-1.8.13.jar
BOOT-INF/lib/brave-5.1.0.jar
BOOT-INF/lib/brave-context-log4j2-5.1.0.jar
BOOT-INF/lib/brave-instrumentation-spring-web-5.1.0.jar
BOOT-INF/lib/brave-instrumentation-http-5.1.0.jar
BOOT-INF/lib/brave-instrumentation-spring-rabbit-5.1.0.jar
BOOT-INF/lib/brave-instrumentation-kafka-clients-5.1.0.jar
BOOT-INF/lib/brave-instrumentation-httpclient-5.1.0.jar
BOOT-INF/lib/brave-instrumentation-httpasyncclient-5.1.0.jar
BOOT-INF/lib/brave-instrumentation-spring-webmvc-5.1.0.jar
BOOT-INF/lib/brave-instrumentation-servlet-5.1.0.jar
BOOT-INF/lib/spring-cloud-sleuth-zipkin-2.0.0.RELEASE.jar
BOOT-INF/lib/zipkin-2.9.3.jar
BOOT-INF/lib/zipkin-reporter-2.7.3.jar
BOOT-INF/lib/zipkin-sender-kafka11-2.7.3.jar
BOOT-INF/lib/zipkin-sender-amqp-client-2.7.3.jar
BOOT-INF/lib/core-tech-starter-amqp-1.0.0.jar
BOOT-INF/lib/spring-cloud-stream-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-boot-starter-validation-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-messaging-5.0.7.RELEASE.jar
BOOT-INF/lib/spring-integration-core-5.0.6.RELEASE.jar
BOOT-INF/lib/spring-integration-jmx-5.0.6.RELEASE.jar
BOOT-INF/lib/spring-tuple-1.0.0.RELEASE.jar
BOOT-INF/lib/spring-integration-tuple-1.0.0.RELEASE.jar
BOOT-INF/lib/spring-boot-starter-amqp-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-rabbit-2.0.4.RELEASE.jar
BOOT-INF/lib/spring-amqp-2.0.4.RELEASE.jar
BOOT-INF/lib/amqp-client-5.1.2.jar
BOOT-INF/lib/spring-cloud-stream-binder-rabbit-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-stream-binder-rabbit-core-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-integration-amqp-5.0.6.RELEASE.jar
BOOT-INF/lib/spring-cloud-starter-config-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-starter-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-context-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-security-rsa-1.0.5.RELEASE.jar
BOOT-INF/lib/bcpkix-jdk15on-1.56.jar
BOOT-INF/lib/spring-cloud-config-client-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-boot-starter-aop-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-aop-5.0.7.RELEASE.jar
BOOT-INF/lib/aspectjweaver-1.8.13.jar
BOOT-INF/lib/core-tech-lib-core-1.0.0.jar
BOOT-INF/lib/spring-retry-1.2.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-starter-openfeign-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-openfeign-core-2.0.0.RELEASE.jar
BOOT-INF/lib/feign-core-9.5.1.jar
BOOT-INF/lib/feign-slf4j-9.5.1.jar
BOOT-INF/lib/feign-hystrix-9.5.1.jar
BOOT-INF/lib/feign-java8-9.5.1.jar
BOOT-INF/lib/spring-cloud-starter-netflix-ribbon-2.0.0.RELEASE.jar
BOOT-INF/lib/ribbon-2.2.5.jar
BOOT-INF/lib/ribbon-transport-2.2.5.jar
BOOT-INF/lib/rxnetty-contexts-0.4.9.jar
BOOT-INF/lib/rxnetty-servo-0.4.9.jar
BOOT-INF/lib/rxnetty-0.4.9.jar
BOOT-INF/lib/netty-transport-native-epoll-4.1.25.Final.jar
BOOT-INF/lib/ribbon-core-2.2.5.jar
BOOT-INF/lib/ribbon-httpclient-2.2.5.jar
BOOT-INF/lib/jersey-client-1.19.1.jar
BOOT-INF/lib/jersey-apache-client4-1.19.1.jar
BOOT-INF/lib/servo-core-0.12.21.jar
BOOT-INF/lib/netflix-commons-util-0.3.0.jar
BOOT-INF/lib/ribbon-loadbalancer-2.2.5.jar
BOOT-INF/lib/netflix-statistics-0.1.1.jar
BOOT-INF/lib/commons-text-1.3.jar
BOOT-INF/lib/commons-collections4-4.1.jar
BOOT-INF/lib/core-tech-starter-web-api-1.0.0.jar
BOOT-INF/lib/springfox-swagger2-2.9.2.jar
BOOT-INF/lib/swagger-annotations-1.5.20.jar
BOOT-INF/lib/swagger-models-1.5.20.jar
BOOT-INF/lib/springfox-spi-2.9.2.jar
BOOT-INF/lib/springfox-core-2.9.2.jar
BOOT-INF/lib/springfox-schema-2.9.2.jar
BOOT-INF/lib/springfox-swagger-common-2.9.2.jar
BOOT-INF/lib/springfox-spring-web-2.9.2.jar
BOOT-INF/lib/guava-20.0.jar
BOOT-INF/lib/classmate-1.3.4.jar
BOOT-INF/lib/spring-plugin-core-1.2.0.RELEASE.jar
BOOT-INF/lib/spring-plugin-metadata-1.2.0.RELEASE.jar
BOOT-INF/lib/mapstruct-1.2.0.Final.jar
BOOT-INF/lib/springfox-swagger-ui-2.9.2.jar
BOOT-INF/lib/spring-cloud-starter-netflix-hystrix-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-netflix-core-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-netflix-ribbon-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-netflix-archaius-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-starter-netflix-archaius-2.0.0.RELEASE.jar
BOOT-INF/lib/archaius-core-0.7.6.jar
BOOT-INF/lib/hystrix-core-1.5.12.jar
BOOT-INF/lib/rxjava-1.3.8.jar
BOOT-INF/lib/hystrix-serialization-1.5.12.jar
BOOT-INF/lib/jackson-module-afterburner-2.9.6.jar
BOOT-INF/lib/hystrix-metrics-event-stream-1.5.12.jar
BOOT-INF/lib/hystrix-javanica-1.5.12.jar
BOOT-INF/lib/asm-5.0.4.jar
BOOT-INF/lib/rxjava-reactive-streams-1.2.1.jar
BOOT-INF/lib/reactive-streams-1.0.2.jar
BOOT-INF/lib/spring-cloud-netflix-hystrix-stream-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-commons-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-security-crypto-5.0.6.RELEASE.jar
BOOT-INF/lib/spring-cloud-starter-stream-rabbit-2.0.0.RELEASE.jar
BOOT-INF/lib/postgresql-42.2.2.jar
BOOT-INF/lib/athenaJDBC-1.1.0.jar
BOOT-INF/lib/lib-client-scd-0.0.0.jar
BOOT-INF/lib/core-tech-starter-web-client-1.0.0.jar
BOOT-INF/lib/hibernate-jpa-2.1-api-1.0.2.Final.jar
BOOT-INF/lib/gson-2.8.5.jar
BOOT-INF/lib/spring-boot-starter-data-jpa-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-boot-starter-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-boot-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-boot-autoconfigure-2.0.3.RELEASE.jar
BOOT-INF/lib/javax.annotation-api-1.3.2.jar
BOOT-INF/lib/snakeyaml-1.19.jar
BOOT-INF/lib/spring-boot-starter-jdbc-2.0.3.RELEASE.jar
BOOT-INF/lib/HikariCP-2.7.9.jar
BOOT-INF/lib/spring-jdbc-5.0.7.RELEASE.jar
BOOT-INF/lib/hibernate-core-5.2.17.Final.jar
BOOT-INF/lib/jboss-logging-3.3.2.Final.jar
BOOT-INF/lib/javassist-3.22.0-GA.jar
BOOT-INF/lib/antlr-2.7.7.jar
BOOT-INF/lib/jandex-2.0.3.Final.jar
BOOT-INF/lib/dom4j-1.6.1.jar
BOOT-INF/lib/hibernate-commons-annotations-5.0.1.Final.jar
BOOT-INF/lib/javax.transaction-api-1.2.jar
BOOT-INF/lib/spring-data-jpa-2.0.8.RELEASE.jar
BOOT-INF/lib/spring-data-commons-2.0.8.RELEASE.jar
BOOT-INF/lib/spring-orm-5.0.7.RELEASE.jar
BOOT-INF/lib/spring-context-5.0.7.RELEASE.jar
BOOT-INF/lib/spring-beans-5.0.7.RELEASE.jar
BOOT-INF/lib/spring-aspects-5.0.7.RELEASE.jar
BOOT-INF/lib/mybatis-spring-boot-starter-1.3.1.jar
BOOT-INF/lib/mybatis-spring-boot-autoconfigure-1.3.1.jar
BOOT-INF/lib/mybatis-3.4.5.jar
BOOT-INF/lib/mybatis-spring-1.3.1.jar
BOOT-INF/lib/spring-boot-starter-jooq-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-tx-5.0.7.RELEASE.jar
BOOT-INF/lib/jooq-3.10.7.jar
BOOT-INF/lib/byte-buddy-1.7.11.jar
BOOT-INF/lib/objenesis-2.6.jar
BOOT-INF/lib/spring-core-5.0.7.RELEASE.jar
BOOT-INF/lib/spring-jcl-5.0.7.RELEASE.jar
BOOT-INF/lib/core-tech-starter-web-request-details-1.0.0.jar
BOOT-INF/lib/spark-core_2.11-2.3.0.jar
BOOT-INF/lib/avro-1.7.7.jar
BOOT-INF/lib/jackson-core-asl-1.9.13.jar
BOOT-INF/lib/paranamer-2.3.jar
BOOT-INF/lib/commons-compress-1.4.1.jar
BOOT-INF/lib/xz-1.0.jar
BOOT-INF/lib/avro-mapred-1.7.7-hadoop2.jar
BOOT-INF/lib/avro-ipc-1.7.7.jar
BOOT-INF/lib/avro-ipc-1.7.7-tests.jar
BOOT-INF/lib/chill_2.11-0.8.4.jar
BOOT-INF/lib/kryo-shaded-3.0.3.jar
BOOT-INF/lib/minlog-1.3.0.jar
BOOT-INF/lib/chill-java-0.8.4.jar
BOOT-INF/lib/xbean-asm5-shaded-4.4.jar
BOOT-INF/lib/hadoop-client-2.6.5.jar
BOOT-INF/lib/hadoop-hdfs-2.6.5.jar
BOOT-INF/lib/xercesImpl-2.9.1.jar
BOOT-INF/lib/xml-apis-1.4.01.jar
BOOT-INF/lib/htrace-core-3.0.4.jar
BOOT-INF/lib/hadoop-mapreduce-client-app-2.6.5.jar
BOOT-INF/lib/hadoop-mapreduce-client-common-2.6.5.jar
BOOT-INF/lib/hadoop-mapreduce-client-shuffle-2.6.5.jar
BOOT-INF/lib/hadoop-mapreduce-client-core-2.6.5.jar
BOOT-INF/lib/hadoop-mapreduce-client-jobclient-2.6.5.jar
BOOT-INF/lib/hadoop-annotations-2.6.5.jar
BOOT-INF/lib/spark-launcher_2.11-2.3.0.jar
BOOT-INF/lib/spark-kvstore_2.11-2.3.0.jar
BOOT-INF/lib/leveldbjni-all-1.8.jar
BOOT-INF/lib/jackson-core-2.9.6.jar
BOOT-INF/lib/spark-network-common_2.11-2.3.0.jar
BOOT-INF/lib/spark-network-shuffle_2.11-2.3.0.jar
BOOT-INF/lib/spark-unsafe_2.11-2.3.0.jar
BOOT-INF/lib/jets3t-0.9.4.jar
BOOT-INF/lib/httpcore-4.4.9.jar
BOOT-INF/lib/activation-1.1.1.jar
BOOT-INF/lib/bcprov-jdk15on-1.52.jar
BOOT-INF/lib/java-xmlbuilder-1.1.jar
BOOT-INF/lib/base64-2.3.8.jar
BOOT-INF/lib/curator-recipes-4.0.1.jar
BOOT-INF/lib/curator-framework-4.0.1.jar
BOOT-INF/lib/javax.servlet-api-3.1.0.jar
BOOT-INF/lib/commons-lang3-3.7.jar
BOOT-INF/lib/commons-math3-3.4.1.jar
BOOT-INF/lib/jsr305-1.3.9.jar
BOOT-INF/lib/slf4j-api-1.7.25.jar
BOOT-INF/lib/jul-to-slf4j-1.7.25.jar
BOOT-INF/lib/jcl-over-slf4j-1.7.25.jar
BOOT-INF/lib/compress-lzf-1.0.3.jar
BOOT-INF/lib/snappy-java-1.1.2.6.jar
BOOT-INF/lib/lz4-java-1.4.0.jar
BOOT-INF/lib/zstd-jni-1.3.2-2.jar
BOOT-INF/lib/RoaringBitmap-0.5.11.jar
BOOT-INF/lib/commons-net-2.2.jar
BOOT-INF/lib/scala-library-2.11.8.jar
BOOT-INF/lib/json4s-jackson_2.11-3.2.11.jar
BOOT-INF/lib/json4s-core_2.11-3.2.11.jar
BOOT-INF/lib/json4s-ast_2.11-3.2.11.jar
BOOT-INF/lib/scalap-2.11.0.jar
BOOT-INF/lib/scala-compiler-2.11.0.jar
BOOT-INF/lib/scala-xml_2.11-1.0.1.jar
BOOT-INF/lib/jersey-client-2.26.jar
BOOT-INF/lib/javax.ws.rs-api-2.1.jar
BOOT-INF/lib/javax.inject-2.5.0-b42.jar
BOOT-INF/lib/jersey-common-2.26.jar
BOOT-INF/lib/osgi-resource-locator-1.0.1.jar
BOOT-INF/lib/jersey-server-2.26.jar
BOOT-INF/lib/jersey-media-jaxb-2.26.jar
BOOT-INF/lib/validation-api-2.0.1.Final.jar
BOOT-INF/lib/jersey-container-servlet-2.26.jar
BOOT-INF/lib/jersey-container-servlet-core-2.26.jar
BOOT-INF/lib/netty-all-4.1.25.Final.jar
BOOT-INF/lib/netty-3.9.9.Final.jar
BOOT-INF/lib/stream-2.7.0.jar
BOOT-INF/lib/metrics-core-3.2.6.jar
BOOT-INF/lib/metrics-jvm-3.2.6.jar
BOOT-INF/lib/metrics-json-3.2.6.jar
BOOT-INF/lib/metrics-graphite-3.2.6.jar
BOOT-INF/lib/jackson-databind-2.9.6.jar
BOOT-INF/lib/jackson-module-scala_2.11-2.9.6.jar
BOOT-INF/lib/scala-reflect-2.11.11.jar
BOOT-INF/lib/jackson-module-paranamer-2.9.6.jar
BOOT-INF/lib/ivy-2.4.0.jar
BOOT-INF/lib/oro-2.0.8.jar
BOOT-INF/lib/pyrolite-4.13.jar
BOOT-INF/lib/py4j-0.10.6.jar
BOOT-INF/lib/spark-tags_2.11-2.3.0.jar
BOOT-INF/lib/commons-crypto-1.0.0.jar
BOOT-INF/lib/unused-1.0.0.jar
BOOT-INF/lib/simba-athena-jdbc-driver-2.0.2.jar
BOOT-INF/lib/spark-sql_2.11-2.3.0.jar
BOOT-INF/lib/univocity-parsers-2.5.9.jar
BOOT-INF/lib/spark-sketch_2.11-2.3.0.jar
BOOT-INF/lib/spark-catalyst_2.11-2.3.0.jar
BOOT-INF/lib/scala-parser-combinators_2.11-1.0.4.jar
BOOT-INF/lib/janino-3.0.8.jar
BOOT-INF/lib/commons-compiler-3.0.8.jar
BOOT-INF/lib/antlr4-runtime-4.7.jar
BOOT-INF/lib/orc-core-1.4.1-nohive.jar
BOOT-INF/lib/protobuf-java-2.5.0.jar
BOOT-INF/lib/commons-lang-2.6.jar
BOOT-INF/lib/aircompressor-0.8.jar
BOOT-INF/lib/orc-mapreduce-1.4.1-nohive.jar
BOOT-INF/lib/parquet-column-1.8.2.jar
BOOT-INF/lib/parquet-common-1.8.2.jar
BOOT-INF/lib/parquet-encoding-1.8.2.jar
BOOT-INF/lib/parquet-hadoop-1.8.2.jar
BOOT-INF/lib/parquet-format-2.3.1.jar
BOOT-INF/lib/parquet-jackson-1.8.2.jar
BOOT-INF/lib/arrow-vector-0.8.0.jar
BOOT-INF/lib/arrow-format-0.8.0.jar
BOOT-INF/lib/arrow-memory-0.8.0.jar
BOOT-INF/lib/hppc-0.7.2.jar
BOOT-INF/lib/flatbuffers-1.2.0-3f79e055.jar
BOOT-INF/lib/spark-yarn_2.11-2.3.0.jar
BOOT-INF/lib/hadoop-yarn-api-2.6.5.jar
BOOT-INF/lib/hadoop-yarn-common-2.6.5.jar
BOOT-INF/lib/jaxb-api-2.3.0.jar
BOOT-INF/lib/jetty-util-6.1.26.jar
BOOT-INF/lib/jackson-jaxrs-1.9.13.jar
BOOT-INF/lib/jackson-xc-1.9.13.jar
BOOT-INF/lib/commons-cli-1.2.jar
BOOT-INF/lib/guice-servlet-3.0.jar
BOOT-INF/lib/commons-io-2.4.jar
BOOT-INF/lib/guice-3.0.jar
BOOT-INF/lib/javax.inject-1.jar
BOOT-INF/lib/aopalliance-1.0.jar
BOOT-INF/lib/log4j-1.2.17.jar
BOOT-INF/lib/hadoop-yarn-server-web-proxy-2.6.5.jar
BOOT-INF/lib/hadoop-yarn-server-common-2.6.5.jar
BOOT-INF/lib/jetty-6.1.26.jar
BOOT-INF/lib/hadoop-yarn-client-2.6.5.jar
BOOT-INF/lib/hadoop-aws-2.7.3.jar
BOOT-INF/lib/hadoop-common-2.7.3.jar
BOOT-INF/lib/xmlenc-0.52.jar
BOOT-INF/lib/commons-collections-3.2.2.jar
BOOT-INF/lib/servlet-api-2.5.jar
BOOT-INF/lib/jsp-api-2.1.jar
BOOT-INF/lib/jersey-core-1.19.1.jar
BOOT-INF/lib/jsr311-api-1.1.1.jar
BOOT-INF/lib/jersey-json-1.9.jar
BOOT-INF/lib/jettison-1.1.jar
BOOT-INF/lib/jaxb-impl-2.2.3-1.jar
BOOT-INF/lib/jersey-server-1.19.1.jar
BOOT-INF/lib/commons-logging-1.1.3.jar
BOOT-INF/lib/commons-configuration-1.6.jar
BOOT-INF/lib/commons-digester-1.8.jar
BOOT-INF/lib/commons-beanutils-1.7.0.jar
BOOT-INF/lib/commons-beanutils-core-1.8.0.jar
BOOT-INF/lib/hadoop-auth-2.7.3.jar
BOOT-INF/lib/apacheds-kerberos-codec-2.0.0-M15.jar
BOOT-INF/lib/apacheds-i18n-2.0.0-M15.jar
BOOT-INF/lib/api-asn1-api-1.0.0-M20.jar
BOOT-INF/lib/api-util-1.0.0-M20.jar
BOOT-INF/lib/jsch-0.1.54.jar
BOOT-INF/lib/curator-client-2.7.1.jar
BOOT-INF/lib/htrace-core-3.1.0-incubating.jar
BOOT-INF/lib/zookeeper-3.4.6.jar
BOOT-INF/lib/jackson-annotations-2.9.0.jar
BOOT-INF/lib/aws-java-sdk-1.7.4.jar
BOOT-INF/lib/spark-hive_2.11-2.3.0.jar
BOOT-INF/lib/parquet-hadoop-bundle-1.6.0.jar
BOOT-INF/lib/hive-exec-1.2.1.spark2.jar
BOOT-INF/lib/javolution-5.5.1.jar
BOOT-INF/lib/apache-log4j-extras-1.2.17.jar
BOOT-INF/lib/antlr-runtime-3.4.jar
BOOT-INF/lib/stringtemplate-3.2.1.jar
BOOT-INF/lib/ST4-4.0.4.jar
BOOT-INF/lib/JavaEWAH-0.3.2.jar
BOOT-INF/lib/snappy-0.2.jar
BOOT-INF/lib/stax-api-1.0.1.jar
BOOT-INF/lib/opencsv-2.3.jar
BOOT-INF/lib/hive-metastore-1.2.1.spark2.jar
BOOT-INF/lib/bonecp-0.8.0.RELEASE.jar
BOOT-INF/lib/datanucleus-api-jdo-3.2.6.jar
BOOT-INF/lib/datanucleus-rdbms-3.2.9.jar
BOOT-INF/lib/commons-pool-1.6.jar
BOOT-INF/lib/commons-dbcp-1.4.jar
BOOT-INF/lib/jdo-api-3.0.1.jar
BOOT-INF/lib/jta-1.1.jar
BOOT-INF/lib/commons-httpclient-3.1.jar
BOOT-INF/lib/calcite-avatica-1.2.0-incubating.jar
BOOT-INF/lib/calcite-core-1.2.0-incubating.jar
BOOT-INF/lib/calcite-linq4j-1.2.0-incubating.jar
BOOT-INF/lib/eigenbase-properties-1.1.5.jar
BOOT-INF/lib/httpclient-4.5.5.jar
BOOT-INF/lib/jackson-mapper-asl-1.9.13.jar
BOOT-INF/lib/commons-codec-1.11.jar
BOOT-INF/lib/joda-time-2.9.9.jar
BOOT-INF/lib/jodd-core-3.5.2.jar
BOOT-INF/lib/datanucleus-core-3.2.10.jar
BOOT-INF/lib/libthrift-0.9.3.jar
BOOT-INF/lib/libfb303-0.9.3.jar
BOOT-INF/lib/derby-10.14.1.0.jar
BOOT-INF/lib/core-tech-starter-docker-spring-boot-1.0.0.jar

当尝试运行

spark-submit --master yarn --deploy-mode client --driver-memory 10G --executor-memory 15G --class com.ati.media.ref.scd.ScheduleApplication

我看到的错误是

java.lang.ClassNotFoundException: com.ati.media.ref.scd.ScheduleApplication
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at org.apache.spark.util.Utils$.classForName(Utils.scala:235)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:836)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

当我尝试使用时

java -jar <jar>该应用程序启动时没有任何问题。

java -cp结果与 spark-submit 未找到主 calss 的错误相同。

这里有什么问题。我在 ScheduleApplication 中有主要功能.

最佳答案

您要么需要指定 --jars <path to your jar>或者在集群级别配置 extrac jar 位置

来自AWS documentation :

A bootstrap action or custom configuration is overriding the class paths. When this happens, the class loader only picks up the JAR files that exist in the location that you specified in your configuration.

解决这个问题的方法很少:

1) 指定 --jars带有你的 jar 的路径:

spark-submit --deploy-mode client --class org.apache.spark.examples.SparkPi --master yarn spark-examples.jar 100 --jars /home/hadoop/extrajars/*

2) 将自定义 JAR(包含缺少的类)的路径添加到 Spark 类路径。

在 /etc/spark/conf/spark-defaults.conf 中，将自定义 JAR 的路径附加到错误堆栈跟踪中指定的类名。在以下示例中，/home/hadoop/extrajars/* 是自定义 JAR 路径。

例如(如何修改正在运行的集群):

sudo vim /etc/spark/conf/spark-defaults.conf

spark.driver.extraClassPath :/home/hadoop/extrajars/*
spark.executor.extraClassPath :/home/hadoop/extrajars/*

有关更多信息，或如何在新集群上执行此操作，请参阅 documentation

关于java - EMR ClassNotFoundException java 中的 Spark 提交，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/55747916/

文章推荐： javascript - 从新数组使用 $.makeArray() 功能后获取对象属性

文章推荐： swift - AdMob 是否使用 IDFA？

文章推荐： javascript - D3 force layout start需要5秒才能执行

文章推荐： swift - NSPredicate 根据 MapView 边界限制纬度经度范围

apache-spark - Spark 如何处理比 Spark 存储大得多的数据？
目前正在学习 Spark 的类(class)并了解到执行者的定义: Each executor will hold a chunk of the data to be processed. Thisc
apache-spark - Spark 中的任务是什么？ Spark Worker如何执行jar文件？
阅读了有关 http://spark.apache.org/docs/0.8.0/cluster-overview.html 的一些文档后，我有一些问题想要澄清。以 Spark 为例: JavaSp
apache-spark - Spark 调度器与 Spark 堆栈中的独立调度器
Spark核心中的调度器与以下Spark Stack(来自Learning Spark:Lightning-Fast Big Data Analysis一书)中的Standalone Schedule
apache-spark - Apache Spark : setting spark. eventLog.enabled 和 Spark.eventLog.dir 在提交或 Spark 启动时
我想在 spark-submit 或 start 处设置 spark.eventLog.enabled 和 spark.eventLog.dir -all level -- 不要求在 scala/ja
apache-spark - Spark - Spark DataFrame、Spark SQL 和/或 Databricks 表中的混合区分大小写
我有来自 SQL Server 的数据，需要在 Apache Spark (Databricks) 中进行操作。在 SQL Server 中，此表的三个键列使用区分大小写的 COLLATION 选项
apache-spark - spark.local.ip ,spark.driver.host,spark.driver.bindAddress 和 spark.driver.hostname 是什么？
所有这些有什么区别和用途？ spark.local.ip spark.driver.host spark.driver.bind地址 spark.driver.hostname 如何将机器修复为 Sp
apache-spark - 跨多个 Spark 作业重用 Spark session
我有大约 10 个 Spark 作业，每个作业都会进行一些转换并将数据加载到数据库中。必须为每个作业单独打开和关闭 Spark session ，每次初始化都会耗费时间。是否可以只创建一次 Spar
apache-spark - spark 3.0- spark 聚合函数给出了与预期不同的表达式
/Downloads/spark-3.0.1-bin-hadoop2.7/bin$ ./spark-shell 20/09/23 10:58:45 WARN Utils: Your hostname,
apache-spark - 提交 Spark 作业到 Spark 集群
我是 Spark 的完全新手，并且刚刚开始对此进行更多探索。我选择了更长的路径，不使用任何 CDH 发行版安装 hadoop，并且我从 Apache 网站安装了 Hadoop 并自己设置配置文件以了解
apache-spark - Spark 显示的内核数与使用 spark-submit 传递给它的内核数不同
TL; 博士 Spark UI 显示的内核和内存数量与我在使用 spark-submit 时要求的数量不同更多细节: 我在独立模式下运行 Spark 1.6。当我运行 spark-submit 时
apache-spark - Spark pyspark 与 spark-submit
spark-submit 上的文档说明如下: The spark-submit script in Spark’s bin directory is used to launch applicatio
apache-spark - 在同一集群中同时进行 Spark 流和 Spark 批处理作业的最佳实践
关闭。这个问题是opinion-based .它目前不接受答案。想改善这个问题吗？更新问题，以便可以通过 editing this post 用事实和引文回答问题. 6 个月前关闭。 Improve
apache-spark - Spark : Is receiver in spark streaming a bottleneck?
我想了解接收器如何在 Spark Streaming 中工作。根据我的理解，将有一个接收器任务在执行器中运行，用于收集数据并保存为 RDD。当调用 start() 时，接收器开始读取。需要澄清以下内容
apache-spark - 如何使用相同的 spark 上下文并行运行多个 spark 作业？
有没有办法在不同线程中使用相同的 spark 上下文并行运行多个 spark 作业？我尝试使用 Vertx 3，但看起来每个作业都在排队并按顺序启动。如何让它在相同的 spark 上下文中同时运行
apache-spark - 如何在不停止 Spark 流的情况下清理 Spark 历史事件日志
我们有一个 Spark 流应用程序，这是一项长期运行的任务。事件日志指向 hdfs 位置 hdfs://spark-history，当我们开始流式传输应用程序时正在其中创建 application_X
apache-spark - 使用 Spark - Spark JobServer 的基于请求的实时推荐？
我们正在尝试找到一种加载 Spark (2.x) ML 训练模型的方法，以便根据请求(通过 REST 接口(interface))我们可以查询它并获得预测，例如http://predictor.com
apache-spark - spark-sql 与 spark-shell REPL 中的 Spark SQL 性能差异
Spark newb 问题:我在 spark-sql 中进行完全相同的 Spark SQL 查询并在 spark-shell . spark-shell版本大约需要 10 秒，而 spark-sql版
apache-spark - Spark 累加器未显示在 Spark WebUI 中
我正在使用 Spark 流。根据 Spark 编程指南(参见 http://spark.apache.org/docs/latest/programming-guide.html#accumulato
scala - Spark : how to run spark file from spark shell
我正在使用 CDH 5.2。我可以使用 spark-shell 运行命令。如何运行包含spark命令的文件(file.spark)。有没有办法在不使用 sbt 的情况下在 CDH 5.2 中运行/
apache-spark - Spark-Cassandra 与 Spark-Elasticsearch
我使用 Elasticsearch 已经有一段时间了，但使用 Cassandra 的经验很少。现在，我有一个项目想要使用 Spark 来处理数据，但我需要决定是否应该使用 Cassandra 还是

行者123

个人简介

我是一名优秀的程序员,十分优秀！

作者热门文章

滴滴打车优惠券免费领取

全站热门文章

首页

博学

6Ren·AI

商城

java - EMR ClassNotFoundException java 中的 Spark 提交