amazon-web-services - Maven 使用本地 spark 库

Question

由于最近 AWS 上的 EKS 更新，我无法再在 AWS 上运行 spark 作业(必须升级 kubernetes 客户端版本)。因此，我一直在成功构建最后一个 Spark 快照版本(2.4.5-SNAPSHOT，它包含我需要的错误修复)。现在我想将它添加到我的项目中，替换旧的 2.3.3 版本。

不幸的是，我遇到了一些编译错误(见下文)。

我的 pom.xml 文件可能有问题。最终目标是从远程和本地(repo)获取 jar 文件

想法？谢谢!

附言Ubuntu 18.04 + 智能

        The relevant part of the pom.xml file are the following:


        <?xml version="1.0" encoding="UTF-8"?>
        <project xmlns="http://maven.apache.org/POM/4.0.0"
                 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
        <modelVersion>4.0.0</modelVersion>


    I add my local repo...

         <!-- My local repo where the jar file has been placed -->
            <repositories>
                <repository>
                    <id>Local</id>
                    <name>Repository Spark</name>
                    <url>/home/cristian/repository/sparkyspark/spark</url>
                </repository>
            </repositories>

        <groupId>sparkjob</groupId>
        <artifactId>sparkjob</artifactId>
        <version>1.0-SNAPSHOT</version>

        <properties>
            <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
            <maven.compiler.source>1.8</maven.compiler.source>
            <maven.compiler.target>1.8</maven.compiler.target>
            <maven.test.skip>true</maven.test.skip>
        </properties>

        <build>
            <plugins>
                <plugin>
                    <artifactId>maven-assembly-plugin</artifactId>
                    <configuration>
                        <archive>
                            <manifest>
                                <mainClass>entry.Main</mainClass>
                            </manifest>
                        </archive>
                        <descriptorRefs>
                            <descriptorRef>jar-with-dependencies</descriptorRef>
                        </descriptorRefs>
                    </configuration>

                    <executions>
                        <execution>
                            <id>make-assembly</id>
                            <!-- bind to the packaging phase -->
                            <phase>package</phase>
                            <goals>
                                <goal>single</goal>
                            </goals>
                        </execution>
                    </executions>
                </plugin>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-enforcer-plugin</artifactId>
                    <version>1.4.1</version>
                    <configuration>
                        <rules><dependencyConvergence/></rules>
                    </configuration>
                </plugin>
            </plugins>
        </build>



        ...

        <dependencies>
        .... 
        ....
    here it is, the jar file I need
         <!-- The last Spark jar file -->
            <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-core_2.11</artifactId>
                <version>2.4.5-SNAPSHOT</version>
                <exclusions>
                    <exclusion>
                        <groupId>com.fasterxml.jackson.core</groupId>
                        <artifactId>jackson-core</artifactId>
                    </exclusion>
                </exclusions>
            </dependency>
        ...
        ....
         </dependencies>

This is the error message, the path is correct...the file is there.
Ideas? :)

错误:

Could not resolve dependencies for project sparkjob:sparkjob:jar:1.0-SNAPSHOT: Failed to collect dependencies at org.apache.spark:spark-core_2.11:jar:2.4.5-SNAPSHOT: Failed to read artifact descriptor for org.apache.spark:spark-core_2.11:jar:2.4.5-SNAPSHOT: Could not transfer artifact org.apache.spark:spark-core_2.11:pom:2.4.5-SNAPSHOT from/to Local (/home/cristian/repository/sparkyspark/spark): Cannot access /home/cristian/repository/sparkyspark/spark with type default using the available connector factories.....

更新:硬连接路径似乎是一个很好的解决方法......

<dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.11</artifactId>
        <version>2.4.5-SNAPSHOT</version>
        <scope>system</scope>
        <systemPath>/home/cristian/repository/sparkyspark/spark/spark-core_2.11-2.4.5-SNAPSHOT.jar</systemPath>
        <exclusions>
            <exclusion>
                <groupId>com.fasterxml.jackson.core</groupId>
                <artifactId>jackson-core</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

Answer 1

如果您想将文件夹用作存储库，则必须使用 file:// 协议(protocol)。

所以你的存储库配置应该是。

<repositories>
    <repository>
       <id>Local</id>
       <name>Repository Spark</name>
       <url>file:///home/cristian/repository/sparkyspark/spark</url>
    </repository>
</repositories>