scala - 使用 SBT 离线解决 org.apache.hadoop 依赖性的错误

Question

我正在尝试卡住 spark 项目的依赖项以便能够离线工作(sbt 无法再下载依赖项)。这是我遵循的过程:

1. 创建 sbt 项目并使用 internet 连接编译
2. 停止互联网连接
3. 验证项目是否继续编译
4. 复制 SBT 项目并删除 TARGET 文件夹
5. 告诉 Build.sbt 文件从/.ivy2/cache 文件夹中解析依赖

这是 build.sbt:

name := "Test"

version := "1.0"

scalaVersion := "2.10.4"

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.3.0"

resolvers += Resolver.file("Frozen IVY2 Cache Dependences", file("/home/luis/.ivy2/cache")) (Resolver.ivyStylePatterns) ivys "/home/luis/.ivy2/cache/[organisation]/[module]/ivy-[revision].xml"  artifacts  "/home/luis/.ivy2/cache/[organisation]/[module]/[type]s/[module]-[revision].[type]"

事实上，到达此 Build.sbt 的过程与此处描述的完全相同(未回答):

Troubles with sbt compiling offline using org.apache.hadoop/* dependencies

我包含了适当的 ivy 样式模式以指向正确的 ivy-[revision].xml 文件。

当我编译时，sbt 正在为每个依赖项找到到 .ivy2/cache“frozen”存储库的正确路径，但是我收到与解析文件“ivy-[revision].xml”相关的警告和错误。 original"对于这四个依赖:

[warn]  Note: Unresolved dependencies path:
[warn]          org.apache.hadoop:hadoop-mapreduce-client-app:2.2.0
[warn]            +- org.apache.hadoop:hadoop-client:2.2.0
[warn]            +- org.apache.spark:spark-core_2.10:1.3.0 (/home/luis/Test/build.sbt#L7-8)
[warn]            +- Test:Test_2.10:1.0
[warn]          org.apache.hadoop:hadoop-yarn-api:2.2.0
[warn]            +- org.apache.hadoop:hadoop-client:2.2.0
[warn]            +- org.apache.spark:spark-core_2.10:1.3.0 (/home/luis/Test/build.sbt#L7-8)
[warn]            +- Test:Test_2.10:1.0
[warn]          org.apache.hadoop:hadoop-mapreduce-client-core:2.2.0
[warn]            +- org.apache.hadoop:hadoop-client:2.2.0
[warn]            +- org.apache.spark:spark-core_2.10:1.3.0 (/home/luis/Test/build.sbt#L7-8)
[warn]            +- Test:Test_2.10:1.0
[warn]          org.apache.hadoop:hadoop-mapreduce-client-jobclient:2.2.0
[warn]            +- org.apache.hadoop:hadoop-client:2.2.0
[warn]            +- org.apache.spark:spark-core_2.10:1.3.0 (/home/luis/Test/build.sbt#L7-8)
[warn]            +- Test:Test_2.10:1.0

让我们专注于其中一个依赖项，因为它们的警告和错误都是相同的。假设 org.apache.hadoop:hadoop-mapreduce-client-app:2.2.0

解析文件“ivy-[revision].xml.original”的警告示例如下:

[warn] xml parsing: ivy-2.2.0.xml.original:18:69: schema_reference.4: Failed to read schema document 'http://maven.apache.org/xsd/maven-4.0.0.xsd', because 1) could not find the document; 2) the document could not be read; 3) the root element of the document is not <xsd:schema>.
[warn] xml parsing: ivy-2.2.0.xml.original:19:11: schema_reference.4: Failed to read schema document 'http://maven.apache.org/xsd/maven-4.0.0.xsd', because 1) could not find the document; 2) the document could not be read; 3) the root element of the document is not <xsd:schema>.
[warn] xml parsing: ivy-2.2.0.xml.original:20:17: schema_reference.4: Failed to read schema document 'http://maven.apache.org/xsd/maven-4.0.0.xsd', because 1) could not find the document; 2) the document could not be read; 3) the root element of the document is not <xsd:schema>.
.......
.......

[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  ::          UNRESOLVED DEPENDENCIES         ::
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  :: org.apache.hadoop#hadoop-mapreduce-client-app;2.2.0: java.text.ParseException: [xml parsing: ivy-2.2.0.xml.original:18:69: cvc-elt.1: Cannot find the declaration of element 'project'. in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[warn] , unknown tag project in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[warn] , unknown tag parent in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[warn] , unknown tag artifactId in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[warn] , unknown tag groupId in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[warn] , unknown tag version in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[warn] , unknown tag modelVersion in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[warn] , unknown tag groupId in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[warn] , unknown tag artifactId in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[warn] , unknown tag version in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[warn] , unknown tag name in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[warn] , unknown tag properties in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[warn] , unknown tag applink.base in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[warn] , unknown tag mr.basedir in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original

导致错误:

[error] (*:update) sbt.ResolveException: unresolved dependency: org.apache.hadoop#hadoop-mapreduce-client-app;2.2.0: java.text.ParseException: [xml parsing: ivy-2.2.0.xml.original:18:69: cvc-elt.1: Cannot find the declaration of element 'project'. in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[error] , unknown tag project in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[error] , unknown tag parent in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[error] , unknown tag artifactId in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[error] , unknown tag groupId in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[error] , unknown tag version in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[error] , unknown tag modelVersion in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[error] , unknown tag groupId in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[error] , unknown tag artifactId in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[error] , unknown tag version in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[error] , unknown tag name in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[error] , unknown tag properties in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[error] , unknown tag applink.base in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[error] , unknown tag mr.basedir in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[error] ]

澄清一下，ivy-2.2.0.xml.original 文件的内容如下所示:

<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
                      http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <parent>
    <artifactId>hadoop-yarn</artifactId>
    <groupId>org.apache.hadoop</groupId>
    <version>2.2.0</version>
  </parent>
  <modelVersion>4.0.0</modelVersion>
  <groupId>org.apache.hadoop</groupId>
  <artifactId>hadoop-yarn-api</artifactId>
  <version>2.2.0</version>
  <name>hadoop-yarn-api</name>

  <properties>
    <!-- Needed for generating FindBugs warnings using parent pom -->
    <yarn.basedir>${project.parent.basedir}</yarn.basedir>
  </properties>

  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-maven-plugins</artifactId>
        <executions>
          <execution>
            <id>compile-protoc</id>
            <phase>generate-sources</phase>
            <goals>
              <goal>protoc</goal>
            </goals>
            <configuration>
              <protocVersion>${protobuf.version}</protocVersion>
              <protocCommand>${protoc.path}</protocCommand>
              <imports>
                <param>${basedir}/../../../hadoop-common-project/hadoop-common/src/main/proto</param>
                <param>${basedir}/src/main/proto</param>
                <param>${basedir}/src/main/proto/server</param>
              </imports>
              <source>
                <directory>${basedir}/src/main/proto</directory>
                <includes>
                  <include>yarn_protos.proto</include>
                  <include>yarn_service_protos.proto</include>
                  <include>applicationmaster_protocol.proto</include>
                  <include>applicationclient_protocol.proto</include>
                  <include>containermanagement_protocol.proto</include>
                  <include>server/yarn_server_resourcemanager_service_protos.proto</include>
                  <include>server/resourcemanager_administration_protocol.proto</include>
                </includes>
              </source>
              <output>${project.build.directory}/generated-sources/java</output>
            </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>

</project>

在所有这些介绍之后......这些是我的问题:

• 文件“ivy-[revision].xml.original”的工作是什么，谁在尝试解析它。
• 为什么不能识别 xml 标签。

我将不胜感激!

SBT 版本:0.13.8

谢谢。

Answer 1

我问了您在帖子中提到的未回答的问题，我很高兴地宣布它已在几天前得到回答，并且建议的解决方案对我有用。

尝试更新到 sbt 0.13.9-RC3(按照 http://www.scala-sbt.org/release/tutorial/Manual-Installation.html 的说明进行操作并在 https://dl.bintray.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/0.13.9-RC3/ 获取 jar)。

最好的问候，

/马丁