java - 从 Java 中的多模块项目生成数据流模板

Question

我对 Java 没有太多经验，尤其是多模块项目，因此我无法从多模块项目创建数据流模板。

要从数据流模板生成模板，您必须使用如下内容:

mvn compile exec:java \
     -Dexec.mainClass=com.example.myclass \
     -Dexec.args="--runner=DataflowRunner \
                  --project=YOUR_PROJECT_ID \
                  --stagingLocation=gs://YOUR_BUCKET_NAME/staging \
                  --templateLocation=gs://YOUR_BUCKET_NAME/templates/YOUR_TEMPLATE_NAME"

这在一个简单的 Java 项目中对我来说效果很好，但目前我需要在具有以下简化结构的项目中使用以下内容:

C:.
|   pom.xml
|
+---configuration
|   |   dependency-reduced-pom.xml
|   |   pom.xml
|   |
|   +---src
|   |   \---main
|   |       \---java
|   |           \---com
|   |               \---xxx
|   |                   \---gcp
|   |                       \---dataflow
|   |                           \---yyy
|   |                               +---package
|   |                               |   |   java files
|   |
+---pipeline
|   |   dependency-reduced-pom.xml
|   |   pom.xml
|   |
|   +---src
|   |   \---main
|   |       \---java
|   |           \---com
|   |               \---xxx
|   |                   \---gcp
|   |                       \---dataflow
|   |                           \---yyy
|   |                               \---package
|   |                                       MAINJAVACLASS.java
|   |
\---transform
|   |   dependency-reduced-pom.xml
|   |   pom.xml
|   |
|   +---src
|   |   \---main
|   |       +---java
|   |       |   +---com
|   |       |   |   \---xxx
|   |       |   |       \---gcp
|   |       |   |           \---dataflow
|   |       |   |               \---yyy
|   |       |   |                   +---package
|   |       |   |                   |       java files

<小时/>

我已经执行了mvn package，没有任何错误，输出如下:

[INFO] Reactor Build Order:
[INFO]
[INFO] pipeline-framework                                                 [pom]
[INFO] configuration                                                      [jar]
[INFO] transform                                                          [jar]
[INFO] pipeline                                                           [jar]

<...>

[INFO] Reactor Summary for pipeline-framework 0.1:
[INFO]
[INFO] pipeline-framework ................................. SUCCESS [ 19.076 s]
[INFO] configuration ...................................... SUCCESS [ 25.070 s]
[INFO] transform .......................................... SUCCESS [ 21.625 s]
[INFO] pipeline ........................................... SUCCESS [ 19.365 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS

但是当我尝试执行时:

mvn compile exec:java -Dexec.mainClass=com.xxx.gcp.dataflow.yyy.pipeline.MAINJAVACLASS -Dexec.args=...

我有以下错误:

如果我从根目录执行它:

[INFO] Reactor Summary for pipeline-framework 0.1:
[INFO]
[INFO] pipeline-framework ................................. FAILURE [  5.287 s]
[INFO] configuration ...................................... SKIPPED
[INFO] transform .......................................... SKIPPED
[INFO] pipeline ........................................... SKIPPED

<...>

Caused by: java.lang.ClassNotFoundException: com.xxx.gcp.dataflow.yyy.pipeline.MAINJAVACLASS

我也尝试过:

mvn compile exec:java -pl pipeline <...>

如果我在管道目录中执行它:

Could not resolve dependencies for project com.xxx.gcp.dataflow:pipeline:jar:0.1: The following artifacts could not be resolved: com.xxx.gcp.dataflow:transform:jar:0.1, com.xxx.gcp.dataflow:configuration:jar:0.1: Failure to find com.xxx.gcp.dataflow:transform:jar:0.1 in https://repo.maven.apache.org/maven2

<小时/>

我应该执行哪个命令来构建模板？

<小时/>

主 pom.xml 文件

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>com.xxx.gcp.dataflow</groupId>
  <artifactId>pipeline-framework</artifactId>
  <version>0.1</version>
  <packaging>pom</packaging>

  <modules>
    <module>configuration</module>
    <module>transform</module>
    <module>pipeline</module>
  </modules>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <maven.compiler.source>1.8</maven.compiler.source>
    <maven.compiler.target>1.8</maven.compiler.target>

    <beam.version>2.16.0</beam.version>

    <maven-compiler-plugin.version>3.7.0</maven-compiler-plugin.version>
    <maven-exec-plugin.version>1.6.0</maven-exec-plugin.version>
    <maven-jar-plugin.version>3.1.2</maven-jar-plugin.version>
    <slf4j.version>1.7.25</slf4j.version>

    <autovalue.annotations.version>1.6</autovalue.annotations.version>
    <autovalue.version>1.6.2</autovalue.version>
  </properties>

  <repositories>
    <repository>
      <id>apache.snapshots</id>
      <name>Apache Development Snapshot Repository</name>
      <url>https://repository.apache.org/content/repositories/snapshots/</url>
      <releases>
        <enabled>false</enabled>
      </releases>
      <snapshots>
        <enabled>true</enabled>
      </snapshots>
    </repository>
  </repositories>

  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>${project.groupId}</groupId>
        <artifactId>configuration</artifactId>
        <version>${project.version}</version>
      </dependency>
      <dependency>
        <groupId>${project.groupId}</groupId>
        <artifactId>transform</artifactId>
        <version>${project.version}</version>
      </dependency>
      <dependency>
        <groupId>${project.groupId}</groupId>
        <artifactId>pipeline</artifactId>
        <version>${project.version}</version>
      </dependency>
    </dependencies>
  </dependencyManagement>

  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>${maven-compiler-plugin.version}</version>
        <configuration>
          <source>1.8</source>
          <target>1.8</target>
        </configuration>
      </plugin>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>2.22.1</version>
        <configuration>
          <useSystemClassLoader>false</useSystemClassLoader>
        </configuration>
      </plugin>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-jar-plugin</artifactId>
        <version>${maven-jar-plugin.version}</version>
        <configuration>
          <archive>
            <manifest>
              <mainClass>com.xxx.gcp.dataflow.yyy.pipeline.TerraformPipeline</mainClass>
            </manifest>
          </archive>
        </configuration>
      </plugin>

      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>3.0.0</version>
        <executions>
          <execution>
            <id>bundle-and-repackage</id>
            <phase>package</phase>
            <goals>
              <goal>shade</goal>
            </goals>
            <configuration>

              <artifactSet>
                <includes>
                  <include>*:*</include>
                </includes>
              </artifactSet>

              <transformers>
                <transformer
                        implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
              </transformers>

            </configuration>
          </execution>
        </executions>
      </plugin>

    </plugins>

    <pluginManagement>
      <plugins>
        <plugin>
          <groupId>org.codehaus.mojo</groupId>
          <artifactId>exec-maven-plugin</artifactId>
          <version>${maven-exec-plugin.version}</version>
          <configuration>
            <cleanupDaemonThreads>false</cleanupDaemonThreads>
          </configuration>
        </plugin>


      </plugins>


    </pluginManagement>
  </build>

  <dependencies>
    <...>
  </dependencies>
</project>

Answer 1

我相信这是多模块 Maven 项目的常见问题，并不是 Dataflow 特有的。也许其他线程可以提供帮助:Maven exec:java goal on a multi-module project

其中提到了您遇到的未找到 MAINJAVACLASS 的问题。另一半我不太确定，我认为 jar 丢失的原因是因为 package 生命周期阶段尚未在其 .jar 的模块上运行> 你需要。据我所知， exec 插件不会在构建生命周期的任何特定阶段运行，因此根据您的信息，我猜测它仅在 compile 阶段之后运行，这不会不会生成任何 jar(发生在 package 中)。

有关构建生命周期的信息:http://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html信息