java - Apache Storm - java.lang.NoClassDefFoundError : com/google/gson/Gson

Question

我正在使用 Apache Storm 0.9.4。我设置了一个 5 节点集群，运行良好。(实际上有5个supervisor作为docker容器工作在5个不同的物理节点上。)

我的环境在这里

<小时/>

$cat /etc/redhat-release
CentOS release 6.6 (Final)

<小时/>

$docker -v
Docker version 1.4.1, build 5bc2ff8/1.4.1

<小时/>

$java -version
java version "1.7.0_79" OpenJDK Runtime Environment
(rhel-2.5.5.1.el6_6-x86_64 u79-b14) OpenJDK 64-Bit Server VM (build
24.79-b02, mixed mode)

<小时/>

接下来，我创建一个 Maven 项目以使用 Netbeans 提交拓扑。

有1个拓扑和1个spout。(非常简单)

• MyTopology.java
• LoggerSpout.java

MyTopology 在没有 GSON 的情况下也能正常工作。但是，在我在 nextTuple() 函数中添加 Gson gson = new Gson(); 后，我收到 java.lang.NoClassDefFoundError: com/google/gson/Gson。

这就是我所做的 - 1. mvn clean -> mvncompile -> mvnpackage -> 构建成功 2. storm jar ~/Desktop/teststorm/target/teststorm-1.0.jar jp.soushi.teststorm.MyTopology -> 提交即可。

Running: /Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home/bin/java -client -Dstorm.options= -Dstorm.home=/usr/local/apache-storm-0.9.4 -Dstorm.log.dir=/usr/local/apache-storm-0.9.4/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /usr/local/apache-storm-0.9.4/lib/asm-4.0.jar:/usr/local/apache-storm-0.9.4/lib/carbonite-1.4.0.jar:/usr/local/apache-storm-0.9.4/lib/chill-java-0.3.5.jar:/usr/local/apache-storm-0.9.4/lib/clj-stacktrace-0.2.2.jar:/usr/local/apache-storm-0.9.4/lib/clj-time-0.4.1.jar:/usr/local/apache-storm-0.9.4/lib/clojure-1.5.1.jar:/usr/local/apache-storm-0.9.4/lib/clout-1.0.1.jar:/usr/local/apache-storm-0.9.4/lib/commons-codec-1.6.jar:/usr/local/apache-storm-0.9.4/lib/commons-exec-1.1.jar:/usr/local/apache-storm-0.9.4/lib/commons-fileupload-1.2.1.jar:/usr/local/apache-storm-0.9.4/lib/commons-io-2.4.jar:/usr/local/apache-storm-0.9.4/lib/commons-lang-2.5.jar:/usr/local/apache-storm-0.9.4/lib/commons-logging-1.1.3.jar:/usr/local/apache-storm-0.9.4/lib/compojure-1.1.3.jar:/usr/local/apache-storm-0.9.4/lib/core.incubator-0.1.0.jar:/usr/local/apache-storm-0.9.4/lib/disruptor-2.10.1.jar:/usr/local/apache-storm-0.9.4/lib/hiccup-0.3.6.jar:/usr/local/apache-storm-0.9.4/lib/jetty-6.1.26.jar:/usr/local/apache-storm-0.9.4/lib/jetty-util-6.1.26.jar:/usr/local/apache-storm-0.9.4/lib/jgrapht-core-0.9.0.jar:/usr/local/apache-storm-0.9.4/lib/jline-2.11.jar:/usr/local/apache-storm-0.9.4/lib/joda-time-2.0.jar:/usr/local/apache-storm-0.9.4/lib/json-simple-1.1.jar:/usr/local/apache-storm-0.9.4/lib/kryo-2.21.jar:/usr/local/apache-storm-0.9.4/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/apache-storm-0.9.4/lib/logback-classic-1.0.13.jar:/usr/local/apache-storm-0.9.4/lib/logback-core-1.0.13.jar:/usr/local/apache-storm-0.9.4/lib/math.numeric-tower-0.0.1.jar:/usr/local/apache-storm-0.9.4/lib/minlog-1.2.jar:/usr/local/apache-storm-0.9.4/lib/objenesis-1.2.jar:/usr/local/apache-storm-0.9.4/lib/reflectasm-1.07-shaded.jar:/usr/local/apache-storm-0.9.4/lib/ring-core-1.1.5.jar:/usr/local/apache-storm-0.9.4/lib/ring-devel-0.3.11.jar:/usr/local/apache-storm-0.9.4/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/apache-storm-0.9.4/lib/ring-servlet-0.3.11.jar:/usr/local/apache-storm-0.9.4/lib/servlet-api-2.5.jar:/usr/local/apache-storm-0.9.4/lib/slf4j-api-1.7.5.jar:/usr/local/apache-storm-0.9.4/lib/snakeyaml-1.11.jar:/usr/local/apache-storm-0.9.4/lib/storm-core-0.9.4.jar:/usr/local/apache-storm-0.9.4/lib/tools.cli-0.2.4.jar:/usr/local/apache-storm-0.9.4/lib/tools.logging-0.2.3.jar:/usr/local/apache-storm-0.9.4/lib/tools.macro-0.1.0.jar:/Users/soushi/Desktop/teststorm/target/teststorm-1.0.jar:/Users/soushi/.storm:/usr/local/apache-storm-0.9.4/bin -Dstorm.jar=/Users/soushi/Desktop/teststorm/target/teststorm-1.0.jar jp.soushi.teststorm.MyTopology
491  [main] INFO  backtype.storm.StormSubmitter - Jar not uploaded to master yet. Submitting jar...
499  [main] INFO  backtype.storm.StormSubmitter - Uploading topology jar /Users/soushi/Desktop/teststorm/target/teststorm-1.0.jar to assigned location: storm-local/nimbus/inbox/stormjar-b6added0-ffb8-4602-9d89-b567ed87d335.jar
509  [main] INFO  backtype.storm.StormSubmitter - Successfully uploaded topology jar to assigned location: storm-local/nimbus/inbox/stormjar-b6added0-ffb8-4602-9d89-b567ed87d335.jar
509  [main] INFO  backtype.storm.StormSubmitter - Submitting topology MyTopology in distributed mode with conf {"topology.workers":31,"topology.debug":true,"topology.max.spout.pending":5000}
766  [main] INFO  backtype.storm.StormSubmitter - Finished submitting topology: MyTopology

我使用 Storm Web UI 检查了拓扑。过了一会儿，我得到了 NoClassDefFoundError。

我编辑了 nimbus.yaml，并添加了 java.library.path: "/usr/local/lib:/opt/local/lib:/usr/lib:/usr/lib/java"。并确保/usr/lib/java 可以在 docker 容器中访问。

这是错误日志(worker-6727.log)。

2015-04-28T10:05:18.497+0000 b.s.d.executor [ERROR]
java.lang.NoClassDefFoundError: com/google/gson/Gson
at jp.soushi.adcontex.SensorSpout.nextTuple(SensorSpout.java:43) ~[stormjar.jar:na]
at backtype.storm.daemon.executor$fn__4654$fn__4669$fn__4698.invoke(executor.clj:565) ~[storm-core-0.9.4.jar:0.9.4]
at backtype.storm.util$async_loop$fn__458.invoke(util.clj:463) ~[storm-core-0.9.4.jar:0.9.4]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
Caused by: java.lang.ClassNotFoundException: com.google.gson.Gson
at java.net.URLClassLoader$1.run(URLClassLoader.java:366) ~[na:1.7.0_79]
at java.net.URLClassLoader$1.run(URLClassLoader.java:355) ~[na:1.7.0_79]
at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_79]
at java.net.URLClassLoader.findClass(URLClassLoader.java:354) ~[na:1.7.0_79]
at java.lang.ClassLoader.loadClass(ClassLoader.java:425) ~[na:1.7.0_79]
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) ~[na:1.7.0_79]
at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ~[na:1.7.0_79]
... 5 common frames omitted
2015-04-28T10:05:18.564+0000 b.s.util [ERROR] Halting process: ("Worker died")
java.lang.RuntimeException: ("Worker died")
at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:325) [storm-core-0.9.4.jar:0.9.4]
at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker$fn__5102$fn__5103.invoke(worker.clj:495) [storm-core-0.9.4.jar:0.9.4]
at backtype.storm.daemon.executor$mk_executor_data$fn__4555$fn__4556.invoke(executor.clj:240) [storm-core-0.9.4.jar:0.9.4]
at backtype.storm.util$async_loop$fn__458.invoke(util.clj:473) [storm-core-0.9.4.jar:0.9.4]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]

• pom.xml我尝试更改 gson 的范围(运行时、提供、编译)，但它仍然不起作用

<小时/>

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>jp.soushi</groupId>
    <artifactId>teststorm</artifactId>
    <version>1.0</version>
    <packaging>jar</packaging>

    <name>teststorm</name>
    <url>http://maven.apache.org</url>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <maven.compiler.source>1.7</maven.compiler.source>
        <maven.compiler.target>1.7</maven.compiler.target>
    </properties>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.3</version>
                <configuration>
                    <source>1.7</source>
                    <target>1.7</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-jar-plugin</artifactId>
                <configuration>
                    <archive>
                        <manifest>
                            <addClasspath>true</addClasspath>
                            <classpathPrefix>lib/</classpathPrefix>
                            <mainClass>jp.soushi.adcontex.AdContexTopology</mainClass>
                        </manifest>
                    </archive>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-dependency-plugin</artifactId>
                <executions>
                    <execution>
                        <id>copy</id>
                        <phase>install</phase>
                        <goals>
                            <goal>copy-dependencies</goal>
                        </goals>
                        <configuration>
                            <outputDirectory>${project.build.directory}/lib</outputDirectory>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

    <repositories>
        <repository>
            <id>central</id>
            <url>https://repo1.maven.org/maven2</url>
            <releases>
                <enabled>true</enabled>
            </releases>
        </repository>
        <repository>
            <id>github-releases</id>
            <url>http://oss.sonatype.org/content/repositories/github-releases/</url>
        </repository>
        <repository>
            <id>clojars.org</id>
            <url>http://clojars.org/repo</url>
        </repository>
        <repository>
            <id>local-project-libraries</id>
            <name>Local project libraries</name>
            <url>file://${project.basedir}/lib</url>
            <layout>default</layout>
        </repository>
    </repositories>

    <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>3.8.1</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>storm</groupId>
            <artifactId>storm-lib</artifactId>
            <version>0.9.0</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>com.google.code.gson</groupId>
            <artifactId>gson</artifactId>
            <version>2.3.1</version>
            <type>jar</type>
            <scope>runtime</scope>
        </dependency>
    </dependencies>
</project>

• MyTopology.java

<小时/>

package jp.soushi.teststorm;
import backtype.storm.Config;
import backtype.storm.StormSubmitter;
import backtype.storm.topology.TopologyBuilder;

public class MyTopology {
  public static void main(String[] args) throws Exception {
    TopologyBuilder builder = new TopologyBuilder();

    builder.setSpout("Logs", new LoggerSpout(), 5);

    Config conf = new Config();
    conf.setDebug(true);
    conf.setNumWorkers(31);
    conf.setMaxSpoutPending(5000);
    StormSubmitter.submitTopology( "MyTopology", conf, builder.createTopology() );
  }
}

• LoggerSpout.java

<小时/>

package jp.soushi.teststorm;

import backtype.storm.Config;
import backtype.storm.spout.SpoutOutputCollector;
import backtype.storm.task.TopologyContext;
import backtype.storm.topology.OutputFieldsDeclarer;
import backtype.storm.topology.base.BaseRichSpout;
import backtype.storm.tuple.Fields;
import backtype.storm.tuple.Values;
import backtype.storm.utils.Utils;
import com.google.gson.Gson;
import java.util.HashMap;

import java.util.Map;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;


public class LoggerSpout extends BaseRichSpout {
    public static Logger LOG = LoggerFactory.getLogger(LoggerSpout.class);
    boolean _isDistributed;
    SpoutOutputCollector _collector;

    public LoggerSpout() {
        this(true);
    }

    public LoggerSpout(boolean isDistributed) {
        _isDistributed = isDistributed;
    }

    public void open(Map conf, TopologyContext context, SpoutOutputCollector collector) {
        _collector = collector;
    }

    public void close() {

    }

    public void nextTuple() {
        Utils.sleep(100);
        Gson gson = new Gson();

        _collector.emit(new Values("Test"));
    }

    public void ack(Object msgId) {

    }

    public void fail(Object msgId) {

    }

    public void declareOutputFields(OutputFieldsDeclarer declarer) {
        declarer.declare(new Fields("Logs"));
    }

    @Override
    public Map<String, Object> getComponentConfiguration() {
        if(!_isDistributed) {
            Map<String, Object> ret = new HashMap<String, Object>();
            ret.put(Config.TOPOLOGY_MAX_TASK_PARALLELISM, 1);
            return ret;
        } else {
            return null;
        }
    }    
}

我找到了这个答案。但我不太明白。 java.lang.NoClassDefFoundError: com/google/gson/Gson

我也阅读了这些答案。

• Getting java.lang.ClassNotFoundException: com.google.gson.Gson for maven project
• GSON is not being imported into the maven pproject
• http://www.ciiycode.com/0JSJ6PgUXWxq/javalangnoclassdeffounderror-comgooglegsongsonbuilder-error

我花了三天时间解决这个问题...我不知道。谁能提供任何解决方案。提前致谢。

Answer 1

您必须使用 fat jar 运行拓扑，其中包括拓扑使用的所有依赖项类。否则，您的拓扑将无法在运行时找到它们。

您已经尝试构建 fat jar，但您的 jar 很可能不包含 Gson。您可以使用 jar tf YOUR.jar 命令进行检查。

为此，我一直在使用 maven-shade-plugin。引用:https://maven.apache.org/plugins/maven-shade-plugin/