gpt4 book ai didi

hdfs - Flume 接收到 HDFS 错误 : java. lang.NoSuchMethodError : com. google.common.base.Preconditions.checkArgument

转载 作者:行者123 更新时间:2023-12-03 20:27:14 24 4
gpt4 key购买 nike

和:

  • Java 1.8.0_231
  • Hadoop 3.2.1
  • 水槽 1.8.0

  • 在 9000 端口上创建了 hdfs 服务。

    jps:
    11688 DataNode
    10120 Jps
    11465 NameNode
    11964 SecondaryNameNode
    12621 NodeManager
    12239 ResourceManager

    水槽配置:
    agent1.channels.memory-channel.type=memory
    agent1.sources.tail-source.type=exec
    agent1.sources.tail-source.command=tail -F /var/log/nginx/access.log
    agent1.sources.tail-source.channels=memory-channel

    #hdfs sink
    agent1.sinks.hdfs-sink.channel=memory-channel
    agent1.sinks.hdfs-sink.type=hdfs
    agent1.sinks.hdfs-sink.hdfs.path=hdfs://cluster01:9000/system.log
    agent1.sinks.hdfs-sink.hdfs.fileType=DataStream
    agent1.channels=memory-channel
    agent1.sources=tail-source
    agent1.sinks=log-sink hdfs-sink

    然后启动水槽:
    ./bin/flume-ng agent --conf conf -conf-file conf/test1.conf --name agent1 -Dflume.root.logger=INFO,console

    然后遇到错误:
    Info: Including Hadoop libraries found via (/usr/local/hadoop-3.2.1/bin/hadoop) for HDFS access
    ...
    2019-11-04 14:48:24,818 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: SINK, name: hdfs-sink started
    2019-11-04 14:48:28,823 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.HDFSDataStream.configure(HDFSDataStream.java:57)] Serializer = TEXT, UseRawLocalFileSystem = false
    2019-11-04 14:48:28,836 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:447)] process failed
    java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
    at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357)
    at org.apache.hadoop.conf.Configuration.set(Configuration.java:1338)
    at org.apache.hadoop.conf.Configuration.setBoolean(Configuration.java:1679)
    at org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:226)
    at org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:541)
    at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:401)
    at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
    at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
    at java.lang.Thread.run(Thread.java:748)
    Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
    at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357)
    at org.apache.hadoop.conf.Configuration.set(Configuration.java:1338)
    at org.apache.hadoop.conf.Configuration.setBoolean(Configuration.java:1679)
    at org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:226)
    at org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:541)
    at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:401)
    at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
    at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
    at java.lang.Thread.run(Thread.java:748)

    我已经搜索了一段时间,但在网上没有发现同样的错误。有什么建议可以解决这个问题吗?

    最佳答案

    这可能是由 lib/guava 引起的。

    我删除了 lib/guava-11.0.2.jar,并重新启动了水槽,发现它有效。

    输出:

    2019-11-04 16:52:58,062 (hdfs-hdfs-sink-call-runner-0) [WARN - org.apache.hadoop.util.NativeCodeLoader.<clinit>(NativeCodeLoader.java:60)] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    2019-11-04 16:53:01,532 (Thread-9) [INFO - org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:239)] SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false

    但我仍然不知道它现在使用哪个版本的 Guava 。

    关于hdfs - Flume 接收到 HDFS 错误 : java. lang.NoSuchMethodError : com. google.common.base.Preconditions.checkArgument,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/58688470/

    24 4 0
    Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
    广告合作:1813099741@qq.com 6ren.com