java - Flume java.lang.IllegalStateException : File has changed size since being read

Question

我有一个java应用程序，它从不同来源收集数据并将输出写入特定目录下的文件中。

我有一个 Flume 代理，配置为使用 spooldir 源从该目录读取数据，并使用 MorphlineSolrSink 将输出写入 Solr。

flume代理抛出以下异常

java.lang.IllegalStateException: File has changed size since being read

这里是flume代理的配置

agent02.sources = s1
agent02.sinks = solrSink
agent02.channels = ch1

agent02.channels.ch1.type = file
agent02.channels.ch1.checkpointDir=/home/flume/prod_solr_chkpoint/file-channel/checkpoint
agent02.channels.ch1.dataDirs= /home/flume/prod_solr_chkpoint/file-channel/data

agent02.sources.s1.type = spooldir
agent02.sources.s1.channels = ch1

agent02.sources.s1.spoolDir = /DataCollection/json_output/solr/
agent02.sources.s1.deserializer.maxLineLength = 100000

agent02.sinks.solrSink.type = org.apache.flume.sink.solr.morphline.MorphlineSolrSink
agent02.sinks.solrSink.channel = ch1
agent02.sinks.solrSink.batchSize = 10000
agent02.sinks.solrSink.batchDurationMillis = 10000
agent02.sinks.solrSink.morphlineFile = morphlines.conf 
agent02.sinks.solrSink.morphlineId = morphline

我从异常中了解到的是，flume 代理开始处理文件，而 java 应用程序尚未完成写入。

如何解决这个问题？

编辑

我不知道这些信息是否有值(value)。这些配置之前可以正常工作，没有任何问题。我们运行 Flume 的机器遇到了硬盘故障。从失败中恢复后，flume 会抛出此异常。

Answer 1

如有关 Spooling Directory Source 的文档中所述:

In exchange for this reliability, only immutable, uniquely-named filesmust be dropped into the spooling directory. Flume tries to detectthese problem conditions and will fail loudly if they are violated:

• If a file is written to after being placed into the spooling directory, Flume will print an error to its log file and stopprocessing.
• If a file name is reused at a later time, Flume will print an error to its log file and stop processing.

我建议您的 Java 应用程序将数据桶转储到临时文件中；通过添加创建时间戳来命名它们。一旦存储桶已满(即达到一定大小)，则将文件移动到假脱机目录。