docker - Nifi : Could not write to StandardFlowfileRecord 中的错误

Question

我最近创建了一个 Nifi docker 镜像，以便能够使用我的自定义配置和状态运行 nifi docker。

一切正常，但过了一会儿，我收到以下错误:

2019-04-08 10:42:26,107 ERROR [Timer-Driven Process Thread-10] o.a.n.c.r.StandardProcessSession Failed to commit session java.lang.IllegalStateException: Cannot update journal file ./flowfile_repository/journals/449486.journal because no header has been written yet.. Will roll back.
2019-04-08 10:42:26,108 ERROR [Timer-Driven Process Thread-10] d.f.fit.processors.opcua.GetOPCData GetOPCData[id=e8affa98-0169-1000-1717-d9e8f54293ea] GetOPCData[id=e8affa98-0169-1000-1717-d9e8f54293ea] failed to process session due to java.lang.IllegalStateException: Cannot update journal file ./flowfile_repository/journals/449486.journal because no header has been written yet.; Processor Administratively Yielded for 1 sec: 
 java.lang.IllegalStateException: Cannot update journal file ./flowfile_repository/journals/449486.journal because no header has been written yet.
    at org.apache.nifi.wali.LengthDelimitedJournal.update(LengthDelimitedJournal.java:230)
    at org.apache.nifi.wali.SequentialAccessWriteAheadLog.update(SequentialAccessWriteAheadLog.java:123)
    at org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.updateRepository(WriteAheadFlowFileRepository.java:309)
    at org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.updateRepository(WriteAheadFlowFileRepository.java:259)
    at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:368)
    at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:343)
    at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28)
    at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
    at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:205)
    at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    Suppressed: java.lang.IllegalStateException: Cannot update journal file ./flowfile_repository/journals/449486.journal because no header has been written yet.
            at org.apache.nifi.wali.LengthDelimitedJournal.update(LengthDelimitedJournal.java:230)
            at org.apache.nifi.wali.SequentialAccessWriteAheadLog.update(SequentialAccessWriteAheadLog.java:123)
            at org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.updateRepository(WriteAheadFlowFileRepository.java:309)
            at org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.updateRepository(WriteAheadFlowFileRepository.java:259)
            at org.apache.nifi.controller.repository.StandardProcessSession.rollback(StandardProcessSession.java:1060)
            at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:501)
            ... 12 common frames omitted

它与java堆或内存有关吗？我该如何纠正这个错误？

Answer 1

在我的情况下，我的打开文件句柄用完了，不幸的是，错误没有给出任何指示。我怎么认为它是:

查询 NiFi 诊断。前任:curl http://localhost:8080/nifi-api/system-diagnostics
检查所有“空闲”字段(例如“provenanceRepositoryStorageUsage.freeSpace”)并确保您有足够的资源。

对我来说，这返回的值如下:
flowFileRepositoryStorageUsage.totalSpaceBytes=-1
-1 当 NiFi 无法确定这些值时报告。作为诊断的一部分，将记录正确的错误消息。您可能需要快速检查日志。在我的例子中，我设置了滚动日志，所以日志消息很快就消失了(很多处理器都在记录故障并填满日志)。错误看起来像:

2019-06-03 18:48:59,619 ERROR [pool-10-thread-1] org.apache.nifi.controller.repository.WriteAheadFlowFileRepository Unable to checkpoint FlowFile Repository due to java.io.FileNotFoundException: ./flowfile_repository/journals/8243.journal (Too many open files)
java.io.FileNotFoundException: ./flowfile_repository/journals/8243.journal (Too many open files)
    at java.io.FileOutputStream.open0(Native Method)
    at java.io.FileOutputStream.open(FileOutputStream.java:270)
    at java.io.FileOutputStream.\u003cinit\u003e(FileOutputStream.java:213)
    at java.io.FileOutputStream.\u003cinit\u003e(FileOtutputStream.java:162)\n\tat org.apache.nifi.wali.LengthDelimitedJournal.getOutputStream(LengthDelimitedJournal.java:136)
    at org.apache.nifi.wali.LengthDelimitedJournal.writeHeader(LengthDelimitedJournal.java:151)
    at org.apache.nifi.wali.SequentialAccessWriteAheadLog.checkpoint(SequentialAccessWriteAheadLog.java:306)
    at org.apache.nifi.wali.SequentialAccessWriteAheadLog.checkpoint(SequentialAccessWriteAheadLog.java:251)
    at org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.checkpoint(WriteAheadFlowFileRepository.java:735)
    at org.apache.nifi.controller.repository.WriteAheadFlowFileRepository$1.run(WriteAheadFlowFileRepository.java:693)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

这里的重要部分是“打开的文件太多”。

使用 linux 获取当前限制: ulimit -x -n
使用 linux 获取打开文件的数量: sudo ls /proc/$NIFI_PID/fd | wc -l
我正在使用 docker，所以我使用 docker run 命令增加了限制: docker run --ulimit nofile=1048576:1048576 ...
其他解决方案是:增加对主机的限制，确定哪些处理器正在占用句柄并重新处理它们(不是那么容易)。