java - zeromq - 从 Tomcat 登录时处理 IOException

Question

抱歉，我对 Java 比较陌生。我正在编写一个库，使用 zeromq (jeromq) 将日志从 Java 应用程序发送到 Logstash。

我的测试服务器之一是运行在 Tomcat 中的繁忙的 Jenkins master。我的库“Logit”(https://github.com/stuart-warren/logit)使用 PUSHPULL 套接字将其 JSON 格式的日志 (JUL) 喷射到已配置的 Logstash 端点列表(用于一些基本的负载平衡)以提高速度。

不幸的是，几天后它遇到了异常。 Tomcat/Jenkins 继续工作并将日志写入其正常文件，但 Logit 停止通过网络发送消息。

18-Nov-2013 14:45:24 hudson.model.Run execute
INFO: EmployeeManagementSystems » Project Phase 2 branch » Int Tests » Sync Int Tests (Project - Prl) #326 aborted
java.lang.InterruptedException
        at java.lang.Object.wait(Native Method)
        at hudson.remoting.Request.call(Request.java:146)
        at hudson.remoting.Channel.call(Channel.java:714)
        at hudson.maven.ProcessCache$MavenProcess.call(ProcessCache.java:156)
        at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.doRun(MavenModuleSetBuild.java:815)
        at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:565)
        at hudson.model.Run.execute(Run.java:1592)
        at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:508)
        at hudson.model.ResourceController.execute(ResourceController.java:88)
        at hudson.model.Executor.run(Executor.java:237)
logit:WARN IOException thrown, will try sending log again shortly.
zmq.ZError$IOException: java.nio.channels.ClosedByInterruptException
        at zmq.Signaler.send(Signaler.java:108)
        at zmq.Mailbox.send(Mailbox.java:90)
        at zmq.Ctx.send_command(Ctx.java:351)
        at zmq.ZObject.send_command(ZObject.java:364)
        at zmq.ZObject.send_activate_read(ZObject.java:217)
        at zmq.Pipe.flush(Pipe.java:284)
        at zmq.LB.send(LB.java:120)
        at zmq.Push.xsend(Push.java:64)
        at zmq.SocketBase.send(SocketBase.java:598)
        at org.jeromq.ZMQ$Socket.send(ZMQ.java:932)
        at com.stuartwarren.logit.zmq.ZmqTransport.appendString(ZmqTransport.java:115)
        at com.stuartwarren.logit.jul.ZmqAppender.publish(ZmqAppender.java:77)
        at java.util.logging.Logger.log(Logger.java:481)
        at java.util.logging.Logger.doLog(Logger.java:503)
        at java.util.logging.Logger.log(Logger.java:592)
        at hudson.model.Run.execute(Run.java:1610)
        at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:508)
        at hudson.model.ResourceController.execute(ResourceController.java:88)
        at hudson.model.Executor.run(Executor.java:237)
Caused by: java.nio.channels.ClosedByInterruptException
        at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
        at sun.nio.ch.SinkChannelImpl.write(SinkChannelImpl.java:154)
        at zmq.Signaler.send(Signaler.java:106)
        ... 18 more
logit:ERROR Logit got interrupted while waiting to send failed message again.
java.lang.InterruptedException: sleep interrupted
        at java.lang.Thread.sleep(Native Method)
        at com.stuartwarren.logit.zmq.ZmqTransport.appendString(ZmqTransport.java:121)
        at com.stuartwarren.logit.jul.ZmqAppender.publish(ZmqAppender.java:77)
        at java.util.logging.Logger.log(Logger.java:481)
        at java.util.logging.Logger.doLog(Logger.java:503)
        at java.util.logging.Logger.log(Logger.java:592)
        at hudson.model.Run.execute(Run.java:1610)
        at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:508)
        at hudson.model.ResourceController.execute(ResourceController.java:88)
        at hudson.model.Executor.run(Executor.java:237)
18-Nov-2013 14:45:35 hudson.model.Run execute

我正在使用相同版本的库从测试 6 节点 Cassandra 集群日志记录，没有问题 (log4j)，但我想这可能只是运气？

我的问题是，在发送消息时应该如何处理上述异常？ https://github.com/stuart-warren/logit/blob/master/src/main/java/com/stuartwarren/logit/zmq/ZmqTransport.java#L115 .

public void appendString(final String line) {
    final String log = line.substring(0, line.length() - 1);
    if (LogitLog.isTraceEnabled()) {
        LogitLog.trace("Sending log: [" + log + "].");
    }
    try {
        socket.send(log, ZMQ.NOBLOCK);
        // Has occasionally been known to throw a java.nio.channels.ClosedByInterruptException
    } catch (IOException e) {
        LogitLog.warn("IOException thrown, will try sending log again shortly.", e);
        // Try again after sleeping for a second
        try {
            Thread.sleep(1000);
            socket.send(log, ZMQ.NOBLOCK);
        } catch (InterruptedException i) {
            LogitLog.error("Logit got interrupted while waiting to send failed message again.", i);
        } catch (IOException e2) {
            LogitLog.error("Could not send following log on the second attempt: [" + log + "].", e2);
        }
    } catch (Exception g) {
        LogitLog.error("Something threw an exception that wasn't IOException.", g);
    }
}

我应该关闭并重新打开套接字/上下文，还是应该在 Jeromq 库中处理？也许 Tomcat 更有可能抛出这些并且它应该在我的 JUL appender 中处理？

谢谢。

Answer 1

看起来你的答案可以找到HERE在这个线程中。 #cafebabe 在下面的评论中为他们的答案添加了更多信息，可以更直接地回答您的问题:

“如果应用程序是多线程的，您应该寻找可能中断在 channel 上执行 IO 操作的线程的#interrupt() 调用。如果这是一个网络应用程序或其他类型的在托管环境中，线程管理不取决于您的应用程序(如 Servlet/EJB 容器)，您应该查找线程安全违规。另一个要查看的地方是应用程序关闭或使用线程池时(Servlet/EJB 容器!)。然后注意池大小的动态管理!”