tomcat - Amazon ELB 未能提供响应

Question

我有一个在 Amazon Web Services 上运行的网站，该网站使用 Elastic Beanstalk 进行部署并在单个 EC2 微型实例上运行。这是一个暂存环境，我是唯一可以访问它的人。我使用 Apache JMeter 模拟六个用户在网站上导航，平均每 3 秒一次请求(图像、CSS、JS 和其他静态资源由 CloudFront 提供，不在 EC2 实例上产生流量)。

问题是一段时间后(通常是环境搭建后30-60分钟)，网站停止响应。我确信 Tomcat 仍在正常运行，因为我可以在日志 (catalina.out) 中看到 cronjobs 仍在执行。似乎只有ELB无法响应。

分析日志，Tomcat 上完全没有错误(/opt/tomcat7/logs/tail_catalina.log 或/opt/tomcat7/logs/catalina.out 中没有)。一旦网站无法访问，以下错误就会开始出现在/etc/httpd/logs/elasticbeanstalk-error_log 上:

[Thu Jun 14 20:26:42 2012] [error] (111)Connection refused: proxy: HTTP: attempt to connect to 127.0.0.1:8999 (localhost) failed
[Thu Jun 14 20:26:42 2012] [error] ap_proxy_connect_backend disabling worker for (localhost)
[Thu Jun 14 20:26:50 2012] [error] (111)Connection refused: proxy: HTTP: attempt to connect to 127.0.0.1:8999 (localhost) failed
[Thu Jun 14 20:26:50 2012] [error] ap_proxy_connect_backend disabling worker for (localhost)
[Thu Jun 14 20:27:20 2012] [error] (111)Connection refused: proxy: HTTP: attempt to connect to 127.0.0.1:8999 (localhost) failed
[Thu Jun 14 20:27:20 2012] [error] ap_proxy_connect_backend disabling worker for (localhost)
[Thu Jun 14 20:27:43 2012] [error] (111)Connection refused: proxy: HTTP: attempt to connect to 127.0.0.1:8999 (localhost) failed
[Thu Jun 14 20:27:43 2012] [error] ap_proxy_connect_backend disabling worker for (localhost)
[Thu Jun 14 20:27:50 2012] [error] (111)Connection refused: proxy: HTTP: attempt to connect to 127.0.0.1:8999 (localhost) failed
[Thu Jun 14 20:27:50 2012] [error] ap_proxy_connect_backend disabling worker for (localhost)
[Thu Jun 14 20:28:20 2012] [error] (111)Connection refused: proxy: HTTP: attempt to connect to 127.0.0.1:8999 (localhost) failed
[Thu Jun 14 20:28:20 2012] [error] ap_proxy_connect_backend disabling worker for (localhost)
[Thu Jun 14 20:28:42 2012] [error] (111)Connection refused: proxy: HTTP: attempt to connect to 127.0.0.1:8999 (localhost) failed
[Thu Jun 14 20:28:42 2012] [error] ap_proxy_connect_backend disabling worker for (localhost)
[Thu Jun 14 20:28:50 2012] [error] (111)Connection refused: proxy: HTTP: attempt to connect to 127.0.0.1:8999 (localhost) failed
[Thu Jun 14 20:28:50 2012] [error] ap_proxy_connect_backend disabling worker for (localhost)
[Thu Jun 14 20:29:20 2012] [error] (111)Connection refused: proxy: HTTP: attempt to connect to 127.0.0.1:8999 (localhost) failed
[Thu Jun 14 20:29:20 2012] [error] ap_proxy_connect_backend disabling worker for (localhost)
[Thu Jun 14 20:29:42 2012] [error] (111)Connection refused: proxy: HTTP: attempt to connect to 127.0.0.1:8999 (localhost) failed
[Thu Jun 14 20:29:42 2012] [error] ap_proxy_connect_backend disabling worker for (localhost)
[Thu Jun 14 20:29:50 2012] [error] (111)Connection refused: proxy: HTTP: attempt to connect to 127.0.0.1:8999 (localhost) failed
[Thu Jun 14 20:29:50 2012] [error] ap_proxy_connect_backend disabling worker for (localhost)
[Thu Jun 14 20:30:20 2012] [error] (111)Connection refused: proxy: HTTP: attempt to connect to 127.0.0.1:8999 (localhost) failed
[Thu Jun 14 20:30:20 2012] [error] ap_proxy_connect_backend disabling worker for (localhost)
[Thu Jun 14 20:30:43 2012] [error] (111)Connection refused: proxy: HTTP: attempt to connect to 127.0.0.1:8999 (localhost) failed
[Thu Jun 14 20:30:43 2012] [error] ap_proxy_connect_backend disabling worker for (localhost)
[Thu Jun 14 20:30:50 2012] [error] (111)Connection refused: proxy: HTTP: attempt to connect to 127.0.0.1:8999 (localhost) failed
[Thu Jun 14 20:30:50 2012] [error] ap_proxy_connect_backend disabling worker for (localhost)
[Thu Jun 14 20:31:20 2012] [error] (111)Connection refused: proxy: HTTP: attempt to connect to 127.0.0.1:8999 (localhost) failed
[Thu Jun 14 20:31:20 2012] [error] ap_proxy_connect_backend disabling worker for (localhost)
[Thu Jun 14 20:31:43 2012] [error] (111)Connection refused: proxy: HTTP: attempt to connect to 127.0.0.1:8999 (localhost) failed
[Thu Jun 14 20:31:43 2012] [error] ap_proxy_connect_backend disabling worker for (localhost)
[Thu Jun 14 20:31:50 2012] [error] (111)Connection refused: proxy: HTTP: attempt to connect to 127.0.0.1:8999 (localhost) failed
[Thu Jun 14 20:31:50 2012] [error] ap_proxy_connect_backend disabling worker for (localhost)
[Thu Jun 14 20:32:20 2012] [error] (111)Connection refused: proxy: HTTP: attempt to connect to 127.0.0.1:8999 (localhost) failed
[Thu Jun 14 20:32:20 2012] [error] ap_proxy_connect_backend disabling worker for (localhost)

...直到 EC2 实例最终终止(并自动启动一个新实例)。

如果我不发出任何请求(或者如果我发出的请求更少)，这个问题就不会发生。

非常感谢任何帮助。

谢谢!

Answer 1

让我从一个假设开始:

• 您的 Tomcat 应用程序应该监听 127.0.0.1:8999

如果是，则记录事件:

[Thu Jun 14 20:26:42 2012] [error] (111)Connection refused: proxy: HTTP: attempt to   connect to 127.0.0.1:8999 (localhost) failed
[Thu Jun 14 20:26:42 2012] [error] ap_proxy_connect_backend disabling worker for (localhost)

..建议应用程序监听器终止。您可以通过以下方式确认这一点:

curl -v http://127.0.0.1:8999/

该 curl 命令应该在站点正常运行时返回有效的 HTTP 响应，并且可能会返回 Connection refused 或 couldn't connect to host 当您遇到中断时。您还可以使用以下命令检查应用程序端口上的有效监听器:

netstat -an | grep LISTEN | grep 8999

应用程序监听器挂掉的原因有很多，包括但不限于:

• JVM 硬崩溃(使用 ps 查看 JVM 进程是否仍在运行)
• 应用程序软崩溃(查看 Tomcat 应用程序日志)
• 用完文件描述符(使用 lsof | wc -l 并与应用程序用户的 ulimit -n 进行比较)

但是，大多数错误会导致将错误消息写入 JVM 进程的 stderr，这通常会记录下来。那是最好看的地方。如果一切都失败了，您可能想尝试在启用调试日志记录的情况下在前台运行您的 Tomcat 应用程序。