node.js - Google App Engine - 间歇性 502/连接由同行重置

Question

在 Google App Engine (GAE) flex 实例上运行 nodejs 服务器 我有客户从我的应用程序中收到间歇性 502 错误。这些请求从未到达我的 Node 服务，但它们似乎与与由对等方重置的连接相关的 nginx 日志一致:

[error] 34#34: *25817 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 1.2.3.4, server: , request: "POST /endpoint HTTP/1.1", upstream: "http://172.17.0.1:8080/endpoint"
[error] 34#34: *27919 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 1.2.3.4, server: , request: "POST /endpoint HTTP/1.1", upstream: "http://172.17.0.1:8080/endpoint"
[error] 34#34: *28746 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 1.2.3.4, server: , request: "POST /endpoint HTTP/1.1", upstream: "http://172.17.0.1:8080/endpoint"
[error] 34#34: *28747 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 1.2.3.4, server: , request: "POST /endpoint HTTP/1.1", upstream: "http://172.17.0.1:8080/endpoint"
[error] 34#34: *24022 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 1.2.3.4, server: , request: "POST /endpoint HTTP/1.1", upstream: "http://172.17.0.1:8080/endpoint"
[error] 34#34: *29214 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 1.2.3.4, server: , request: "POST /endpoint HTTP/1.1", upstream: "http://172.17.0.1:8080/endpoint"

是什么导致了这种行为？ CPU/内存负载没有接近资源限制，尽管当服务器处于某些负载下时它似乎更频繁地发生。

Answer 1

当部署到 Google App Engine 时，负载平衡器被放置在实例之前。此负载均衡器的 HTTP 保持事件设置为 600 seconds .

负载均衡器然后连接到实例上的 nginx 服务，该服务使用 650 秒的保活时间，它甚至在配置中有一个有用的注释，说它需要更长的时间来防止竞争条件。

# GCLB uses a 10 minutes keep-alive timeout. Setting it to a bit more here
# to avoid a race condition between the two timeouts.
keepalive_timeout 650;

最后，nginx 反向代理到您的 Node 应用程序，它使用默认的保持事件... 5 seconds

这会导致超时之间的竞争条件 (duh)，您需要将 Node 服务器的超时设置为高于 650 秒。如果您使用的是像这样的 expressjs:

const app = express();
const server = app.listen(process.env.PORT);

//nginx uses a 650 second keep-alive timeout on GAE. Setting it to a bit more here to avoid a race condition between the two timeouts.
server.keepAliveTimeout = 700000; 

//ensure the headersTimeout is set higher than the keepAliveTimeout due to this nodejs regression bug: https://github.com/nodejs/node/issues/27363
server.headersTimeout = 701000; 

可以查看Analyze ‘Connection reset’ error in Nginx upstream with keep-alive enabled有关上游服务器为何需要更大超时的技术(TCP 级别)解释。