I'm developing what I thought would be a straightforward Nginx reverse proxy. Its goal is to forward/proxy all requests to another load balancer (ALB) behind which static assets reside. The traffic on the proxy is expected to be minimal and currently it is receiving only test traffic (one user at a time).

我正在开发我认为会是一个简单的Nginx反向代理的东西。它的目标是将所有请求转发/代理到静态资产所在的另一个负载平衡器(ALB)。预计代理上的流量很小，目前它只接收测试流量(一次一个用户)。

Details:

详细信息：

• Deployed on EC2 instance (m5.xlarge).
  
  
  

  • amzn-linux 2: CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"

    AMZN-Linux 2：CPE_NAME=“CPE：2.3：O：Amazon：Amazon_Linux：2”

    
    
  
  

  • Docker Compose version v2.13.0

    Docker Compose版本v2.13.0

    
    
  
  

  • Docker version 20.10.17, build 100c701

    Docker版本20.10.17，内部版本100c701

    
    
  
  

  • Nothing else weird has been done setting it up (just enough to get docker and docker-compose working). No other services run on it.

    没有完成任何其他奇怪的设置(仅足以让docker和docker-compose工作)。没有其他服务在其上运行。

    
    
  
  
  
  


• EC2 instance is inside a target group, behind an AWS ALB. It is currently the only instance in the target group


• It is currently run by ssh'ing to the box and using docker-compose up -d


• It is the latest nginx image

When it first starts, everything is great. No errors. Then, approximately 1-3 days later it begins returning 404's intermittently. Sometimes it is on the index.html initial load, sometimes on static assets like .js, .css, or image files. Every time I refresh the page, there are usually 2-3 404s and on different requests each time. The rest of the files return 200s. From the logs below, it appears to happen on almost every other request.

当它第一次开始的时候，一切都很好。没有错误。然后，大约1-3天后，它开始间歇性地返回404‘S。有时是在index.html初始加载上，有时是在.js、.css或图像文件等静态资源上。每次我刷新页面时，通常都有2-3个404，每次都有不同的请求。其余的文件返回200秒。从下面的日志来看，几乎所有其他请求都会发生这种情况。

If I let it sit for several hours, it continuously occurs. It does not "resolve itself".

如果我让它静置几个小时，它会持续发生。它不会“自行解决”。

Workaround/Fix:

解决方法/修复：

• Restart the docker-compose just doing a docker-compose down then up. I do nothing else to fix it. Restarting immediately fixes the problem until 1-3 days later it begins happening again.

Examples from docker-compose logs:

来自docker-compose日志的示例：

nginx  | 10.0.51.10 - - [10/Sep/2023:01:58:55 +0000] "GET /dataportal/favicon.ico HTTP/1.1" 200 1706 "https://xxxxxxx.xxxxxxxx.com/dataportal/dashboards" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36"
nginx  | 10.0.51.10 - - [10/Sep/2023:01:58:55 +0000] "GET /dataportal/static/css/main.3fb3fe79.css HTTP/1.1" 200 86129 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36"
nginx  | 10.0.51.10 - - [10/Sep/2023:01:58:55 +0000] "GET /dataportal/static/js/main.5c5d85b7.js HTTP/1.1" 404 967 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36"
nginx  | 10.0.51.10 - - [10/Sep/2023:01:58:56 +0000] "GET /api/dataportal/dashboards HTTP/1.1" 200 2043 "https://xxxxxxx.xxxxxxxx.com/dataportal/dashboards" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36"
nginx  | 10.0.52.62 - - [10/Sep/2023:01:58:57 +0000] "GET /dataportal/static/css/main.51fa6b06.css HTTP/1.1" 200 1706 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US) AkamaiImageServer VelocitudeMP/1.0;IM/1.0"
nginx  | 10.0.51.10 - - [10/Sep/2023:01:58:57 +0000] "GET /dataportal/static/js/main.d4bbd5f8.js HTTP/1.1" 404 967 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US) AkamaiImageServer VelocitudeMP/1.0;IM/1.0"

There are no errors in docker-compose logs besides the 404's.

除《S》外，其余编录日志均无错误。

~/nginx.conf file:

~/nginx.conf文件：

events {
  worker_connections  4096;  ## Default: 1024
}

http {
  server {
    listen 80;
  
    location = /health {
            access_log off;
            add_header 'Content-Type' 'application/json';
            return 200 '{"status":"UP"}';
    }

    location / {
        proxy_pass https://XXXXXXXXXXXX.us-east-1.elb.amazonaws.com;
        proxy_set_header Host $host;
    }
 }
}

~/docker-compose.yml file:

~/docker-compose.yml文件：

version: "3"
services:

  nginx:
    container_name: nginx
    image: nginx:latest
    ports:
      - "80:80"
      #- "443:443"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf

I've tried:

我试过了：

1. I've checked that the docker-compose is still running. Nothing in logs beyond the 200 and 404 status calls.


2. I verified that no one else is accessing/modifying the server/container.


3. I've restarted the docker-compose, which works for 1-3 days. I've done this three times and each time it temporarily fixes it. Then it begins happening again.


4. I've checked top for CPU/memory and df for disk space while it is returning 404s. Top shows basically no usage anywhere, as expected. Disk usage is at 30%. EC2 instance is healthy and Cloudwatch monitoring shows no issues.


5. Downstream ALB has healthy resources that don't report issues. Since it is temporarily "fixed" when I restart the docker-compose reverse proxy, that seems to me to isolate it to that service.

The fact that this is intermittent confuses me. And given how it works great for a day then degrades over time, I would expect it to be a resource issue, but as I mentioned there is basically zero load and resources seem abundant.

这是间歇性的，这让我感到困惑。考虑到它是如何在一天内工作得很好，然后随着时间的推移而降级，我预计这将是一个资源问题，但正如我前面提到的，基本上没有负载，资源似乎很充足。

Any help would be appreciated, thanks!

如有任何帮助，我们将不胜感激，谢谢！