node.js - 如何限制 Node 中简化HTTP请求的内容长度响应？

Question

我想设置 simplified HTTP request() client package中止下载太大的 HTTP 资源。

假设 request() 设置为下载一个 url，资源大小为 5 GB。我希望 request() 在 10MB 后停止下载。通常，当请求得到答复时，它会得到所有 HTTP header 和后面的所有内容。一旦你操作了数据，你就已经拥有了所有下载的数据。

在 axios 中，有一个名为 maxContentLength 的参数，但我找不到与 request() 类似的参数。

我还必须提到，我不是为了捕获错误而只是至少下载资源的 header 和开头。

Answer 1

const request = require('request');
const URL = 'http://de.releases.ubuntu.com/xenial/ubuntu-16.04.3-desktop-amd64.iso';
const MAX_SIZE = 10 * 1024 * 1024 // 10MB , maximum size to download
let total_bytes_read = 0;

1 - If the response from the server is gzip-compressed , you should enable gzip option. https://github.com/request/request#examples For backwards-compatibility, response compression is not supported by default. To accept gzip-compressed responses, set the gzip option to true.

request
    .get({
        uri: URL,
        gzip: true
    })
    .on('error', function (error) {
        //TODO: error handling
        console.error('ERROR::', error);
    })
    .on('data', function (data) {
        // decompressed data 
        console.log('Decompressed  chunck Recived:' + data.length, ': Total downloaded:', total_bytes_read)
        total_bytes_read += data.length;
        if (total_bytes_read >= MAX_SIZE) {
            //TODO: handle exceeds max size event
            console.error("Request exceeds max size.");
            throw new Error('Request exceeds max size'); //stop
        }
    })
    .on('response', function (response) {
        response.on('data', function (chunk) {
            //compressed data
            console.log('Compressed  chunck Recived:' + chunk.length, ': Total downloaded:', total_bytes_read)
        });
    })
    .on('end', function () {
        console.log('Request completed! Total size downloaded:', total_bytes_read)
    });

NB: If the server does not compress response but you still use gzip option / decompress, then the decompress chunk & the original chunk will be equal. Hence you can do the Limit check either way(from the decompressed / compressed chunk) However if response is compressed you should check the size limit of the decompressed chunk

2 - if the response is not compressed you don't need gzip option to decompress

request
    .get(URL)
    .on('error', function (error) {
        //TODO: error handling
        console.error('ERROR::', error);
    })
    .on('response', function (response) {
        response.on('data', function (chunk) {
            //compressed data
            console.log('Recived chunck:' + chunk.length, ': Total downloaded:', total_bytes_read)
            total_bytes_read += chunk.length;
            if (total_bytes_read >= MAX_SIZE) {
                //TODO: handle exceeds max size event
                console.error("Request as it exceds max size:")
                throw new Error('Request as it exceds max size');
            }
            console.log("...");
        });
    })
    .on('end', function () {
        console.log('Request completed! Total size downloaded:', total_bytes_read)
    });