javascript - 如何通过适当的 async/await/promises 提高此代码的速度

Question

使用 Node.js，我的任务是改进我创建的代码。此代码执行 60 个 HTTP 请求并为此使用库。

完成所有 HTTP 请求并将每个请求保存到文件需要 30 秒!

据说可以在 3 秒内完成这些请求:

<强>1。正确管理异步 promise

<强>2。更智能的缓存

<强>3。不使用集群

<强>4。仅添加一次开销

我担心我不知道从哪里开始了解我到底能做什么。

下面的代码获取了一个包含 60 个项目的数组，其中每个项目都是一个 HTTP 请求:

const exchanges = ccxt.exchanges

这些进入:worker = async 函数，并在代码末尾:await Promise.all(workers) 等待它们完成。

我不知道从哪里开始才能真正减少到 3 秒。如何才能提高这段代码的速度？

'use strict';

const ccxt  = require ('ccxt')
    , log   = require ('ololog').noLocate // npm install ololog
    , fs    = require ('fs')

    // the numWorkers constant defines the number of concurrent workers
    // those aren't really threads in terms of the async environment
    // set this to the number of cores in your CPU * 2
    // or play with this number to find a setting that works best for you
    , numWorkers = 8

;(async () => {

    // make an array of all exchanges
    const exchanges = ccxt.exchanges

        .filter (id => ![ 'cap1', 'cap2' ].includes (id))

        // instantiate each exchange and save it to the exchanges list
        .map (id => new ccxt[id] ({
            'enableRateLimit': true,
        }))

    // the worker function for each "async thread"
    const worker = async function () {

        // while the array of all exchanges is not empty
        while (exchanges.length > 0) {

            // pop one exchange from the array
            const exchange = exchanges.pop()

            // check if it has the necessary method implemented
            if (exchange.has['fetchTickers']) {

                // try to do "the work" and handle errors if any
                try {

                    // fetch the response for all tickers from the exchange
                    const tickers = await exchange.fetchTickers()

                    // make a filename from exchange id
                    const filename = '/myproject/tickers/' + exchange.id + 'Tickers.json'

                    // save the response to a file
                    fs.writeFileSync(filename, JSON.stringify({ tickers }));

                } catch (e) { } //Error
            }
        }
    }

    // create numWorkers "threads" (they aren't really threads)
    const workers = [ ... Array (numWorkers) ].map (_ => worker ())

    // wait for all of them to execute or fail
    await Promise.all (workers)

}) ()

Answer 1

我认为你让事情变得比他们需要的更复杂。您可以在 map 回调中完成所有工作，然后使用 Promise.all(promises) 等待所有操作完成。这个过程确实比预期的“3秒”(在我的例子中是15秒)要长，并且产生了很多错误(比如缺少 apiToken，或者 fetchTickers 没有被实现)，但这可能是我的环境的问题(我以前从未使用过 ccxt 并且我没有任何 apiTokens)。

这是我想出的实现，希望它可以帮助您满足您的需求:

const ccxt = require('ccxt');
const fs = require('fs');
const path = require('path');

(async () => {
    const start = Date.now();

    const dumps = ccxt.exchanges
        .filter((id) => !['coinmarketcap', 'theocean'].includes(id))
        .map(async (id) => {
            const Exchange = ccxt[id];
            const exchange = new Exchange({enableRateLimit: true});
            if (exchange.has['fetchTickers']) {
                try {
                    const tickers = await exchange.fetchTickers();
                    const dumpFile = path.join(__dirname, 'exchanges', `${id}-Tickers.json`);
                    await fs.promises.writeFile(dumpFile, JSON.stringify(tickers));
                } catch (e) {
                    console.error(e);
                }
            }
        });

    await Promise.all(dumps);

    const end = Date.now();
    console.log(`Done in ${(end - start) / 1000} seconds`);
})();