javascript - 如何在超过超时时阻止 NodeJS 脚本崩溃

Question

您可以在下面找到发生的错误。

我正在尝试使用 NodeJS 和 puppeteer 抓取网站内容。有时代码会因超时错误而停止。有没有一种方法，如果超过页面加载超时，我可以运行一个函数来重新加载页面，或者让脚本等待几秒钟，然后重新加载页面，直到它正确获取数据，而不会崩溃？如果是这样，我将如何着手实现它？

谢谢。

(node:8300) UnhandledPromiseRejectionWarning: TimeoutError: Navigation Timeout Exceeded: 30000ms exceeded
    at Promise.then (C:\Users\danie\node_modules\puppeteer\lib\LifecycleWatcher.js:143:21)
  -- ASYNC --
    at Frame.<anonymous> (C:\Users\danie\node_modules\puppeteer\lib\helper.js:108:27)
    at Page.goto (C:\Users\danie\node_modules\puppeteer\lib\Page.js:656:49)
    at Page.<anonymous> (C:\Users\danie\node_modules\puppeteer\lib\helper.js:109:23)
    at scrape (C:\Users\danie\Documents\Node Projects\p-download.js:23:14)
    at process._tickCallback (internal/process/next_tick.js:68:7)
(node:8300) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:8300) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

我的代码:

const puppeteer = require('puppeteer');

let scrape = async () => {
    const browser = await puppeteer.launch({headless: false});
    const page = await browser.newPage();

    await page.setRequestInterception(true);    
    page.on('request', (req) => {
        if(req.resourceType() == 'stylesheet' || req.resourceType() == 'script' || req.resourceType() == 'font' || req.resourceType() == 'media' || req.resourceType() == 'image'){
            req.abort();
        }
        else {
            req.continue();
        }
    }); //Disables loading CSS, images and scripts

    for(i=0; i<5000; i++){
        await page.goto('https://website.com/' + i);
        let result = await page.evaluate(() => {
            var result = '';
            for (i=1; i<=10; i++){
                result += document.getElementsByTagName('td')[i].innerText;
                result += ',';
            }
            result += '\n';
            return result;
        });
    }
}
scrape();

Answer 1

将您的代码放在 try/catch block 中以避免崩溃...我会将循环代码移入另一个函数

for(i=0; i<5000; i++){
    result = await open_page(page , i );
}


async function open_page(page , i ){
    try {
            await page.goto('https://website.com/' + i);
            let result = await page.evaluate(() => {
                var result = '';
                for (i=1; i<=10; i++){
                    result += document.getElementsByTagName('td')[i].innerText;
                    result += ',';
                }
                result += '\n';
                return result;
            });


            return {stat:1 , result : result } ;

    }
    catch(e){
        return {stat:0 , error : e } ;
    }

}