javascript - puppeteer 师得到错误 : Evaluation failed: TypeError: Cannot read properties of null (reading 'innerText' ) when returning multiple values

Question

使用 puppeteer 从站点获取一些数据。我需要返回从站点获得的多个值，但由于某种原因我一次只能返回一个值，每当我尝试返回多个值(如下面代码中的值)时，我都会收到以下错误:错误:评估失败:TypeError:返回多个值时无法读取 null 的属性(读取“innerText”)，我找不到原因。

代码

(async () => {
    try {
        const chromeBrowser = await puppeterr.launch({ headless: true });
        const page = await chromeBrowser.newPage();
        await page.goto("https://www.sec.gov/edgar/search/#/category=form-cat2", {timeout: 0});

    const getInfo = await page.evaluate(() => {
        const secTableEN = document.querySelector(".table td.entity-name");
        const secTableFiled = document.querySelector(".table td.entity-filed");
        const secTableLink = document.querySelector(".table td.filetype");

        return {
            secTableEN: secTableEN.innerText,
            secTableFiled: secTableFiled.innerText,
        };
    })

    console.log(getInfo);
    await page.close();
    await chromeBrowser.close();
    } catch (e) {
        console.error(e)
    }
})();

Answer 1

两个问题:

1. 页面动态加载数据，所以在查询之前应该waitForSelector。
2. .entity-filed 应该是 .filed。

const puppeteer = require("puppeteer"); // ^19.0.0

const url = "<your URL>";

let browser;
(async () => {
  browser = await puppeteer.launch();
  const [page] = await browser.pages();
  const $ = (...args) => page.waitForSelector(...args);
  const text = async (...args) =>
    (await $(...args)).evaluate(el => el.textContent.trim());
  await page.goto(url, {waitUntil: "domcontentloaded"});
  const info = {
    secTableEN: await text(".table td.entity-name"),
    secTableFiled: await text(".table td.filed"),
    secTableLink: await text(".table td.filetype"),
  };
  console.log(info);
})()
  .catch(err => console.error(err))
  .finally(() => browser?.close());

顺便说一句，我不会使用{timeout: 0}。如果页面在一分钟左右后仍未加载，则说明出了点问题，您应该报告错误，而不是永远挂起脚本。

---

另一种方法是避开 DOM 并使用您感兴趣的有效负载简单地拦截 API 响应:

// ... same boilerplate as above ...
browser = await puppeteer.launch();
const [page] = await browser.pages();
const resP = page.waitForResponse(res =>
  res.url() === "https://efts.sec.gov/LATEST/search-index"
);
await page.goto(url, {waitUntil: "domcontentloaded"});
const res = await resP;
const data = JSON.parse(await res.text());
const hit = data.hits.hits[0]._source;
const info = {
  secTableEN: hit.display_names[0],
  secTableFiled: hit.file_date,
  secTableLink: hit.file_type // slightly different output than from the DOM
};
console.log(info);
// ...