- html - 出于某种原因,IE8 对我的 Sass 文件中继承的 html5 CSS 不友好?
- JMeter 在响应断言中使用 span 标签的问题
- html - 在 :hover and :active? 上具有不同效果的 CSS 动画
- html - 相对于居中的 html 内容固定的 CSS 重复背景?
我知道如何在页面上获取所有可见的纯文本:const text = await page.$eval('*', el => el.innerText);
但我还需要知道每段文本属于页面的哪个元素,我找不到办法做到这一点。
最佳答案
在客户端,您可以使用 TreeWalker 以保留顺序的方式执行此操作。 .这是来自 Web Scraper Testing Ground 的示例内容示例:
const IGNORE = ["style", "script"];
const walker = document.createTreeWalker(document.body, NodeFilter.SHOW_TEXT);
const pairs = [];
let node;
while ((node = walker.nextNode()) !== null) {
const parent = node.parentNode.tagName;
if (IGNORE.includes(parent)) {
continue;
}
const value = node.nodeValue.trim();
if (value.length === 0) {
continue;
}
pairs.push([parent.toLowerCase(), value]);
}
console.log(pairs);
<div id="topbar"></div>
<a href="/" style="text-decoration: none">
<div id="title">WEB SCRAPER TESTING GROUND</div>
<div id="logo"></div>
</a>
<div id="content">
<h1>BLOCKS: Price List </h1>
<div id="caseinfo">In this test, the web scraper needs to scrape a price list organized in a block layout. Specifically, it has to:
<ol>
<li>Extract all the products (their names, descriptions and prices), while skipping advertisements</li>
<li>Scrape discounted products only</li>
<li>Scrape products with red prices only</li>
</ol>
<p>
</p><p>There is a <b>ver</b> parameter (which varies from 1 to 5) to show different table versions (with different product numbers, best price and advertisement positions).</p>
<p>Also there are two tables presented:
</p><ul>
<li><b>Case 1</b> (simple one, with products and prices placed into the same block)
</li><li><b>Case 2</b> (complicated one, with products and prices placed into separate blocks)</li>
</ul>
<p></p>
<p>For testing, you may use the following sample links. The scraper should sufficiently scrape all data from a certain case using the same project:
</p><ul>
<li><a href="/blocks?ver=1">Price list 1</a></li>
<li><a href="/blocks?ver=2">Price list 2</a></li>
<li><a href="/blocks?ver=3">Price list 3</a></li>
<li><a href="/blocks?ver=4">Price list 4</a></li>
<li><a href="/blocks?ver=5">Price list 5</a></li>
</ul>
<p></p>
</div>
<div id="case_blocks">
<h2>Case 1</h2>
<div id="case1">
<div class="prod2"><span style="float: left"><div class="name">Dell Latitude D610-1.73 Laptop Wireless Computer</div>2 GHz Intel Pentium M, 1 GB DDR2 SDRAM, 40 GB, Microsoft Windows XP Professional</span><span style="float: right">$239.95</span></div><div class="prod1"><span style="float: left"><div class="name">Samsung Chromebook (Wi-Fi, 11.6-Inch)</div>1.7 GHz, 2 GB DDR3 SDRAM, 16 GB, Chrome</span><span style="float: right" class="best">$249.00</span><span style="float: right;margin-right:10px" class="best">BEST<br>PRICE!</span></div><div class="ads">ADVERTISEMENT</div><div class="prod2"><span style="float: left"><div class="name">Apple MacBook Pro MD101LL/A 13.3-Inch Laptop (NEWEST VERSION)</div>2.5 GHz Intel Core i5, 4 GB DDR3 SDRAM, 500 GB Serial ATA, Mac OS X v10.7 Lion</span><span style="float: right">$1,099.99</span></div><div class="prod1"><span style="float: left"><div class="name">Acer Aspire AS5750Z-4835 15.6-Inch Laptop (Black)</div>2 GHz Pentium B940, 4 GB SDRAM, 500 GB, Windows 7 Home Premium 64-bit</span><span style="float: right" class="best">$385.72</span><span style="float: right;margin-right:10px" class="best">BEST<br>PRICE!</span></div><div class="ads">ADVERTISEMENT</div><div class="prod2"><span style="float: left"><div class="name">HP Pavilion g7-2010nr 17.3-Inch Laptop (Black)</div>2.3 GHz Core i3-2350M, 6 GB SDRAM, 640 GB, Windows 7 Home Premium 64-bit</span><span style="float: right">$549.99<div class="disc">discount 7%</div></span></div><div class="prod1"><span style="float: left"><div class="name">ASUS A53Z-AS61 15.6-Inch Laptop (Mocha)</div>1.4 GHz A-Series Quad-Core A6-3420M, 4 GB DIMM, 750 GB, Windows 7 Home Premium 64-bit</span><span style="float: right">$399.99</span></div></div>
<h2 style="margin-top: 50px">Case 2</h2>
<div id="case2">
<div class="left"><div class="prod2"><div class="name">Dell Latitude D610-1.73 Laptop Wireless Computer</div>2 GHz Intel Pentium M, 1 GB DDR2 SDRAM, 40 GB, Microsoft Windows XP Professional</div><div class="prod1"><div class="name">Samsung Chromebook (Wi-Fi, 11.6-Inch)</div>1.7 GHz, 2 GB DDR3 SDRAM, 16 GB, Chrome</div><div class="ads">ADVERTISEMENT</div><div class="prod2"><div class="name">Apple MacBook Pro MD101LL/A 13.3-Inch Laptop (NEWEST VERSION)</div>2.5 GHz Intel Core i5, 4 GB DDR3 SDRAM, 500 GB Serial ATA, Mac OS X v10.7 Lion</div><div class="prod1"><div class="name">Acer Aspire AS5750Z-4835 15.6-Inch Laptop (Black)</div>2 GHz Pentium B940, 4 GB SDRAM, 500 GB, Windows 7 Home Premium 64-bit</div></div><div class="right"><div class="price2">$239.95</div><div class="price1 best">$249.00</div><div class="ads"></div><div class="price2">$1,099.99</div><div class="price1 best">$385.72</div></div><div class="ads" style="clear: both">ADVERTISEMENT</div><div class="left"><div class="prod2"><div class="name">HP Pavilion g7-2010nr 17.3-Inch Laptop (Black)</div>2.3 GHz Core i3-2350M, 6 GB SDRAM, 640 GB, Windows 7 Home Premium 64-bit</div><div class="prod1"><div class="name">ASUS A53Z-AS61 15.6-Inch Laptop (Mocha)</div>1.4 GHz A-Series Quad-Core A6-3420M, 4 GB DIMM, 750 GB, Windows 7 Home Premium 64-bit</div></div><div class="right"><div class="price2">$549.99<div class="disc">discount 7%</div></div><div class="price1">$399.99</div></div></div>
</div>
<br><br><br>
</div>
evaluate
根据 Grant Miller 的回答,在 Puppeteer 中调用它:
const pairs = await page.evaluate(() => {
const IGNORE = ["style", "script"];
const NONWHITESPACE_RE = /\S/;
const result = document.evaluate(
"//*[child::text()]",
document,
null,
XPathResult.ORDERED_NODE_SNAPSHOT_TYPE,
null
);
const pairs = [];
for (let i = 0, j = result.snapshotLength; i < j; i++) {
const element = result.snapshotItem(i);
if (IGNORE.includes(element.tagName.toLowerCase())) {
continue;
}
const nodes = [...element.childNodes];
for (const node of nodes) {
if (node.nodeType !== document.TEXT_NODE) {
continue;
}
if (node.nodeValue.search(NONWHITESPACE_RE) === -1) {
continue;
}
pairs.push({
tag: element.tagName.toLowerCase(),
text: node.nodeValue.trim()
});
}
}
return pairs;
});
console.log(pairs);
const IGNORE = ["style", "script"];
const NONWHITESPACE_RE = /\S/;
// get all text nodes in the document
const result = document.evaluate(
// matches any node in the document that has at least one direct
// text node child, including whitespace-only nodes
"//*[child::text()]",
document,
null,
XPathResult.ORDERED_NODE_SNAPSHOT_TYPE,
null
);
// the result doesn't use the JavaScript iterator protocol, so we have
// to manually iterate over the elements
const pairs = [];
for (let i = 0, j = result.snapshotLength; i < j; i++) {
const element = result.snapshotItem(i);
if (IGNORE.includes(element.tagName.toLowerCase())) {
continue;
}
const nodes = [...element.childNodes];
for (const node of nodes) {
if (node.nodeType !== document.TEXT_NODE) {
continue;
}
// filter out whitespace-only nodes
if (node.nodeValue.search(NONWHITESPACE_RE) === -1) {
continue;
}
pairs.push({
tag: element.tagName.toLowerCase(),
// remove the `.trim()` to preserve leading & trailing whitespace
text: node.nodeValue.trim()
});
}
}
console.log(pairs);
<div id="topbar"></div>
<a href="/" style="text-decoration: none">
<div id="title">WEB SCRAPER TESTING GROUND</div>
<div id="logo"></div>
</a>
<div id="content">
<h1>BLOCKS: Price List </h1>
<div id="caseinfo">In this test, the web scraper needs to scrape a price list organized in a block layout. Specifically, it has to:
<ol>
<li>Extract all the products (their names, descriptions and prices), while skipping advertisements</li>
<li>Scrape discounted products only</li>
<li>Scrape products with red prices only</li>
</ol>
<p>
</p><p>There is a <b>ver</b> parameter (which varies from 1 to 5) to show different table versions (with different product numbers, best price and advertisement positions).</p>
<p>Also there are two tables presented:
</p><ul>
<li><b>Case 1</b> (simple one, with products and prices placed into the same block)
</li><li><b>Case 2</b> (complicated one, with products and prices placed into separate blocks)</li>
</ul>
<p></p>
<p>For testing, you may use the following sample links. The scraper should sufficiently scrape all data from a certain case using the same project:
</p><ul>
<li><a href="/blocks?ver=1">Price list 1</a></li>
<li><a href="/blocks?ver=2">Price list 2</a></li>
<li><a href="/blocks?ver=3">Price list 3</a></li>
<li><a href="/blocks?ver=4">Price list 4</a></li>
<li><a href="/blocks?ver=5">Price list 5</a></li>
</ul>
<p></p>
</div>
<div id="case_blocks">
<h2>Case 1</h2>
<div id="case1">
<div class="prod2"><span style="float: left"><div class="name">Dell Latitude D610-1.73 Laptop Wireless Computer</div>2 GHz Intel Pentium M, 1 GB DDR2 SDRAM, 40 GB, Microsoft Windows XP Professional</span><span style="float: right">$239.95</span></div><div class="prod1"><span style="float: left"><div class="name">Samsung Chromebook (Wi-Fi, 11.6-Inch)</div>1.7 GHz, 2 GB DDR3 SDRAM, 16 GB, Chrome</span><span style="float: right" class="best">$249.00</span><span style="float: right;margin-right:10px" class="best">BEST<br>PRICE!</span></div><div class="ads">ADVERTISEMENT</div><div class="prod2"><span style="float: left"><div class="name">Apple MacBook Pro MD101LL/A 13.3-Inch Laptop (NEWEST VERSION)</div>2.5 GHz Intel Core i5, 4 GB DDR3 SDRAM, 500 GB Serial ATA, Mac OS X v10.7 Lion</span><span style="float: right">$1,099.99</span></div><div class="prod1"><span style="float: left"><div class="name">Acer Aspire AS5750Z-4835 15.6-Inch Laptop (Black)</div>2 GHz Pentium B940, 4 GB SDRAM, 500 GB, Windows 7 Home Premium 64-bit</span><span style="float: right" class="best">$385.72</span><span style="float: right;margin-right:10px" class="best">BEST<br>PRICE!</span></div><div class="ads">ADVERTISEMENT</div><div class="prod2"><span style="float: left"><div class="name">HP Pavilion g7-2010nr 17.3-Inch Laptop (Black)</div>2.3 GHz Core i3-2350M, 6 GB SDRAM, 640 GB, Windows 7 Home Premium 64-bit</span><span style="float: right">$549.99<div class="disc">discount 7%</div></span></div><div class="prod1"><span style="float: left"><div class="name">ASUS A53Z-AS61 15.6-Inch Laptop (Mocha)</div>1.4 GHz A-Series Quad-Core A6-3420M, 4 GB DIMM, 750 GB, Windows 7 Home Premium 64-bit</span><span style="float: right">$399.99</span></div></div>
<h2 style="margin-top: 50px">Case 2</h2>
<div id="case2">
<div class="left"><div class="prod2"><div class="name">Dell Latitude D610-1.73 Laptop Wireless Computer</div>2 GHz Intel Pentium M, 1 GB DDR2 SDRAM, 40 GB, Microsoft Windows XP Professional</div><div class="prod1"><div class="name">Samsung Chromebook (Wi-Fi, 11.6-Inch)</div>1.7 GHz, 2 GB DDR3 SDRAM, 16 GB, Chrome</div><div class="ads">ADVERTISEMENT</div><div class="prod2"><div class="name">Apple MacBook Pro MD101LL/A 13.3-Inch Laptop (NEWEST VERSION)</div>2.5 GHz Intel Core i5, 4 GB DDR3 SDRAM, 500 GB Serial ATA, Mac OS X v10.7 Lion</div><div class="prod1"><div class="name">Acer Aspire AS5750Z-4835 15.6-Inch Laptop (Black)</div>2 GHz Pentium B940, 4 GB SDRAM, 500 GB, Windows 7 Home Premium 64-bit</div></div><div class="right"><div class="price2">$239.95</div><div class="price1 best">$249.00</div><div class="ads"></div><div class="price2">$1,099.99</div><div class="price1 best">$385.72</div></div><div class="ads" style="clear: both">ADVERTISEMENT</div><div class="left"><div class="prod2"><div class="name">HP Pavilion g7-2010nr 17.3-Inch Laptop (Black)</div>2.3 GHz Core i3-2350M, 6 GB SDRAM, 640 GB, Windows 7 Home Premium 64-bit</div><div class="prod1"><div class="name">ASUS A53Z-AS61 15.6-Inch Laptop (Mocha)</div>1.4 GHz A-Series Quad-Core A6-3420M, 4 GB DIMM, 750 GB, Windows 7 Home Premium 64-bit</div></div><div class="right"><div class="price2">$549.99<div class="disc">discount 7%</div></div><div class="price1">$399.99</div></div></div>
</div>
<br><br><br>
</div>
关于puppeteer - 获取所有可见的纯文本并找出每段文本属于哪个 HTML 标签或 DOM 元素,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/52269201/
所以我使用一个带有整个 block 的标签作为链接,它是一个产品展示,所以你点击它会转到产品页面。现在我创建了一个标签作为链接到购物车页面的按钮。所以我让它工作,但是当我点击购物车按钮时,两个页面都会
根据 Web 标准,创建带有标题 1 的链接的正确代码是什么? 是吗 stackoverflow 或 stackoverflow 谢谢 最佳答案 根据网络标准,您不能将 block 元素放入内
在Java中它是这样写的..当我移植这段代码时...意识到没有这样的东西 break 和continue . 我知道这些命令没有包含在内,因为在使用带有命令的 goto 时必须有一种更简洁的方法来执
我们有一个相当标准的发布过程,使用 Visual Source Safe 在发布之前标记构建。这允许我们在出现任何问题时从该标签中获取,并在需要更改时使用它进行分支。 我们有几个不同的项目,并且总是使
我必须创建一个搜索内容,其中包含搜索框、标题和段落描述。默认情况下,描述被禁用,当我输入一些与描述文本匹配的文本时,描述段落标签应该打开。一些匹配的演示是这样的: [ fiddle ][1] 但默认情
我一直在阅读有关 的文档标签,我似乎无法理解它与简单地使用 有何不同那是 display: none; 文档:template tag 例子 对比 例子
我需要一个脚本来复制当开关按钮打开时标记,当开关按钮关闭时删除标记。我需要一个简单的方法。这是开关按钮: 我试过这个: var change
JSF 是一个 MVC 框架,但我很困惑为什么我们已经有了这么多 HTML 标签还需要 JSF 标签。毫无疑问,JSF 简化了很多事情。我想进一步了解 JSF 中的模型 View 和 Controll
我在这个 website 上看到了那些 html 代码: Homepa
我添加了 photoswipe 插件,可以使用 搜索我的所有照片。标签,如果点击,照片就会变成全屏。我让它工作了,但现在我的导航栏(有 标签)在点击时会触发 photoswipe 插件。 在 ph
标签
我正在尝试截断显示自 的文本标签,但它不工作。我将样式应用于其他标签样式并且它确实有效(我看到的示例中没有一个使用 标签)。我想知道是否有人可以向我解释为什么会这样(我不是最擅长 HTML/CSS
HTML 是这样的: Menu 1 Menu 2 Sub menu 2
我可以更改 TextInputLayout 的位置 float 标签(底部 float 标签)吗?我需要为波纹管 float 标签设置正确的位置。 最佳答案 我解决了我的问题,这是我的 xml:
我的代码是 printMsg : function(data) { $("#message").html(data.bodyText); ... } 这里 data.body
我是 Scrapy 和 Xpath 的初学者,我正在寻找解析具有以下结构的网站 cat1 value1 value2
关闭。这个问题不符合Stack Overflow guidelines .它目前不接受答案。 要求提供代码的问题必须表现出对所解决问题的最低限度理解。包括尝试过的解决方案、为什么它们不起作用,以及
我必须从 xml 中解析数据。这是我的 xml- 或者它的 url 是:http://mobileecommerce.site247365.com/admin/catdata.xml News f
如何创建应该允许多行数据的标记。不要说使用textarea标签。我知道,但我只想 标记因为标签具有 value 属性。所以当我从 xml 文件获取值时,我应该使用 jquery 语法动态获取.. 最佳
我有一个页面使用我定义的某些样式。 在同一页面上,我刚刚导入了一个使用自己样式的外部 jQuery 插件,例如,包括 。被我自己覆盖的标签样式。 如何确保我的样式表中的样式不会覆盖 jQuery 插件
关闭。这个问题是opinion-based .它目前不接受答案。 想要改进这个问题? 更新问题,以便 editing this post 可以用事实和引用来回答它. 关闭 8 年前。 Improve
我是一名优秀的程序员,十分优秀!