php - 使用 PHP 在任意 HTML 中查找重要文本？

Question

我有一些随机的 HTML 布局，其中包含我想提取的重要文本。我不能只使用 strip_tags()，因为那样会在侧边栏/页脚/页眉/等处留下一堆额外的垃圾。

我找到了一个 method built in Python我想知道 PHP 中是否有类似的东西。

The concept is rather simple: use information about the density of text vs. HTML code to work out if a line of text is worth outputting. (This isn’t a novel idea, but it works!) The basic process works as follows:

1. Parse the HTML code and keep track of the number of bytes processed.
2. Store the text output on a per-line, or per-paragraph basis.
3. Associate with each text line the number of bytes of HTML required to describe it.
4. Compute the text density of each line by calculating the ratio of text t> o bytes.
5. Then decide if the line is part of the content by using a neural network.

You can get pretty good results just by checking if the line’s density is above a fixed threshold (or the average), but the system makes fewer mistakes if you use machine learning - not to mention that it’s easier to implement!

更新:我发起了一项赏金事件，征求可以从随机 HTML 模板中提取主要内容的答案。由于我无法共享我将使用的文档 - 只需选择任何随机博客站点并尝试从布局中提取正文。请记住，页眉、侧边栏和页脚也可能包含文本。有关想法，请参阅上面的链接。

Answer 1

• phpQuery is a server-side, chainable, CSS3 selector driven Document Object Model (DOM) API based on jQuery JavaScript Library.

---

更新 2

• DEMO: http://so.lucafilosofi.com/find-important-text-in-arbitrary-html-using-php/
• tested on a casual blogs list taken from Technorati Top 100 and Best Blogs of 2010

1. 许多博客使用 CMS ;
2. 博客的 html 结构几乎一直都是一样的。
3. 避免使用常见的选择器，例如 #sidebar、#header、#footer、#comments 等。
4. 通过标签名称script, iframe避免任何小部件>
5. 清晰的众所周知的内容，例如:
   1. /\d+\scomment(?:[s])/im
   2. /(阅读其余部分|阅读更多).*/im
   3. /(?:.*(?:by|post|submitt?)(?:ed)?.*\s(at|am|pm))/im
   4. /[^a-z0-9]+/im

---

搜索众所周知的类和 ID:

• typepad.com .entry-content
• wordpress.org .post-entry .entry .post
• movabletype.com .post
• blogger.com .post-body .entry-content
• drupal.com .content
• tumblr.com .post
• squarespace.com .journal-entry-text
• expressionengine.com .entry

• gawker.com .post-body

• 引用: The blog platforms of choice among the top 100 blogs

---

$selectors = array('.post-body','.post','.journal-entry-text','.entry-content','.content');
$doc = phpQuery::newDocumentFile('http://blog.com')->find($selectors)->children('p,div');

---

基于如下所示的常见 html 结构进行搜索:

<div>
<h1|h2|h3|h4|a />
<p|div />
</div>

---

$doc = phpQuery::newDocumentFile('http://blog.com')->find('h1,h2,h3,h4')->parent()->children('p,div');