python - 使用 Beautifulsoup 获取正文后清理文本字符串

Question

我正在尝试从各个网页上的文章中获取文本并将它们编写为干净的文本文档。我不想要所有可见的文本，因为这通常包括网页一侧不相关的链接。我正在使用 Beautifulsoup 从页面中提取信息。但是，额外的链接不仅出现在页面侧面，有时还出现在正文中间和文章底部，有时也会出现在最终产品中。

有谁知道如何处理转换为实际上不属于真实文章文本一部分的文本的额外链接的问题？

#Some of the imports are for other portions of the code not shown here.
#I'm new to Python and am bad at remembering which library has which functions.
import os
import sys
import urllib2
import webbrowser
from bs4 import BeautifulSoup
from os import path
from cookielib import CookieJar

#I made an opener to deal with proxies and put *** instead of my information
#cookielib helps me get articles from nytimes
proxy = urllib2.ProxyHandler({'http': '***' % '***'})
auth = urllib2.HTTPBasicAuthHandler()
cj = CookieJar()
opener = urllib2.build_opener(proxy, auth, urllib2.HTTPHandler, urllib2.HTTPCookieProcessor(cj))
urllib2.install_opener(opener)

#Uses url input as a string to upen a webpage and and pulls out all the information.
def baumeister(url):
    req = urllib2.Request(url)
    opened = urllib2.urlopen(req)
    html_doc = opened.read()
    soup = BeautifulSoup(html_doc)
    return soup

#Gets the body text from that html information.
def substanz(url):
    soup = baumeister(url)
    body = soup.find_all("p") #This is where I have tried to fix the problem and failed
    result = ""    
    for e in body:
        i = e.getText().replace("\t", "").replace("  ", " ").strip().encode(errors="ignore")
        result += i + "\r\n\r\n"
    return result

我用来测试以我想要的方式清洁的物质的一篇文章是:

http://blogs.hbr.org/2014/06/do-you-really-want-to-be-yourself-at-work/

我正在尝试使用来自不同站点的更多文章进行测试。所以我试图清理 substanz 的结果(结果是一个大字符串)。我遇到的问题是这篇文章:

http://www.cnbc.com/id/101790001?__source=yahoo%7Cfinance%7Cheadline%7Cheadline%7Cstory&par=yahoo&doc=101790001%7CThink%20college%20is%20expensiv# .

我刚刚使用了 print substanz('url') 来查看结果。通过 cnbc 的文章，我得到了一些额外的链接，这些链接变成了文本，但这些链接实际上并不是文章的一部分。而在《哈佛商业评论》文章中，一切都很顺利，因为包含的链接是实际文本的一部分。

我不会在此处附加每篇文章的完整结果以供查看，因为它们每篇都有一整页的文本长度。

如果您完全尝试我在上面发布的代码，则打开器将无法工作，因此请使用您喜欢的任何打开器来访问网站。我必须在工作中访问某个代理，所以这就是适合我的格式。

最后一点，我使用的是 python 3.4，并且正在 ipython 笔记本中编写代码。

Answer 1

import requests
from bs4 import BeautifulSoup
r = requests.get("http://www.cnbc.com/id/101790001?__source=yahoo%7Cfinance%7Cheadline%7Cheadline%7Cstory&par=yahoo&doc=101790001%7CThink%20college%20is%20expensiv#")
soup = BeautifulSoup(r.content)
text =[''.join(s.findAll(text=True))for s in soup.findAll('p')]
print (text)


  ['>> View All Results for ""', 'Enter multiple symbols separated by commas', 'London quotes now available', 'Interest rates on loans to jump', "Because federal student loans are tied to the 10-year Treasury note, CNBC's Sharon Epperson reports borrowers will see the impact of the rise in Treasury yields over the past year.", '  Congratulations, graduates, on your diploma. Now what about that $29,000 student loan debt? ', '  More than 70 percent of graduates will carry student debt into the real world, according to the Institute for College Access and Success. And the average debt is just shy of $30,000.  ', '  But the news will get worse next week when interest rates on student loans are set to rise again.   ', '  Though federal student loan rates are fixed for the life of the loan, these rates reset for new borrowers every July 1, thanks to legislation that ties the rates to the performance of the financial markets.  ', '  The interest rate on federal Stafford loans will go from its current fixed rate of just under 4 percent to 4.66 percent for loans that are distributed between July 1 and June 30, 2015.  ', ' Read MoreStudent loan problem an easy fix: Sen. Warren ', '  For graduate students, the rate on Stafford loans will rise from just over 5 percent  to 6.21 percent.  ', '  Direct PLUS Loans for graduates and parents are still the most expensive, with rates rising to 7.21 percent.', 'Which college major pays off most?', "CNBC's Sharon Epperson reports majoring in engineering is the most lucrative. ", "  The increase in monthly federal student loan payments can add up quickly, but shouldn't be too burdensome for most students. For every $10,000 in loans, new borrowers will pay about $4 more a month based on a 10-year repayment period.   ", " Read MoreWhy millennial women don't save for retirement ", '  Still, experts warn that this is only just the beginning.  ', '  "Federal student loan rates will continue to increase in the next few years and will likely hit the maximum rate caps which are as high as 10.5 percent for some loans," said Mark Kantrowitz, senior vice president and publisher of Edvisors.com.  ', '  For sophomore student Samantha Cook, the decision to go to George Washington University was a big one financially. She says she had doubts about it.  ', '  "My parents wanted to assure me that no matter what I picked, we\'d find a way to make it work," Cook said. Like most families, Cook and her parents are making it work by combining their household savings, scholarships and grants—and student loans.    ', ' Read MoreCramer: Offset high cost of higher education ', '  Despite rising tuition and borrowing costs, the Cook family decided against Samantha transferring to an in-state university.  ', '  Despite the debt load she is taking on, she said, "the value of a GW degree for me at least would be more valuable when looking for jobs later on." ', " —By CNBC's Sharon Epperson ", 'Hosting a yard sale may not be the most profitable way to get rid of your old junk.', 'Many Americans with debit cards tied to their checking accounts are still confused about how these programs work. ', "Here's how to avoid these deadly sins if you're contemplating or already in a divorce.", "The IRS offers a lot of help for students. Problem is, the educational tax breaks and how they work together -- or don't -- are confusing.", 'Get the best of CNBC in your inbox', 'Tips for home buyers that will help you find the right home for your bank account.', 'Complaints about movers are down. How to find the right one—and save.', "Forget bathing suit season. Why it's really time to join the gym. ", 'Drivers might see lower gas prices this year, but smart shopping tactics could help them save even more.', 'Data is a real-time snapshot *Data is delayed at least 15 minutesGlobal Business and Financial News, Stock Quotes, and Market Data and Analysis', '© 2014 CNBC LLC.  All Rights Reserved.', 'A Division of NBCUniversal']

从链接中的网站获取主文章中的文本。

import requests
from bs4 import BeautifulSoup
r = requests.get("http://www.cnbc.com/id/101790001?__source=yahoo%7Cfinance%7Cheadline%7Cheadline%7Cstory&par=yahoo&doc=101790001%7CThink%20college%20is%20expensiv#")
soup = BeautifulSoup(r.content)
text =[''.join(s.findAll(text=True)) for s in soup.findAll("div", {"class":"group"})]
print (text)
['\n  Congratulations, graduates, on your diploma. Now what about that $29,000 student loan debt? \n  More than 70 percent of graduates will carry student debt into the real world, according to the Institute for College Access and Success. And the average debt is just shy of $30,000.  \n  But the news will get worse next week when interest rates on student loans are set to rise again.   \n  Though federal student loan rates are fixed for the life of the loan, these rates reset for new borrowers every July 1, thanks to legislation that ties the rates to the performance of the financial markets.  \n  The interest rate on federal Stafford loans will go from its current fixed rate of just under 4 percent to 4.66 percent for loans that are distributed between July 1 and June 30, 2015.  \n Read MoreStudent loan problem an easy fix: Sen. Warren \n  For graduate students, the rate on Stafford loans will rise from just over 5 percent  to 6.21 percent.  \n  Direct PLUS Loans for graduates and parents are still the most expensive, with rates rising to 7.21 percent.\n', '\n  The increase in monthly federal student loan payments can add up quickly, but shouldn\'t be too burdensome for most students. For every $10,000 in loans, new borrowers will pay about $4 more a month based on a 10-year repayment period.   \n Read MoreWhy millennial women don\'t save for retirement \n  Still, experts warn that this is only just the beginning.  \n  "Federal student loan rates will continue to increase in the next few years and will likely hit the maximum rate caps which are as high as 10.5 percent for some loans," said Mark Kantrowitz, senior vice president and publisher of Edvisors.com.  \n  For sophomore student Samantha Cook, the decision to go to George Washington University was a big one financially. She says she had doubts about it.  \n  "My parents wanted to assure me that no matter what I picked, we\'d find a way to make it work," Cook said. Like most families, Cook and her parents are making it work by combining their household savings, scholarships and grants—and student loans.    \n Read MoreCramer: Offset high cost of higher education \n  Despite rising tuition and borrowing costs, the Cook family decided against Samantha transferring to an in-state university.  \n  Despite the debt load she is taking on, she said, "the value of a GW degree for me at least would be more valuable when looking for jobs later on." \n —By CNBC\'s Sharon Epperson \n']