python - 如何使用 BeautifulSoup 从网页中获取整个正文文本？

Question

我想从自然语言处理项目的医疗文档网页中获取一些文本，但在使用 BeautifulSoup 提取必要信息时遇到问题。我正在查看的网站可以在以下地址找到:https://www.mtsamples.com/site/pages/sample.asp?Type=24-Gastroenterology&Sample=2332-Abdominal%20Abscess%20I&D

我想做的是从此页面抓取整个文本正文，然后用光标执行此操作，只需应用复制/粘贴即可获得我感兴趣的适当文本:

Sample Type / Medical Specialty: Gastroenterology
Sample Name: Abdominal Abscess I&D
Description: Incision and drainage (I&D) of abdominal abscess, excisional debridement of nonviable and viable skin, subcutaneous tissue and muscle, then removal of foreign body.
(Medical Transcription Sample Report)
PREOPERATIVE DIAGNOSIS: Abdominal wall abscess.

... (body text) ...

The finished wound size was 9.0 x 5.3 x 5.2 cm in size. Patient tolerated the procedure well. Dressing was applied, and he was taken to recovery room in stable condition. 

但是，我想使用 BeautifulSoup 来实现这一点，因为我想执行循环以从同一网站获取多个医疗文档。

import requests  
r = requests.get('https://www.mtsamples.com/site/pages/sample.asp?Type=24-Gastroenterology&Sample=2332-Abdominal%20Abscess%20I&D')

from bs4 import BeautifulSoup  
soup = BeautifulSoup(r.text, 'html.parser')  
results = soup.find_all('div', attrs={'id':'sampletext'})

# Here I am able to specify the <h1> tag to get 'Sample Type / Medical Specialty' as well as 'Sample Name' text fields

record.find('h1').text.replace('\n', ' ')

但是，我无法为其余文本(即描述、术前诊断、术后诊断、程序等)复制此内容，因为没有唯一标签来标识这些文本字段

如果有人熟悉使用 BeautifulSoup 进行网页抓取的概念，我将不胜感激!我的目标再次是从网页中获取全文，我最终希望将其添加到 Pandas Dataframe 中。谢谢!

Answer 1

好吧，我花了一段时间，但是没有一种简单的方法可以提取可用的文本，除非您手动迭代所有元素:

import requests
import re
from bs4 import BeautifulSoup, Tag, NavigableString, Comment

url = 'https://www.mtsamples.com/site/pages/sample.asp?Type=24-Gastroenterology&Sample=2332-Abdominal%20Abscess%20I&D'
res = requests.get(url)
res.raise_for_status()
html = res.text
soup = BeautifulSoup(html, 'html.parser')

到目前为止没有什么特别的。

title_el = soup.find('h1')
page_title = title_el.text.strip()
first_hr = title_el.find_next_sibling('hr')

description_title = title_el.find_next_sibling('b', text=re.compile('description', flags=re.I))
description_text_parts = []
for s in description_title.next_siblings:
    if s is first_hr:
        break
    if isinstance(s, Tag):
        description_text_parts.append(s.text.strip())
    elif isinstance(s, NavigableString):
        description_text_parts.append(str(s).strip())
description_text = '\n'.join(p for p in description_text_parts if p.strip())

这里我们得到page_title来自<h1>

'Sample Type / Medical Specialty:  Gastroenterology\nSample Name: Abdominal Abscess I&D'

和description在我们看到文本 Description: 后遍历元素.

'Incision and drainage (I&D) of abdominal abscess, excisional debridement of nonviable and viable skin, subcutaneous tissue and muscle, then removal of foreign body.\n(Medical Transcription Sample Report)'

现在，所有标题都放置在水平线下:

# titles are all bold and uppercase
titles = [b for b in first_hr.find_next_siblings('b') if b.text.strip().isupper()]

我们找到标题之间的文本并将其分配给我们之前看到的标题

docs = []
for t in titles:
    text_parts = []
    for s in t.next_siblings:
        # go until next title
        if s in titles:
            break
        if isinstance(s, Comment):
            continue
        if isinstance(s, Tag):
            if s.name == 'div':
                break
            text_parts.append(s.text.strip())
        elif isinstance(s, NavigableString):
            text_parts.append(str(s).strip())
    text = '\n'.join(p for p in text_parts if p.strip())
    docs.append({
        'title': t.text.strip(),
        'text': text
    })

打印文档给出:

[
{'title': 'PREOPERATIVE DIAGNOSIS:', 'text': 'Abdominal wall abscess.'}, 
{'title': 'POSTOPERATIVE DIAGNOSIS:', 'text': 'Abdominal wall abscess.'}, 
{'title': 'PROCEDURE:', 'text': 'Incision and drainage (I&D) of abdominal abscess, excisional debridement of nonviable and viable skin, subcutaneous tissue and muscle, then removal of foreign body.'}, 
{'title': 'ANESTHESIA:', 'text': 'LMA.'}, 
{'title': 'INDICATIONS:', 'text': 'Patient is a pleasant 60-year-old gentleman, who initially had a sigmoid colectomy for diverticular abscess, subsequently had a dehiscence with evisceration.  Came in approximately 36 hours ago with pain across his lower abdomen.  CT scan demonstrated presence of an abscess beneath the incision.  I recommended to the patient he undergo the above-named procedure.  Procedure, purpose, risks, expected benefits, potential complications, alternatives forms of therapy were discussed with him, and he was agreeable to surgery.'}, 
{'title': 'FINDINGS:', 'text': 'The patient was found to have an abscess that went down to the level of the fascia.  The anterior layer of the fascia was fibrinous and some portions necrotic.  This was excisionally debrided using the Bovie cautery, and there were multiple pieces of suture within the wound and these were removed as well.'},
{'title': 'TECHNIQUE:', 'text': 'Patient was identified, then taken into the operating room, where after induction of appropriate anesthesia, his abdomen was prepped with Betadine solution and draped in a sterile fashion.  The wound opening where it was draining was explored using a curette.  The extent of the wound marked with a marking pen and using the Bovie cautery, the abscess was opened and drained.  I then noted that there was a significant amount of undermining.  These margins were marked with a marking pen, excised with Bovie cautery; the curette was used to remove the necrotic fascia.  The wound was irrigated; cultures sent prior to irrigation and after achievement of excellent hemostasis, the wound was packed with antibiotic-soaked gauze.  A dressing was applied.  The finished wound size was 9.0 x 5.3 x 5.2 cm in size.  Patient tolerated the procedure well.  Dressing was applied, and he was taken to recovery room in stable condition.'}
]