python - 如何使用正则表达式从 HTML 解析数据？

Question

我想使用正则表达式从 Indeed(仅限第一页)抓取职位名称、地点和职位描述，并将结果存储到数据框架中。这是链接:https://www.indeed.com/jobs?q=data+scientist&l=California

我已经使用 BeautifulSoup 完成了任务，它们工作得很好:

from urllib.request import urlopen
from bs4 import BeautifulSoup as BS
import pandas as pd

url = 'https://www.indeed.com/jobs?q=data+scientist&l=California'
htmlfile = urlopen(url)
soup = BS(htmlfile,'html.parser')

companies = []
locations = []
summaries = []

company = soup.findAll('span', attrs={'class':'company'})
for c in company:
    companies.append(c.text.replace("\n",""))
  
location = soup.findAll(class_ = 'location accessible-contrast-color-location')
for l in location:
    locations.append(l.text)
        
summary = soup.findAll('div', attrs={'class':'summary'})
for s in summary:
    summaries.append(s.text.replace("\n",""))

jobs_df = pd.DataFrame({'Company':companies, 'Location':locations, 'Summary':summaries})
jobs_df

BS 结果:

    Company Location    Summary
0   Cisco Careers   San Jose, CA    Work on massive structured, unstru...
1   AllyO   Palo Alto, CA   Extensive knowledge of scientific ...
2   Driven Brands   Benicia, CA 94510   Develop scalable statistical, mach...
3   eBay Inc.   San Jose, CA    These problems require deep analys...
4   Disney Streaming Services   San Francisco, CA   Deep knowledge of machine learning...
5   Trimark Associates, Inc.    Sacramento, CA  The primary focus is in applying d...

但是当我尝试在正则表达式中使用相同的标签时，它失败了。

import urllib.request, urllib.parse, urllib.error
import re
import pandas as pd

url = 'https://www.indeed.com/jobs?q=data+scientist&l=California'
text = urllib.request.urlopen(url).read().decode()

companies = []
locations = []
summaries = []

company = re.findall('<span class="company">(.*?)</span>', text)
for c in company:
    companies.append(str(c))
  
location = re.findall('<div class="location accessible-contrast-color-location">(.*?)</div>', text)
for l in location:
    locations.append(str(l))
        
summary = re.findall('<div class="summary">(.*?)</div>', text)
for s in summary:
    summaries.append(str(s))

print(companies)
print(locations)
print(summaries)

出现错误，指出列表的长度不匹配，因此我检查了各个列表。结果发现无法获取内容。我从上面得到了什么:

[]
['Palo Alto, CA', 'Sunnyvale, CA', 'San Francisco, CA', 'South San Francisco, CA 94080', 'Pleasanton, CA 94566', 'Aliso Viejo, CA', 'Sacramento, CA', 'Benicia, CA 94510', 'San Bruno, CA']
[]

我做错了什么？

Answer 1

. 匹配除换行符之外的任何字符。在 HTML 代码中，也有换行符。
因此，您需要在 re.findall 中使用 re.DOTALL 作为标志选项，如下所示:

company = re.findall('<span class="company">(.*?)</span>', text, flags=re.DOTALL)

从上面的代码中，您不会仅获得名称。相反，您将获得您所选择的 span 元素的所有后代。因此，您只需选择您想要的正则表达式部分。

for c in company:
  # selecting only the company name, discarding everything in the anchor tag.
  name = re.findall('<a.*>(.*)</a>', c, flags = re.DOTALL)
  for n in name:
    # doing a little cleanup by removing the newlines and spaces.
    companies.append(str(n.strip()))

print(companies)

输出:

['Driven Brands', 'Southern California Edison', 'Paypal', "Children's Hospital Los Angeles", 'Cisco Careers', 'University of California, Santa Cruz', 'Beyond Limits', 'Shutterfly', 'Walmart', 'Trimark Associates, Inc.']

对于位置和摘要，没有其他 HTML 标记。
仅存在文本。

因此，只有 re.DOTALL 并剥离文本才能完成这项工作。
不需要第二个 for 循环和第二个 findall。