html - Beautifulsoup 解析特定标签下的数据

Question

现在我正在用这段代码解析一个网页:

boards = soup(itemprop="name")
prices = soup("span", { "class" : "price-currency" })

for board, price in zip(boards, prices):
    print(board.text.strip(), price.next_sibling)

它会像这样打印电路板和价格:

SURFBOARD RACK free delivery to your door 120.00
Huge Beginner Surfboard Sale! Kids & Adult Softboards all 1/2 Price!! 90.00
Mega Softboard Clearance Sale! Beginner Foam SurfBoards 1/2 Price! 90.00
Surfboard 6'2" Simon Anderson Spudnick 360.00
Surfboard Cover, Surfboard Bags, Cheap Single Surf Board Bags 50.00

我正在解析的网页分为 3 个部分:赞助商链接、热门广告和最近的广告。我正在打印所有这 3 个部分的数据，但只需要来自最近的广告部分的数据，该部分具有以下 html:

<div class="module__body ad-listing">

如何指定我只希望从该部分下方打印电路板和价格？

页码:https://www.gumtree.com.au/s-surfing/mona-vale-sydney/surfboard/k0c18568l3003999r10?fromSearchBox=true

Answer 1

您可能讨厌这个答案。当我看到像那样复杂的 HTML 时，我倾向于使用 lxml 模块，因为我可以使用 xpath 表达式。

在这种情况下，第一个 xpath 会在您想要的 HTML 中找到 li 元素的集合。该循环使用两个 xpath 表达式，一个用于在 li 元素中查找诸如“Quicksale 6'4 Dylan Surfboard RX5”之类的内容，另一个用于查找价格信息的文本集合在同一元素内。第 12 项似乎编码不同；我还没有调查过。

>>> import requests
>>> from lxml import etree
>>> page = requests.get('https://www.gumtree.com.au/s-surfing/mona-vale-sydney/surfboard/k0c18568l3003999r10?fromSearchBox=true').text
>>> parser = etree.HTMLParser()
>>> tree = etree.fromstring(page, parser=parser)
>>> recents = tree.xpath('.//div[@class="module__body ad-listing"]/ul/li')
>>> for i, recent in enumerate(recents):
...     try:
...         i, recent.xpath('.//span[@itemprop="name"]/text()')[0].strip()
...     except:
...         '-------------> item', i, 'failed'
...         continue
...     one_span = first_recent.xpath('.//span[@class="j-original-price"]')[0]
...     ' '.join([_.strip() for _ in list(one_span.itertext()) if _.strip()])
... 
(0, "Quicksale 6'4 Dylan Surfboard RX5")
'$ 450.00 Negotiable'
(1, 'DHD 5\'9 "Switchblade" Surfboard')
'$ 450.00 Negotiable'
(2, '6ft Modern Surfboards - Highline')
'$ 450.00 Negotiable'
(3, "5'11 Channel Island T-Low surfboard")
'$ 450.00 Negotiable'
(4, 'Chill Rare Bird Surfboard 5"8')
'$ 450.00 Negotiable'
(5, 'Vintage surfboard')
'$ 450.00 Negotiable'
(6, "5'7 Annesley Blonde model")
'$ 450.00 Negotiable'
(7, 'McCoy single fin surfboard')
'$ 450.00 Negotiable'
(8, 'Sculpt surfboard')
'$ 450.00 Negotiable'
(9, '8\'1" longboard surfboard travel cover')
'$ 450.00 Negotiable'
(10, 'Longboard Surfboard')
'$ 450.00 Negotiable'
(11, "5'10 Custom Chaos Surfboard")
'$ 450.00 Negotiable'
('-------------> item', 12, 'failed')
(13, "6'0 JS lowdown")
'$ 450.00 Negotiable'
(14, 'Mega Softboard Clearance Sale! Beginner Foam SurfBoards 1/2 Price!')
'$ 450.00 Negotiable'
(15, 'Surfboard')
'$ 450.00 Negotiable'
(16, 'Surfboard 5\'10" 30 lt')
'$ 450.00 Negotiable'
(17, 'Christenson Super Sport Surfboard')
'$ 450.00 Negotiable'
(18, 'TOMO Firewire V4 Surfboard')
'$ 450.00 Negotiable'
(19, "Surfboard 6'6 baked bean")
'$ 450.00 Negotiable'
(20, 'foam surfboards')
'$ 450.00 Negotiable'
(21, 'Channel Islands surfboard')
'$ 450.00 Negotiable'
(22, 'Channel Islands Surfboard')
'$ 450.00 Negotiable'
(23, 'JS surfboard')
'$ 450.00 Negotiable'
(24, 'CLASSIC RETRO SURF FACTORY MINI MAL')
'$ 450.00 Negotiable'
(25, 'Surfboard JS')
'$ 450.00 Negotiable'