Python:如何使用 LXML/Requests 遍历 HTML 元素对象？

Question

我正在尝试使用 LXML 和 Requests 从网站创建数据表。我需要标签内的文本和标签内包含的文本。这是 HTML:

<div class="houses">
    <input type="hidden" class="houseNumber" value="107">
    <input type="hidden" class="houseState" value="MT">
    <input type="hidden" class="houseStatus" value="Occupied">
<div class="houseInfo">
    <div class="houseCity">Helena</div>
    <div class="houseArea">Helena Valley</div>
</div>
</div>
<div class="houses">
    <input type="hidden" class="houseNumber" value="237">
    <input type="hidden" class="houseState" value="MT">
    <input type="hidden" class="houseStatus" value="Occupied">
<div class="houseInfo">
    <div class="houseCity">East Helena</div>
    <div class="houseArea">Helena Valley</div>
</div>
</div>
<div class="houses">
    <input type="hidden" class="houseNumber" value="104">
    <input type="hidden" class="houseState" value="MT">
    <input type="hidden" class="houseStatus" value="Vacant">
<div class="houseInfo">
    <div class="houseCity">Helena</div>
    <div class="houseArea">Helena Valley</div>
</div>
</div>

据此，我想创建一个这样的表:

['107', 'MT', 'Occupied', 'Helena', 'Helena Valley']
['237', 'MT', 'Occupied', 'East Helena', 'Helena Valley']
['104', 'MT', 'Vacant', 'Helena', 'Helena Valley']

使用 Requests 和 LXML，我尝试遍历 div class="houses" 以获得我需要的内容，但每次我尝试打印这些值时，它都会打印以下内容:

['107', '237', '104']
['MT', 'MT', 'MT']
['Occupied', 'Occupied', 'Vacant']
['Helena', 'East Helena', 'Helena']
['Helena Valley', 'Helena Valley', 'Helena Valley']
['107', '237', '104']
['MT', 'MT', 'MT']
['Occupied', 'Occupied', 'Vacant']
['Helena', 'East Helena', 'Helena']
['Helena Valley', 'Helena Valley', 'Helena Valley']
['107', '237', '104']
['MT', 'MT', 'MT']
['Occupied', 'Occupied', 'Vacant']
['Helena', 'East Helena', 'Helena']
['Helena Valley', 'Helena Valley', 'Helena Valley']

这是我的部分代码:

link = "example.com"
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}
response = requests.get(link, headers=headers, allow_redirects=False) 
sourceCode = response.content

htmlElem = html.document_fromstring(sourceCode)
houses = htmlElem.find_class('houses')
for house in houses:
    houseNumber = house.xpath('//input[@class="houseNumber"]/@value')
    houseState = house.xpath('//input[@class="houseState"]/@value')
    houseStatus = house.xpath('//input[@class="houseStatus"]/@value')

如上图所示，我如何捕获表格中的数据？我可以用不同的方式遍历 houses 对象吗？

更新:@efirvida 我已将代码修改为以下内容:

link = "example.com"
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}
response = requests.get(link, headers=headers, allow_redirects=False) 
sourceCode = response.content

htmlElem = html.document_fromstring(sourceCode)
houses = htmlElem.find_class('houses')
houseNumber = []
houseState = []
houseStatus = []

for house in houses:
    houseNumber.append(house.xpath('//input[@class="houseNumber"]/@value'))
    print(houseNumber)
    houseState.append(house.xpath('//input[@class="houseState"]/@value'))
    houseStatus.append(house.xpath('//input[@class="houseStatus"]/@value'))

data = map(list, zip(*[houseNumber,houseState,houseStatus]))

当我这样做时，会打印出以下内容:

[['107', '237', '104']]
[['107', '237', '104'], ['107', '237', '104']]
[['107', '237', '104']], ['107', '237', '104'], ['107', '237', '104']]

Answer 1

尝试转置结果，参见 this thread理解我的代码。

# create a list with elements
houseNumber = []
houseState = []
houseStatus = []

# append each element to it's list
for house in houses:
    houseNumber.append(house.xpath('//input[@class="houseNumber"]/@value'))
    houseState.append(house.xpath('//input[@class="houseState"]/@value'))
    houseStatus.append(house.xpath('//input[@class="houseStatus"]/@value'))


# transpose the lists, and turn into a list of list
data = map(list, zip(*[houseNumber,houseState,houseStatus]))

>>> list(data)
#[['107', 'MT', 'Occupied'], ['237', 'MT', 'Occupied'], ['104', 'MT', 'Vacant']]

如果你可以把它当作一个元组来使用，就把 map 去掉

#just transpose
data = zip(*[houseNumber,houseState,houseStatus])

>>> list(data)
#[('107', 'MT', 'Occupied'), ('237', 'MT', 'Occupied'), ('104', 'MT', 'Vacant') ]