python - Scrapy 迭代同级节点 - xpath 产生空列表

Question

我正在迭代元素的同级节点。但是，我在使用 xpath 函数提取其中的数据时遇到问题。我似乎总是得到一个空列表。

# Determine if the cast members are listed in a table or unordered list.
cast_siblings = response.xpath('//h2/span[starts-with(@id,"Cast")]/../following-sibling::*')
for sibling in cast_siblings:
    desc_str = "".join(sibling.xpath('./descendant::text()').extract())
    if "Trailer" in desc_str or "Film Festival" in desc_str or "Comment" in desc_str:
        break

    # Extract the name to get the type of the node.
    node_type = sibling.xpath('name()').extract_first()
    if node_type == 'ul':
        cast_list = sibling.xpath('./ul[1]/li')
        self.parse_list_into_cast_table(cast_list, _id, conn)
    elif node_type == 'table':
        cast_tables = sibling.xpath('./table')
        self.parse_table_into_cast_table(cast_tables, _id, conn)
    else:
        print "Unknown cast element type! " + _id

在上面的代码中，我不断遍历同级节点，直到遇到文本中某处包含“预告片”、“电影节”或“评论”的元素。

否则，对于每个兄弟节点，我首先获取其名称以找出它是什么类型的节点 - 我对表和无序列表感兴趣。

但是，失败的部分是当我调用sibling.xpath('./table')和/或sibling.xpath('ul[1]/li')时。

生成的对象始终是一个空列表。

我使用了 scrapy shell 并检查了从cast_siblings 获得的选择器列表。每个选择器似乎都是有效的，当我提取它们时，我可以看到内部元素。我只是无法使用 xpath 访问它们。

这是我的控制台的摘录，显示了空列表结果:

>>> a = response.xpath('//h2/span[starts-with(@id,"Cas")]/../following-sibling::*')
>>> a
[<Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<h3> <span class="mw-headline" id="Sudo_'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table border="0" cellspacing="3" cellpa'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table border="0" cellspacing="3" cellpa'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<h3> <span class="mw-headline" id="Maruy'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table border="0" cellspacing="3" cellpa'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<h3> <span class="mw-headline" id="Azumi'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table border="0" cellspacing="3" cellpa'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table border="0" cellspacing="3" cellpa'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table border="0" cellspacing="3" cellpa'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<h3> <span class="mw-headline" id="Prese'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table border="0" cellspacing="3" cellpa'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<p><b>Additional Cast Members:</b>\n</p>'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<ul><li> <a href="/Yukari_Ito" title="Yu'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<h2> <span class="mw-headline" id="Trail'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<p>\n<script type="text/javascript" src="'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<div id="mediaplayer"></div>'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<script type="text/javascript">\n  jwplay'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<h2> <span class="mw-headline" id="Comme'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<br>'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<div id="comment-outer"><hr><div id="com'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<div align="center"></div>'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table cellpadding="2" style="border:0px'>]
>>> a[0]
<Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<h3> <span class="mw-headline" id="Sudo_'>
>>> a[1]
<Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table border="0" cellspacing="3" cellpa'>
>>> a[1].xpath('./table')
[]

这真的很奇怪，因为上面示例中的 a[1] 清楚地显示了其中的表格。

Answer 1

似乎您正在寻找已经是表的节点的表子级:

node_type = sibling.xpath('name()').extract_first()
if node_type == 'ul':
    ...
elif node_type == 'table':
    cast_tables = sibling.xpath('./table') <---

这里你已经知道当前节点是表，并且你正在尝试在它下面寻找另一个表。所以在这种情况下它应该是:

cast_tables = sibling

还值得注意的是，当您测试 scrapy 选择器时，结果包括根节点标签，即 '//h1'将返回<h1>...</h1>