python - 如何使用python迭代scrapy中的节点

Question

我正在尝试抓取一个网站，html 的内容看起来像这样

<div class="panel-heading" role="tab" id="heading727654">
            <h4 class="panel-title">
                <a class="collapsed" data-toggle="collapse" data-parent="#accordion" href="#collapse727654" aria-expanded="false" aria-controls="collapse727654">
                    <div class="product-name">
                        <span class="product-title">
                            Aubrey<br><i>AGE DEFYING THERAPY CLEANSER 3.4 OZ</i>
                        </span>
                    </div>
                    <div class="product-price">
                        <span>
                            $10.99 / 3.40 OZ 
                        </span>
                </a>
            </h4>
</div>
<div class="panel-heading" role="tab" id="heading727655">
            <h4 class="panel-title">
                <a class="collapsed" data-toggle="collapse" data-parent="#accordion" href="#collapse727655" aria-expanded="false" aria-controls="collapse727654">
                    <div class="product-name">
                        <span class="product-title">
                            Aubrey<br><i>AGE DEFYING THERAPY LIQUID</i>
                        </span>
                    </div>
                    <div class="product-price">
                        <span>
                            $12.99 / 4.40 OZ 
                        </span>
                </a>
            </h4>
</div>

我的Python代码片段提取它是这样的

def parse(self, response):
        filename = response.url.split("/")[-2] + '.html'
        with open(filename, 'wb') as f:
            for node in response.xpath('//div[re:test(@class, "panel-heading")]'):
                print node.xpath('//span[re:test(@class, "product-title")]//text()').extract()
                print node.xpath('//span[re:test(@class, "product-price")]//text()').extract()

当我在Python中运行上面的scrapy代码时，我没有得到预期的输出，相同的内容被重复了100次。有人可以帮我解决这个问题吗？

Answer 1

您需要在内部 XPath 表达式中添加点，以使它们在 node 上下文中工作。否则，搜索从树的根开始:

def parse(self, response):
    filename = response.url.split("/")[-2] + '.html'
    with open(filename, 'wb') as f:
        for node in response.xpath('//div[re:test(@class, "panel-heading")]'):
            print node.xpath('.//span[re:test(@class, "product-title")]//text()').extract()
            print node.xpath('.//span[re:test(@class, "product-price")]//text()').extract()