python - Scrapy ItemLoader 项目组合

Question

我正在尝试使用 ItemLoader 将三个项目组合到一个数组中，如下所示:

[
    {
        site_title: "Some Site Title",
        anchor_text: "Click Here",
        link: "http://example.com/page"
    }
]

正如您在下面的 JSON 中看到的，它将同一类型的所有项目组合在一起。

我应该如何处理这个问题以输出带有我正在寻找的数组的 JSON？

蜘蛛文件:

import scrapy
from linkfinder.items import LinkfinderItem
from scrapy.loader import ItemLoader

class LinksSpider(scrapy.Spider):
    name = "links"
    allowed_domains = ["wpseotest.com"]
    start_urls = ["https://wpseotest.com"]

    def parse(self, response):

        l = ItemLoader(item=LinkfinderItem(), response=response)
        l.add_xpath('site_title', '//title/text()')
        l.add_xpath('anchor_text', '//a//text()')
        l.add_xpath('link', '//a/@href')
        return l.load_item()

        pass

项目.py

import scrapy
from scrapy import item, Field

class LinkfinderItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    site_title = Field()
    anchor_text = Field()
    link = Field()
    pass

JSON 输出

[
{"anchor_text": ["Globex Corporation", "Skip to content", "Home", "About", "Globex News", "Events", "Contact Us", "3999 Mission Boulevard,\r", "San Diego, CA 92109", "This is a test scheduled\u00a0post.", "Test Title", "Globex Subsidiary Ice Cream Inc. Creates Chicken Wing\u00a0Flavor", "Globex Inc.", "\r\n", "Blog at WordPress.com."], "link": ["https://wpseotest.com/", "#content", "https://wpseotest.com/", "https://wpseotest.com/about/", "https://wpseotest.com/globex-news/", "https://wpseotest.com/events/", "https://wpseotest.com/contact-us/", "http://maps.google.com/maps?z=16&q=3999+mission+boulevard,+san+diego,+ca+92109", "https://wpseotest.com/2016/08/19/this-is-a-test-scheduled-post/", "https://wpseotest.com/2016/06/28/test-title/", "https://wpseotest.com/2015/10/18/globex-subsidiary-ice-cream-inc-creates-chicken-wing-flavor/", "https://wpseotest.wordpress.com", "https://wordpress.com/?ref=footer_blog"], "site_title": ["Globex Corporation \u2013 We make things better, or, sometimes, worse."]}
]

Answer 1

您想为此处的每个链接生成一个项目吗？
为此，您要做的就是找到文章节点，然后迭代它们并找到稍后组合到字典/scrapy.Item 中的字段。

def parse(self, response):
    site_title = response.xpath("//title/text()").extract_first() 
    links = response.xpath("//a")
    for link in links:
        l = ItemLoader(selector=link)
        l.add_value('site_title', site_title)
        l.add_xpath('anchor_text', 'text()')
        l.add_xpath('link', '@href')
        yield l.load_item()

现在你可以运行scrapycrawl myspider -o output.json，你应该得到类似的东西:

{[
    {"site_title": "title",
     "anchor_text": "foo",
     "link": "http://foo.com"},
    {"site_title": "title",
     "anchor_text": "bar",
     "link": "http://bar.com"}
    ...
  ]
}