django - Scrapy 与 DjangoItem 的集成产生错误

Question

我正在尝试使用 DjangoItem 运行 scrapy。当我运行 crawl my spider 时，出现“ExampleDotComItem 不支持字段:标题”错误。我创建了多个项目并试图让它工作，但总是得到同样的错误。我找到了 this教程并下载源码，运行后；我得到同样的错误:

Traceback (most recent call last):
File "c:\programdata\anaconda3\lib\site-packages\twisted\internet\defer.py",line 654, in _runCallbacks current.result = callback(current.result, *args, **kw) File "C:\Users\A\Desktop\django1.7-scrapy1.0.3-master\example_bot\example_bot\spiders\example.py", line 12, in parse return ExampleDotComItem(title=title, description=description) File "c:\programdata\anaconda3\lib\site-packages\scrapy_djangoitem__init__.py", line 29, in init super(DjangoItem, self).init(*args, **kwargs)
File "c:\programdata\anaconda3\lib\site-packages\scrapy\item.py", line 56, in init self[k] = v
File "c:\programdata\anaconda3\lib\site-packages\scrapy\item.py", line 66, in setitem (self.class.name, key)) KeyError: 'ExampleDotComItem does not support field: title'

项目结构:

├───django1.7-scrapy1.0.3-master
   ├───example_bot
   │   └───example_bot
   │       ├───spiders
   │       │   └───__pycache__
   │       └───__pycache__
   └───example_project
       ├───app
       │   ├───migrations
       │   │   └───__pycache__
       │   └───__pycache__
       └───example_project
           └───__pycache__

我的 Django 模型:

from django.db import models

class ExampleDotCom(models.Model):
    title = models.CharField(max_length=255)
    description = models.CharField(max_length=255)

    def __str__(self):
        return self.title

我的“示例”蜘蛛:

from scrapy.spiders import BaseSpider
from example_bot.items import ExampleDotComItem

class ExampleSpider(BaseSpider):
    name = "example"
    allowed_domains = ["example.com"]
    start_urls = ['http://www.example.com/']

    def parse(self, response):
         title = response.xpath('//title/text()').extract()[0]
         description = response.xpath('//body/div/p/text()').extract()[0]
         return ExampleDotComItem(title=title, description=description)

项目.py:

from scrapy_djangoitem import DjangoItem
from app.models import ExampleDotCom

class ExampleDotComItem(DjangoItem):
    django_model = ExampleDotCom

管道.py:

class ExPipeline(object):
    def process_item(self, item, spider):
        print(item)
        item.save()
        return item

设置.py:

import os
import sys

DJANGO_PROJECT_PATH = '/Users/A/DESKTOP/django1.7-scrapy1.0.3-master/example_project'
DJANGO_SETTINGS_MODULE = 'example_project.settings' #Assuming your django application's name is example_project

sys.path.insert(0, DJANGO_PROJECT_PATH)
os.environ['DJANGO_SETTINGS_MODULE'] = DJANGO_SETTINGS_MODULE
BOT_NAME = 'example_bot'



import django
django.setup()
SPIDER_MODULES = ['example_bot.spiders']

ITEM_PIPELINES = {
    'example_bot.pipelines.ExPipeline': 1000,
}

Answer 1

你能展示一下你的 Django 模型吗？这可能是因为 title 未在您的 ExampleDotCom 模型上定义。

如果存在，也许您需要运行 Django 迁移？