python - 如何避免 scrapy UnicodeEncodeError

Question

我的 parse_item 回调中有以下代码:

sel = Selector(response)
item['name'] = sel.xpath('//div[@class="productDescriptionBlock"]/h2/text()').extract()[0]
return item

但是我得到UnicodeEncodeError:

exceptions.UnicodeEncodeError: 'charmap' codec can't encode character u'\uff01' in position 271761: character maps to <undefined>

我也尝试添加 .encode('utf-8') 但仍然出现相同的错误。

Traceback (most recent call last):
  File "/home/scraper/.fakeroot/lib/python2.7/site-packages/twisted/internet/base.py", line 824, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File "/home/scraper/.fakeroot/lib/python2.7/site-packages/twisted/internet/task.py", line 638, in _tick
    taskObj._oneWorkUnit()
  File "/home/scraper/.fakeroot/lib/python2.7/site-packages/twisted/internet/task.py", line 484, in _oneWorkUnit
    result = next(self._iterator)
  File "/home/scraper/.fakeroot/lib/python2.7/site-packages/scrapy/utils/defer.py", line 57, in <genexpr>
    work = (callable(elem, *args, **named) for elem in iterable)
--- <exception caught here> ---
  File "/home/scraper/.fakeroot/lib/python2.7/site-packages/scrapy/utils/defer.py", line 96, in iter_errback
    yield next(it)
  File "/home/scraper/.fakeroot/lib/python2.7/site-packages/scrapy/contrib/spidermiddleware/offsite.py", line 23, in process_spider_output
    for x in result:
  File "/home/scraper/.fakeroot/lib/python2.7/site-packages/scrapy/contrib/spidermiddleware/referer.py", line 22, in <genexpr>
    return (_set_referer(r) for r in result or ())
  File "/home/scraper/.fakeroot/lib/python2.7/site-packages/scrapy/contrib/spidermiddleware/urllength.py", line 33, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/home/scraper/.fakeroot/lib/python2.7/site-packages/scrapy/contrib/spidermiddleware/depth.py", line 50, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/home/scraper/.fakeroot/lib/python2.7/site-packages/scrapy/contrib/spiders/crawl.py", line 73, in _parse_response
    for request_or_item in self._requests_to_follow(response):
  File "/home/scraper/.fakeroot/lib/python2.7/site-packages/scrapy/contrib/spiders/crawl.py", line 52, in _requests_to_follow
    links = [l for l in rule.link_extractor.extract_links(response) if l not in seen]
  File "/home/scraper/.fakeroot/lib/python2.7/site-packages/scrapy/contrib/linkextractors/sgml.py", line 124, in extract_links
    ).encode(response.encoding)
  File "/home/scraper/.fakeroot/lib/python2.7/encodings/cp1252.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_table)
exceptions.UnicodeEncodeError: 'charmap' codec can't encode character u'\x99' in position 349751: character maps to <undefined>

Answer 1

我以前见过这个。如果我没记错的话，您在规则的链接提取器中使用了 restrict_xpaths 参数。

可能的解决方案是:

1. 避免对该特定站点使用 restrict_xpaths。发生这种情况是因为页面内容包含未在声明的编码中定义的字符。
2. 识别无效字符并在规则对其起作用之前替换它们。不过，这可能很棘手。
3. 使用此答案中的中间件将响应重新编码为其声明的编码:UnicodeEncodeError after setting restrict_xpaths settings