当我尝试像这样在 Scrapy 中构造一个 HtmlResponse
对象时:
scrapy.http.HtmlResponse(url=self.base_url + dealer_url[0], body=dealer_html)
我遇到了这个错误:
Traceback (most recent call last):
File "d:\kerja\hit\python~1\<project_name>\<project_name>\lib\site-packages\twisted\internet\defer.py", line 588, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "D:\Kerja\HIT\Python Projects\<project_name>\<project_name>\<project_name>\<project_name>\spiders\fwi.py", line 69, in parse_items
dealer_page = scrapy.http.HtmlResponse(url=self.base_url + dealer_url[0], body=dealer_html)
File "d:\kerja\hit\python~1\<project_name>\<project_name>\lib\site-packages\scrapy\http\response\text.py", line 27, in __init__
super(TextResponse, self).__init__(*args, **kwargs)
File "d:\kerja\hit\python~1\<project_name>\<project_name>\lib\site-packages\scrapy\http\response\__init__.py", line 18, in __init__
self._set_body(body)
File "d:\kerja\hit\python~1\<project_name>\<project_name>\lib\site-packages\scrapy\http\response\text.py", line 43, in _set_body
type(self).__name__)
TypeError: Cannot convert unicode body - HtmlResponse has no encoding
有谁知道如何解决这个错误?
HtmlResponse正在尝试检测编码:
The HtmlResponse class is a subclass of TextResponse which adds encoding auto-discovering support by looking into the HTML meta http-equiv attribute. See TextResponse.encoding.
所以基本上,您提供给 body
参数的 html 字符串(在您的情况下为 dealer_html
)没有指定编码。根据 w3 docs of http-equiv
它应该有:
HTML 4.01: <meta http-equiv="content-type" content="text/html; charset=UTF-8">
HTML5: <meta charset="UTF-8">
在这种情况下,您可以修复 html 或在通过 encoding
参数创建 HtmlResponse
对象时指定编码:
HtmlResponse(url='http://scrapy.org', body=u'some body', encoding='utf-8')
我是一名优秀的程序员,十分优秀!