Python Mechanize 在处理网站 html 时遇到问题

Question

我正在尝试使用 python 模块 mechanize 在网页上填写表单，然后下载生成的 html。网站如下:

“xxx”

但首先，我只想列出表格。我的代码如下:

import mechanize

br = mechanize.Browser()
br.set_handle_robots(False)   # ignore robots
br.set_handle_refresh(False)  # can sometimes hang without this

url = "xxx"
response = br.open(url)
a = response.read()      # the text of the page

for form in br.forms():
    print "Form name:", form.name
    print form

给出的错误列表相当大:

In [75]: % run -i form_filler.py
---------------------------------------------------------------------------
ParseError                                Traceback (most recent call last)
/home/blake/Python/form_filler.py in <module>()
     19 
     20 
---> 21 for form in br.forms():
     22         print "Form name:", form.name
     23         print form

/usr/local/lib/python2.7/dist-packages/mechanize-0.2.5-py2.7.egg/mechanize/_mechanize.pyc in forms(self)
    418         if not self.viewing_html():
    419             raise BrowserStateError("not viewing HTML")
--> 420         return self._factory.forms()
    421 
    422     def global_form(self):

/usr/local/lib/python2.7/dist-packages/mechanize-0.2.5-py2.7.egg/mechanize/_html.pyc in forms(self)
    555             try:
    556                 self._forms_genf = CachingGeneratorFunction(
--> 557                     self._forms_factory.forms())
    558             except:  # XXXX define exception!
    559                 self.set_response(self._response)

/usr/local/lib/python2.7/dist-packages/mechanize-0.2.5-py2.7.egg/mechanize/_html.pyc in forms(self)
    235             _urljoin=_rfc3986.urljoin,
    236             _urlparse=_rfc3986.urlsplit,
--> 237             _urlunparse=_rfc3986.urlunsplit,
    238             )
    239         self.global_form = forms[0]

/usr/local/lib/python2.7/dist-packages/mechanize-0.2.5-py2.7.egg/mechanize/_form.pyc in ParseResponseEx(response, select_default, form_parser_class, request_class, entitydefs, encoding, _urljoin, _urlparse, _urlunparse)
    842                         _urljoin=_urljoin,
    843                         _urlparse=_urlparse,
--> 844                         _urlunparse=_urlunparse,
    845                         )
    846 

/usr/local/lib/python2.7/dist-packages/mechanize-0.2.5-py2.7.egg/mechanize/_form.pyc in _ParseFileEx(file, base_uri, select_default, ignore_errors, form_parser_class, request_class, entitydefs, backwards_compat, encoding, _urljoin, _urlparse, _urlunparse)
    979         data = file.read(CHUNK)
    980         try:
--> 981             fp.feed(data)
    982         except ParseError, e:
    983             e.base_uri = base_uri

/usr/local/lib/python2.7/dist-packages/mechanize-0.2.5-py2.7.egg/mechanize/_form.pyc in feed(self, data)
    758             _sgmllib_copy.SGMLParser.feed(self, data)
    759         except _sgmllib_copy.SGMLParseError, exc:
--> 760             raise ParseError(exc)
    761 
    762     def close(self):

ParseError: expected name token at '<!!--end footer-->\r\n'

我的理解是 html 以某种方式“写得不好”。我已经在 google 等示例网站上尝试了上述代码，并且运行良好。怀疑是回车有关系，但是怎么绕过这个问题呢？

Answer 1

显然，如果我使用

br = mechanize.Browser(factory=mechanize.RobustFactory()) 

代替

br = mechanize.Browser()

它给出了正确的输出:

In [18]: % run -i form_filler.py
Form name: qSearchForm
<qSearchForm GET xxx
  <TextControl(q=Search)>
  <SubmitControl(qSearchBtn=) (readonly)>>
Form name: form1
<form1 POST xxx
  <TextControl(name=)>
  <SelectControl(code=[*LER, Lerwick, Scotland, 29.9 / 358.8, ESK, Eskdalemuir, Scotland, 34.7 / 356.8, HAD, Hartland, England, 39.0 / 355.5, ABN, Abinger, England, 38.8 / 359.6, GRW, Greenwich, England, 38.5 / 0.0])>
  <TextControl(month=)>
  <TextControl(year=)>
  <SubmitControl(<None>=Submit Query) (readonly)>
  <IgnoreControl(<None>=<None>)>>