javascript - QtWebkit 在 Windows 上使 Python 崩溃

Question

我正在尝试抓取一个使用 javascript 的网站。我正在使用以下代码:

import os
import sys
import re
import requests
import mechanize
import cookielib
from bs4 import BeautifulSoup
from PyQt4.QtGui import *  
from PyQt4.QtCore import *  
from PyQt4.QtWebKit import *  
from lxml import html
import pandas as pd
import time

class Render(QWebPage):
    def __init__(self, url):
        self.app = QApplication(sys.argv)
        QWebPage.__init__(self)
        self.loadFinished.connect(self._loadFinished)
        self.mainFrame().load(QUrl(url))
        self.app.exec_()


    def _loadFinished(self, result):
        self.frame = self.mainFrame()
        self.app.quit()

def read_page(url):
    r = Render(url)
    result = r.frame.toHtml()
    text = str(result.toAscii())
    html_source = html.fromstring(text)
    return text, html_source

for url in urls:
    text, html_source = read_page(url)

成功读取第一个 url 后，在第二个 url 上显示以下消息并且 python.exe 崩溃。

content-type missing in HTTP POST, defaulting to application/x-www-form-urlencoded. Use QNetworkRequest::setHeader() to fix this problem.
QObject::connect: Cannot connect (null)::configurationAdded(QNetworkConfiguration) to QNetworkConfigurationManager::configurationAdded(QNetworkConfiguration)
QObject::connect: Cannot connect (null)::configurationRemoved(QNetworkConfiguration) to QNetworkConfigurationManager::configurationRemoved(QNetworkConfiguration)
QObject::connect: Cannot connect (null)::configurationChanged(QNetworkConfiguration) to QNetworkConfigurationManager::configurationChanged(QNetworkConfiguration)
QObject::connect: Cannot connect (null)::onlineStateChanged(bool) to QNetworkConfigurationManager::onlineStateChanged(bool)
QObject::connect: Cannot connect (null)::configurationUpdateComplete() to QNetworkConfigurationManager::updateCompleted()

Answer 1

这有点晚了，但我最近一直在玩网络抓取，我遇到了你的问题。问题是您正在尝试运行多个 QApplications，但这是行不通的(尽管我并不完全理解为什么 :/)。你应该尝试做类似什么 here 的事情.

所以基本上不用尝试启动几个 Qapp，只需创建一个 Qapp 并让它加载所有内容即可。