python - 如何在scrapy中同时使用http和https代理？

Question

我是 scrapy 的新手。我发现要使用 http 代理，但我想同时使用 http 和 https 代理，因为当我抓取链接时，那里有 http 和 https 链接。如何同时使用 http 和 https 代理？

class ProxyMiddleware(object):
    def process_request(self, request, spider):
        request.meta['proxy'] = "http://YOUR_PROXY_IP:PORT"
        #like here request.meta['proxy'] = "https://YOUR_PROXY_IP:PORT"
        proxy_user_pass = "USERNAME:PASSWORD"
        # setup basic authentication for the proxy
        encoded_user_pass = base64.encodestring(proxy_user_pass)
        request.headers['Proxy-Authorization'] = 'Basic ' + encoded_user_pass

Answer 1

您可以结合使用标准环境变量 HttpProxyMiddleware :

This middleware sets the HTTP proxy to use for requests, by setting the proxy meta value for Request objects.

Like the Python standard library modules urllib and urllib2, it obeys the following environment variables:

http_proxy
https_proxy
no_proxy

You can also set the meta key proxy per-request, to a value like http://some_proxy_server:port.