python-3.x - 更改 Scrapy/Splash 用户代理

Question

如何以如下等效的方式使用 Splash 设置 Scrapy 的用户代理:

import requests
from bs4 import BeautifulSoup

ua = {"User-Agent":"Mozilla/5.0"}
url = "http://www.example.com"
page = requests.get(url, headers=ua)
soup = BeautifulSoup(page.text, "lxml")

我的蜘蛛看起来像这样:

import scrapy
from scrapy_splash import SplashRequest


class ExampleSpider(scrapy.Spider):
        name = "example"
        allowed_domains = ["example.com"]
        start_urls = ["https://www.example.com/"]

        def start_requests(self):
            for url in self.start_urls:
                yield SplashRequest(
                    url,
                    self.parse,
                    args={'wait': 0.5}
                )

Answer 1

您需要设置 user_agent 属性来覆盖默认用户代理:

class ExampleSpider(scrapy.Spider):
    name = 'example'
    user_agent = 'Mozilla/5.0'

在这种情况下， UserAgentMiddleware (即 enabled by default )会将 USER_AGENT 设置值覆盖为 'Mozilla/5.0' 。

您还可以覆盖每个请求的 header :

scrapy_splash.SplashRequest(url, headers={'User-Agent': custom_user_agent})