gpt4 book ai didi

python - 如何在scrapy中给出每个请求之间的延迟?

转载 作者:行者123 更新时间:2023-12-03 06:45:21 24 4
gpt4 key购买 nike

我不想同时抓取并被阻止。我想每秒发送一个请求。

最佳答案

有一个setting为此:

DOWNLOAD_DELAY

Default: 0

The amount of time (in secs) that the downloader should wait before downloading consecutive pages from the same website. This can be used to throttle the crawling speed to avoid hitting servers too hard.

DOWNLOAD_DELAY = 0.25    # 250 ms of delay

阅读文档:https://doc.scrapy.org/en/latest/index.html

关于python - 如何在scrapy中给出每个请求之间的延迟?,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/8768439/

24 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com