- html - 出于某种原因,IE8 对我的 Sass 文件中继承的 html5 CSS 不友好?
- JMeter 在响应断言中使用 span 标签的问题
- html - 在 :hover and :active? 上具有不同效果的 CSS 动画
- html - 相对于居中的 html 内容固定的 CSS 重复背景?
我一直在试图弄清楚如何加快速度并获得一些线程知识。
我一直在尝试创建一个放置两个 GET 请求的函数。对于每个链接,我会抓取一些数据,然后将其保存到返回的列表中,然后我将使用它来比较这些链接之一中是否出现了新链接:
"""
def getScrapeLinks(self, siteURL):
response = requests.get(
siteURL,
timeout=5
)
if response.ok:
bs4 = soup(response.text, 'lxml')
links = ['{}'.format( raw_product.find('a').get('href')) for
raw_product in bs4.find_all('div', {'class': 'test'})]
return links
"""
def pollNewProducts(self, storeClass):
# storeClass.siteCatalog = ["https://www.google.com", "https://www.facebook.com"]
LinksLists = reduce(operator.add, [getScrapeLinks(getLinks) for getLinks in storeClass.siteCatalog])
while True:
newLinksLists = reduce(operator.add,
[getScrapeLinks(getLinks) for getLinks in storeClass.siteCatalog]
)
for URL in newLinksLists:
if URL not in LinksLists:
print("New link")
print(URL)
LinksLists.append(URL)
else:
print("Sleep to see new links!")
time.sleep(random.randint(2, 4))
最佳答案
改编自我对 this 问题的回答。
您应该研究异步编程。与线程不同,异步代码在同一个线程中运行,但它在事件循环中运行。当 Python 关键字 await
存在时,此事件循环会自动在不同操作之间切换上下文。
换句话说,将抓取网站视为以下内容:
client sends request -> ... waiting for server reply ... <- server replies
client sends request -> switch operation -> ... wait ... <- server replies
client sends request -> switch operation -> ... wait ... <- server replies
client sends request -> switch operation -> ... wait ... <- server replies
...
asynchro
模块 + PyPi 可安装的
aiohttp
模块),我建议谷歌搜索一下。这是一个代码示例,它将花费不到 4 秒的时间来抓取 100 多个网站(请注意,这非常好,4 秒实际上是由于打印语句......没有,它实际上更接近 2 秒):
import asyncio
import aiohttp
import time
websites = """https://www.youtube.com
https://www.facebook.com
https://www.baidu.com
https://www.yahoo.com
https://www.amazon.com
https://www.wikipedia.org
http://www.qq.com
https://www.google.co.in
https://www.twitter.com
https://www.live.com
http://www.taobao.com
https://www.bing.com
https://www.instagram.com
http://www.weibo.com
http://www.sina.com.cn
https://www.linkedin.com
http://www.yahoo.co.jp
http://www.msn.com
http://www.uol.com.br
https://www.google.de
http://www.yandex.ru
http://www.hao123.com
https://www.google.co.uk
https://www.reddit.com
https://www.ebay.com
https://www.google.fr
https://www.t.co
http://www.tmall.com
http://www.google.com.br
https://www.360.cn
http://www.sohu.com
https://www.amazon.co.jp
http://www.pinterest.com
https://www.netflix.com
http://www.google.it
https://www.google.ru
https://www.microsoft.com
http://www.google.es
https://www.wordpress.com
http://www.gmw.cn
https://www.tumblr.com
http://www.paypal.com
http://www.blogspot.com
http://www.imgur.com
https://www.stackoverflow.com
https://www.aliexpress.com
https://www.naver.com
http://www.ok.ru
https://www.apple.com
http://www.github.com
http://www.chinadaily.com.cn
http://www.imdb.com
https://www.google.co.kr
http://www.fc2.com
http://www.jd.com
http://www.blogger.com
http://www.163.com
http://www.google.ca
https://www.whatsapp.com
https://www.amazon.in
http://www.office.com
http://www.tianya.cn
http://www.google.co.id
http://www.youku.com
https://www.example.com
http://www.craigslist.org
https://www.amazon.de
http://www.nicovideo.jp
https://www.google.pl
http://www.soso.com
http://www.bilibili.com
http://www.dropbox.com
http://www.xinhuanet.com
http://www.outbrain.com
http://www.pixnet.net
http://www.alibaba.com
http://www.alipay.com
http://www.chrome.com
http://www.booking.com
http://www.googleusercontent.com
http://www.google.com.au
http://www.popads.net
http://www.cntv.cn
http://www.zhihu.com
https://www.amazon.co.uk
http://www.diply.com
http://www.coccoc.com
https://www.cnn.com
http://www.bbc.co.uk
https://www.twitch.tv
https://www.wikia.com
http://www.google.co.th
http://www.go.com
https://www.google.com.ph
http://www.doubleclick.net
http://www.onet.pl
http://www.googleadservices.com
http://www.accuweather.com
http://www.googleweblight.com
http://www.answers.yahoo.com"""
async def get(url):
try:
async with aiohttp.ClientSession() as session:
async with session.get(url=url) as response:
resp = await response.read()
print("Successfully got url {} with response of length {}.".format(url, len(resp)))
except Exception as e:
print("Unable to get url {} due to {}.".format(url, e.__class__))
async def main(urls, amount):
ret = await asyncio.gather(*[get(url) for url in urls])
print("Finalized all. ret is a list of len {} outputs.".format(len(ret)))
urls = websites.split("\n")
amount = len(urls)
start = time.time()
asyncio.run(main(urls, amount))
end = time.time()
print("Took {} seconds to pull {} websites.".format(end - start, amount))
Successfully got url http://www.google.com.br with response of length 12188.
Successfully got url http://www.google.it with response of length 12155.
Successfully got url https://www.t.co with response of length 0.
Successfully got url http://www.msn.com with response of length 46335.
Successfully got url http://www.chinadaily.com.cn with response of length 122053.
Successfully got url https://www.google.co.in with response of length 11557.
Successfully got url https://www.google.de with response of length 12135.
Successfully got url https://www.facebook.com with response of length 115258.
Successfully got url http://www.gmw.cn with response of length 120866.
Successfully got url https://www.google.co.uk with response of length 11540.
Successfully got url https://www.google.fr with response of length 12189.
Successfully got url http://www.google.es with response of length 12163.
Successfully got url http://www.google.co.id with response of length 12169.
Successfully got url https://www.bing.com with response of length 117915.
Successfully got url https://www.instagram.com with response of length 36307.
Successfully got url https://www.google.ru with response of length 12128.
Successfully got url http://www.googleusercontent.com with response of length 1561.
Successfully got url http://www.xinhuanet.com with response of length 179254.
Successfully got url http://www.google.ca with response of length 11592.
Successfully got url http://www.accuweather.com with response of length 269.
Successfully got url http://www.googleadservices.com with response of length 1561.
Successfully got url https://www.whatsapp.com with response of length 77951.
Successfully got url http://www.cntv.cn with response of length 3139.
Successfully got url http://www.google.com.au with response of length 11579.
Successfully got url https://www.example.com with response of length 1270.
Successfully got url http://www.google.co.th with response of length 12151.
Successfully got url https://www.amazon.com with response of length 465905.
Successfully got url https://www.wikipedia.org with response of length 76240.
Successfully got url https://www.google.co.kr with response of length 12211.
Successfully got url https://www.apple.com with response of length 63322.
Successfully got url http://www.uol.com.br with response of length 333257.
Successfully got url https://www.aliexpress.com with response of length 59742.
Successfully got url http://www.sohu.com with response of length 215201.
Successfully got url https://www.google.pl with response of length 12144.
Successfully got url https://www.googleweblight.com with response of length 0.
Successfully got url https://www.cnn.com with response of length 1138392.
Successfully got url https://www.google.com.ph with response of length 11561.
Successfully got url https://www.linkedin.com with response of length 71498.
Successfully got url https://www.naver.com with response of length 176038.
Successfully got url https://www.live.com with response of length 3667.
Successfully got url https://www.twitch.tv with response of length 61599.
Successfully got url http://www.163.com with response of length 696338.
Successfully got url https://www.ebay.com with response of length 307068.
Successfully got url https://www.wordpress.com with response of length 76680.
Successfully got url https://www.wikia.com with response of length 291400.
Successfully got url http://www.chrome.com with response of length 161223.
Successfully got url https://www.twitter.com with response of length 291741.
Successfully got url https://www.stackoverflow.com with response of length 105987.
Successfully got url https://www.netflix.com with response of length 83125.
Successfully got url https://www.tumblr.com with response of length 78110.
Successfully got url http://www.doubleclick.net with response of length 129901.
Successfully got url https://www.yahoo.com with response of length 531829.
Successfully got url http://www.soso.com with response of length 174.
Successfully got url https://www.microsoft.com with response of length 187549.
Successfully got url http://www.office.com with response of length 89556.
Successfully got url http://www.alibaba.com with response of length 167978.
Successfully got url https://www.reddit.com with response of length 483295.
Successfully got url http://www.outbrain.com with response of length 24432.
Successfully got url http://www.tianya.cn with response of length 7941.
Successfully got url https://www.baidu.com with response of length 156768.
Successfully got url http://www.diply.com with response of length 3074314.
Successfully got url http://www.blogspot.com with response of length 94478.
Successfully got url http://www.popads.net with response of length 14548.
Successfully got url http://www.answers.yahoo.com with response of length 104726.
Successfully got url http://www.blogger.com with response of length 94478.
Successfully got url http://www.imgur.com with response of length 4008.
Successfully got url http://www.qq.com with response of length 244841.
Successfully got url http://www.paypal.com with response of length 45587.
Successfully got url http://www.pinterest.com with response of length 45692.
Successfully got url http://www.github.com with response of length 86917.
Successfully got url http://www.zhihu.com with response of length 31473.
Successfully got url http://www.go.com with response of length 594291.
Successfully got url http://www.fc2.com with response of length 34546.
Successfully got url https://www.amazon.de with response of length 439209.
Successfully got url https://www.youtube.com with response of length 439571.
Successfully got url http://www.bbc.co.uk with response of length 321966.
Successfully got url http://www.tmall.com with response of length 234388.
Successfully got url http://www.imdb.com with response of length 289339.
Successfully got url http://www.dropbox.com with response of length 103714.
Successfully got url http://www.bilibili.com with response of length 50959.
Successfully got url http://www.jd.com with response of length 18105.
Successfully got url http://www.yahoo.co.jp with response of length 18565.
Successfully got url https://www.amazon.co.jp with response of length 479721.
Successfully got url http://www.craigslist.org with response of length 59372.
Successfully got url https://www.360.cn with response of length 74502.
Successfully got url http://www.ok.ru with response of length 170516.
Successfully got url https://www.amazon.in with response of length 460696.
Successfully got url http://www.booking.com with response of length 408992.
Successfully got url http://www.yandex.ru with response of length 116661.
Successfully got url http://www.nicovideo.jp with response of length 107271.
Successfully got url http://www.onet.pl with response of length 720657.
Successfully got url http://www.alipay.com with response of length 21698.
Successfully got url https://www.amazon.co.uk with response of length 443607.
Successfully got url http://www.sina.com.cn with response of length 579107.
Successfully got url http://www.hao123.com with response of length 295213.
Successfully got url http://www.pixnet.net with response of length 6295.
Successfully got url http://www.coccoc.com with response of length 45822.
Successfully got url http://www.taobao.com with response of length 393128.
Successfully got url http://www.weibo.com with response of length 95482.
Successfully got url http://www.youku.com with response of length 762485.
Finalized all. ret is a list of len 100 outputs.
Took 3.899034023284912 seconds to pull 100 websites.
关于python - 如何使用线程处理多个 GET 请求并进行比较,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/61924004/
我喜欢 smartcase,也喜欢 * 和 # 搜索命令。但我更希望 * 和 # 搜索命令区分大小写,而/和 ?搜索命令遵循 smartcase 启发式。 是否有隐藏在某个地方我还没有找到的设置?我宁
关闭。这个问题是off-topic .它目前不接受答案。 想改进这个问题? Update the question所以它是on-topic对于堆栈溢出。 10年前关闭。 Improve this qu
从以下网站,我找到了执行java AD身份验证的代码。 http://java2db.com/jndi-ldap-programming/solution-to-sslhandshakeexcepti
似乎 melt 会使用 id 列和堆叠的测量变量 reshape 您的数据框,然后通过转换让您执行聚合。 ddply,从 plyr 包看起来非常相似..你给它一个数据框,几个用于分组的列变量和一个聚合
我的问题是关于 memcached。 Facebook 使用 memcached 作为其结构化数据的缓存,以减少用户的延迟。他们在 Linux 上使用 UDP 优化了 memcached 的性能。 h
在 Camel route ,我正在使用 exec 组件通过 grep 进行 curl ,但使用 ${HOSTNAME} 的 grep 无法正常工作,下面是我的 Camel 路线。请在这方面寻求帮助。
我正在尝试执行相当复杂的查询,在其中我可以排除与特定条件集匹配的项目。这是一个 super 简化的模型来解释我的困境: class Thing(models.Model) user = mod
我正在尝试执行相当复杂的查询,我可以在其中排除符合特定条件集的项目。这里有一个 super 简化的模型来解释我的困境: class Thing(models.Model) user = mod
我发现了很多嵌入/内容项目的旧方法,并且我遵循了在这里找到的最新方法(我假设):https://blog.angular-university.io/angular-ng-content/ 我正在尝试
我正在寻找如何使用 fastify-nextjs 启动 fastify-cli 的建议 我曾尝试将代码简单地添加到建议的位置,但它不起作用。 'use strict' const path = req
我正在尝试将振幅 js 与 React 和 Gatsby 集成。做 gatsby developer 时一切看起来都不错,因为它发生在浏览器中,但是当我尝试 gatsby build 时,我收到以下错
我试图避免过度执行空值检查,但同时我想在需要使代码健壮的时候进行空值检查。但有时我觉得它开始变得如此防御,因为我没有实现 API。然后我避免了一些空检查,但是当我开始单元测试时,它开始总是等待运行时异
尝试进行包含一些 NOT 的 Kibana 搜索,但获得包含 NOT 的结果,因此猜测我的语法不正确: "chocolate" AND "milk" AND NOT "cow" AND NOT "tr
我正在使用开源代码共享包在 iOS 中进行 facebook 集成,但收到错误“FT_Load_Glyph failed: glyph 65535: error 6”。我在另一台 mac 机器上尝试了
我正在尝试估计一个标准的 tobit 模型,该模型被审查为零。 变量是 因变量 : 幸福 自变量 : 城市(芝加哥,纽约), 性别(男,女), 就业(0=失业,1=就业), 工作类型(失业,蓝色,白色
我有一个像这样的项目布局 样本/ 一种/ 源/ 主要的/ java / java 资源/ .jpg 乙/ 源/ 主要的/ java / B.java 资源/ B.jpg 构建.gradle 设置.gr
如何循环遍历数组中的多个属性以及如何使用map函数将数组中的多个属性显示到网页 import React, { Component } from 'react'; import './App.css'
我有一个 JavaScript 函数,它进行 AJAX 调用以返回一些数据,该调用是在选择列表更改事件上触发的。 我尝试了多种方法来在等待时显示加载程序,因为它当前暂停了选择列表,从客户的 Angul
可能以前问过,但找不到。 我正在用以下形式写很多语句: if (bar.getFoo() != null) { this.foo = bar.getFoo(); } 我想到了三元运算符,但我认
我有一个表单,在将其发送到 PHP 之前我正在执行一些验证 JavaScript,验证后的 JavaScript 函数会发布用户在 中输入的文本。页面底部的标签;然而,此消息显示短暂,然后消失...
我是一名优秀的程序员,十分优秀!