python - 如何使用 beautiful soup 动态抓取内部链接

Question

我的要求是从网页中获取标题、图像和摘要。

我能够从主页获取标题和摘要。但是，要获取摘要，我需要抓取 anchor 标记中提到的内部 URL。

我成功获取了标题、图像和 anchor 标记链接。但我无法找到如何发送此 anchor 标记链接来获取摘要。

请帮助我。

from bs4 import BeautifulSoup
import requests
source = requests.get('https://www.aitrends.com/category/ai-software/').text
soup = BeautifulSoup(source, 'lxml')
match = soup.find_all('div',class_='td-module-thumb')
for x in match:
 headline = x.a.get('title')
 print(headline)
 imgsrc = x.img.get('src')
 print(imgsrc)
 artlink = x.a.get('href')
 print (artlink)

我正在尝试进入 artlink 并从中提取摘要。

Answer 1

您可以为每个链接运行新请求:

from bs4 import BeautifulSoup as soup
import requests
def get_summary(url):
   new_d = soup(requests.get(url).text, 'html.parser')
   return '\n'.join(i.text for i in new_d.find('div', {'class':'td-post-content'}).find_all('p'))

d = soup(requests.get('https://www.aitrends.com/category/ai-software/').text, 'html.parser')
results = [{'title':i.h3.text, 'img':i.img['src'], 'summary':get_summary(i.a['href'])} for i in d.find_all('div', {'class':'td-block-span6'})]

输出(仅第一个结果，由于 SO 的字符限制):

{'title': 'AI Still Far Away from Mission-Critical Role, DoD’s Porter Says', 'img': 'https://www.aitrends.com/wp-content/uploads/2019/06/6-14Pentagon-2-324x160.jpg', 'summary': 'Dr. Lisa Porter, Deputy Under Secretary of Defense for Research and Engineering, had a lot of good things to say about the promise of artificial intelligence (AI) technologies at the GEOINT Symposium on June 4, with one important caveat: AI isn’t ready for prime time in Department of Defense (DoD) critical applications, and likely won’t be for some time.\nSpeaking on June 4, Porter spoke about AI and DoD apps, and made it clear that the best way to take advantage of AI is to put significant effort into finding a problem that the technology can really help solve. What is most useful, she said, “is a well-structured problem that is suitable to AI … otherwise AI is just a shiny tool.”\n“Not every problem is ideal for AI,” she said, and advised attendees to “understand the problem better” as a first step. “Take more time to understand what problem you are trying to solve,” she said. “Then see if it’s really possible to generate the right kind of AI data.”\nPorter further urged technologists to spend a lot of time at the beginning of the process with end-users evaluating whether a potential AI project features the right data, reasonable outcomes, and proper metrics to evaluate results.\nAs for mission-critical DoD applications, Porter ticked off a list of problems with the current state of AI development that she said collectively constitute a “very big problem” for using the technology in vital situations. Those include:\nShe also said agencies still seeking to move past legacy IT systems to AI-ready systems are facing a “very hard, heavy lift” in that process.\nTo companies looking to pitch the Federal government on AI applications, she strongly urged them to “explain why your product is effective,” and to fully discuss data sources, algorithms, and how applications produce consistently repeatable results. Companies that can’t show enough evidence on that front might win a pilot project from DoD, but “people who try to take short cuts get caught in pilot purgatory,” and aren’t likely to win more lucrative contracts, she said.\nReporting on AI efforts already underway within DoD, Porter said the agency’s Joint Artificial Intelligence Center (JAIC) “is just starting to get going” after being created a year ago. She said the effort “really has the right focus” on “the impact of AI at scale.”\n“They realize this is very hard,” Porter said, adding, “It’s all about how we do AI at enterprise level.”\n“There’s nothing very smart about today’s AI tools … That’s what we need to improve,” she said. The achievement of “common sense” in human-machine teaming would be “nirvana,” Porter added. “That team could be very powerful … All of these things require some degree of cognition.”\nOn the DARPA front, Porter said about one-third of the organization’s current projects involve AI “to some degree.”\nWhile advanced technology development remains a daunting task, “we will always be ahead if we play to our strengths,” she said. “Those who cheat and steal from us will never win if we play to our strengths,” including adhering to the rule of law, Porter said.\nThere are probably few Federal officials better positioned to judge the capabilities timeline for AI than Porter. In her current position, she oversees research, development and prototyping activities across the DoD enterprise, along with the activities of the Defense Advanced Research Projects Agency (DARPA), the Missile Defense Agency, the Strategic Capabilities Office, the Defense Innovation Unit, and the DoD Laboratory and Engineering Center enterprise.\nBefore her current post, she was executive vice president at In-Q-Tel, and was the first director of the Intelligence Advanced Research Projects Activity (IARPA).\nDr. Porter holds a bachelor’s degree in nuclear engineering from the Massachusetts Institute of Technology and a doctorate in applied physics from Stanford University. She received the Office of the Secretary of Defense Medal for Exceptional Public Service in 2005, the NASA Outstanding Leadership Medal in 2008, the National Intelligence Distinguished Service Medal in 2012, and the Presidential Meritorious Rank Award in 2013.\nSee the source article at MeriTalk.'}