python - Scraper 收集了很少的标题并忽略了其余的

Question

我在 python 中编写了一个小脚本，使用 xpath 从 yahoo finance 中抓取显示在左侧区域的标题。该页面中有几个标题，但是当我运行我的脚本时，我只得到三个标题。我不想在这种情况下使用 Selenium ，因为我至少得到了最低限度的结果。我试图在我的脚本中使用“时间”来让网页完全加载。然而，似乎与“时间”无关。除了使用 selenium 之外，我如何从该网页获取所有标题？这是我尝试过的:

import time
import requests
from lxml import html

response = requests.get("https://finance.yahoo.com/").text
time.sleep(5)
root = html.fromstring(response)
for title in root.xpath("//a[u[@class='StretchedBox']]/text()"):
    print(title)

Answer 1

我试用了您的代码，显然我获得的不仅仅是 3 个标题。

Yahoo 可能只是更改了他们网站的动态设置，使您的代码能够解析更多标题。

输出:

This could get 'very ugly, very fast' for Boeing, says one market expert
Toys R Us submits plan to liquidate its US business, will close or sell all US stores
Sears Gains After Tax Benefit Helps Retailer Swing to a Profit
Blood, Fraud and Money Led to Theranos CEO's Fall From Grace
Why Cisco Systems Is Looking Like a Must-Own Stock
Realty Income (O) Announces Hike in Monthly Dividend Payment
Tillerson's exit has 'very big implications' for oil prices: RBC’s Helima Croft
Kevin O'Leary shares his No. 1 trick for tipping at a res...
For J C Penney Co. Inc. Stock Survival is the Victory
Why Ford (F) Stock Popped Despite Recalling 1.4 Million Vehicles
Toys 'R' Us Liquidation Reveals a Dangerous Trend Forming in U.S. Retailing
Theranos CEO Elizabeth Holmes settles with SEC, agrees to...
Manafort Urges Judge to Dismiss Laundering and Lobbying Case
‘Dow Theory’ Warning Signal Spells Trouble for These 5 Stocks
General Electric Company Stock Is Ugly, But It Has Potential for the Gambler
This popular Dow stock looks ready to join GE on the discard pile
Intel's Broadcom Buyout Rumors Put These 3 Stocks at Risk
The SEC isn't sending Theranos CEO Elizabeth Holmes to jail for fraud — here are her punishments
Former Equifax CIO Charged With Insider Trading in Data Breach
Lyft is teaming up with automotive industry giant Magna to develop self-driving cars — and getting a $200 million investment (MGA, GM, GOOGL)
Google, Apple face EU law on business practices
Disney Elevates Two Top Executives in Possible CEO Bake-Off
Nokia Oyj Stock Could Break out over the Next Two Years
Wall Street gains as White House plays down trade war chances
U.S. retail sales falter; inflation creeping higher