python - 有没有一种方法可以使用 python 抓取跨越多个页面的表格的一列？

Question

我正在尝试从超过 46 页的表格中获取 ETF 的股票代码:

http://etfdb.com/type/region/north-america/us/#etfs&sort_name=assets_under_management&sort_order=desc&page=1

我的代码是

import bs4 as bs
import pickle
import requests

def save_ETF_tickers():
    resp = requests.get('http://etfdb.com/type/region/north-america/us/#etfs&sort_name=assets_under_management&sort_order=desc&page=1')
    soup = bs.BeautifulSoup(resp.text, "lxml")
    table = soup.find('table',{'class': 'table mm-mobile-table table-module2 table-default table-striped table-hover table-pagination'})
    tickers = []
    for row in table.findAll('tr')[1:26]:
        ticker = row.findAll('td')[0].text
        tickers.append(ticker)

    with open("ETFtickers.pickle", "wb") as f:
        pickle.dump(tickers, f)

    print(tickers)

    return tickers

save_ETF_tickers()

我知道这个只检查“page=1”，但我不知道如何从所有 46 个页面检索数据。

非常感谢您的帮助

Answer 1

您可以使用 etfdb-api Node.js 包:https://www.npmjs.com/package/etfdb-api

它为您提供:

• 分页
• 排序依据:年初至今的返回、价格、 Assets 管理规模、平均。音量等
• 排序顺序:DESC | ASC

以下是 JSON 响应示例:

  {
    "symbol": {
      "type": "link",
      "text": "VIXM",
      "url": "/etf/VIXM/"
    },
    "name": {
      "type": "link",
      "text": "ProShares VIX Mid-Term Futures ETF",
      "url": "/etf/VIXM/"
    },
    "mobile_title": "VIXM - ProShares VIX Mid-Term Futures ETF",
    "price": "$26.47",
    "assets": "$48.21",
    "average_volume": "69,873",
    "ytd": "25.15%",
    "overall_rating": {
      "type": "restricted",
      "url": "/members/join/"
    },
    "asset_class": "Volatility"
  },
  {
    "symbol": {
      "type": "link",
      "text": "DGBP",
      "url": "/etf/DGBP/"
    },
    "name": {
      "type": "link",
      "text": "VelocityShares Daily 4x Long USD vs GBP ETN",
      "url": "/etf/DGBP/"
    },
    "mobile_title": "DGBP - VelocityShares Daily 4x Long USD vs GBP ETN",
    "price": "$30.62",
    "assets": "$4.85",
    "average_volume": "1,038",
    "ytd": "25.13%",
    "overall_rating": {
      "type": "restricted",
      "url": "/members/join/"
    },
    "asset_class": "Currency"
  }

免责声明:我是作者。 :)