python - 如何在没有页面指示的情况下迭代 API 查询中的页面？

Question

我已经使用这个 JSON 查询进行了 api 调用:

import requests 
import json
import pandas as pd

url = ("https://api.meetup.com/2/groups?zip=b1+1aa&offset=0&format=json&lon=-1.89999997616&category_id=34&photo-host=public&page=500&radius=200.0&fields=&lat=52.4799995422&order=id&desc=false&sig_id=243750775&sig=ed49065d620a34c10e1f0f91dd58da2e36547af1")

data = requests.get(url).json()
df = pd.io.json.json_normalize(data['results'])

这样就变成了一个数据框，但是，我还有 5 个 url 页面要查询，如下所示:

url2 = ("https://api.meetup.com/2/groups?zip=b1+1aa&offset=1&format=json&lon=-1.89999997616&category_id=34&photo-host=public&page=500&radius=200.0&fields=&lat=52.4799995422&order=id&desc=false&sig_id=243750775&sig=ed49065d620a34c10e1f0f91dd58da2e36547af1")

和 url3 类似，只是通过 offset=2 等更改页面是关键。

我想知道是否可以使用 for 循环遍历所有这些页面。

Answer 1

首先不要在 url 中硬编码查询字符串，而是将查询数据作为字典传递给 request，即:

url = "https://api.meetup.com/2/groups"
querydict = {
   "zip":"b1+1aa",
   "offset": 0,
   "format":"json",
   "lon":-1.89999997616,
   "category_id": 34,
   "photo-host":"public",
   # etc
   }

response = requests.get(url, params=querydict)

然后你所要做的就是循环，直到你拥有所有你想要的内容，在每次迭代中更新 querydict["offset"]:

url = "https://api.meetup.com/2/groups"
querydict = {
   "zip":"b1+1aa",
   "offset": 0,
   "format":"json",
   "lon":-1.89999997616,
   "category_id": 34,
   "photo-host":"public",
   # etc
   }

while True: 
    response = requests.get(url, params=querydict)
    # check your response status, check the json data
    # etc
    if we_have_enough(response):
        break
    # ok let's fetch next page
    querydict["offset"] += 1