python - BeautifulSoup 和请求不会使用 .findAll() 产生预期结果

Question

我一直在编写一段代码，用于从 Steam 市场(针对游戏 Unturned)检索元素列表及其相应价格。我正在使用 BeautifulSoup (bs4) 和请求库。到目前为止，这是我的代码:

for page_num in range(1,10):
website = 'http://steamcommunity.com/market/search?appid=304930#p'+str(page_num)+'_popular_desc'
r = requests.get(website)
doc = r.text.split('\n')
soup = BeautifulSoup(''.join(doc), "html.parser")

names = soup.findAll("span", { "class" : "market_listing_item_name" })
for item in range(len(names)):
    items.append(names[item].contents[0])

costs = soup.findAll("span", { "class" : "normal_price" })
for cost in range(len(costs)):
    prices.append(costs[cost].contents[0])

预期输出:

Festive Gift Present :  $0.32 USD
Halloween Gift Present :  $0.26 USD
Carbon Fiber Mystery Box :  $0.47 USD
Festive Hat :  $1.67 USD
Nuclear Matamorez :  $0.39 USD
... and so on

此代码的问题在于，它仅获取第一页的名称。如果我用不同的数字代替 page_num 手动键入 URL，它会更改页面，HTML 文档也会更改。但是，代码似乎并没有从第二页等获取结果。 requests 每次都获取正确的 URL，但 HTML 文档返回相同的 URL？

Answer 1

第 2、3 等页面是通过 ajax(或类似的)请求的，因此当您首次加载页面时源代码不存在。为了绕过这个，我们可以嗅探 ajax url 并直接解析源代码，在这种情况下，json 编码，即:

import json
from bs4 import BeautifulSoup
from urllib2 import urlopen
output = ""
items =[]
prices =[]
for page_num in range(0,100, 10): #
    start = page_num
    count = page_num + 10

    url = urlopen("http://steamcommunity.com/market/search/render/?query=&start={}&count={}&search_descriptions=0&sort_column=popular&sort_dir=desc&appid=304930".format(start, count))
    jsonCode = json.loads(url.read())
    output += jsonCode['results_html']

soup = BeautifulSoup(output, "html.parser")

names = soup.findAll("span", { "class" : "market_listing_item_name" })
for item in range(len(names)):
    items.append(names[item].contents[0])

costs = soup.findAll("span", { "class" : "normal_price" })
for cost in range(len(costs)):
    if "Starting at" not in costs[cost].contents[0]: # we just get the first price
        prices.append(costs[cost].contents[0])



print items
[u'Festive Gift Present', u'Halloween Gift Present', u'Hypertech Timberwolf', u'Holiday Scarf', u'Chill Honeybadger', etc...] 
print prices
[u'$0.34 USD', u'$0.28 USD', u'$1.77 USD', u'$0.31 USD', u'$0.65 USD', etc...]

---

PS:Steam 将在约 50 次请求后临时禁止您的 IP