java - 如何使用 JSoup 抓取 Steam 市场-6ren

java - 如何使用 JSoup 抓取 Steam 市场

转载作者：行者123 更新时间：2023-12-01 12:37:19

我正在尝试使用 JSoup 抓取 Steam 市场，在使用 JSoup Cookbook 和本网站上其他问题的建议后，我陷入了困境。我可以成功解析单个页面，但我很难进入市场列表的第二页。

我知道后续页面上的 URL 是什么样子。第一页如下所示:

http://steamcommunity.com/market/search?q=m4a1

项目的第二页如下所示:

http://steamcommunity.com/market/search?q=m4a1#p2_default_desc

并且“页码”在网络浏览器中按预期合理增加(我使用 Chrome)。

所以我用这种方式构建了网络解析器，它总是返回看起来像原始结果的新查询的内容(前 10 项会一次又一次地显示，无论 URL 中的“页码”编号如何...

因此，我添加了 cookie 来传递 session ID(并且在一次迭代中，仅传回所有 cookie)，但这似乎也不起作用。有什么想法吗？

这是我正在使用的源代码。目前，它没有做任何花哨的事情，只是将项目描述和内容打印到屏幕上。

public class MarketWebParser {

    public static void main(String[] args) {

        MarketWebParser myParser = new MarketWebParser();
        myParser.parseMainUrl("http://steamcommunity.com/market/search?q=m4a1");
    }

    void parseMainUrl(String inputUrl) {
        try {
            int page = 1;
            int searchResultsEnd = 10;
            int searchResultsTotal = 1000;
            String sessionId;
            Map<String, String> steamCookies;
            Response res = Jsoup.connect(inputUrl).method(Method.POST).execute();
            steamCookies = res.cookies();
            sessionId = res.cookie("sessionid");
            System.out.println("Session: " + sessionId);

            while (searchResultsEnd != searchResultsTotal) {

                String pageAddition = "#p" + Integer.toString(page);
                String currUrl = inputUrl + pageAddition;
                System.out.println(currUrl);
                Document doc = Jsoup.connect(currUrl).cookie("sessionid",sessionId).get();
                Element content = doc.getElementById("searchResults");
                Elements marketLinks = content
                        .getElementsByClass("market_listing_row_link");
                for (Element link : marketLinks) {
                    System.out.println(link.text());
                }
                Element searchResultsTotalElem = doc
                        .getElementById("searchResults_total");
                searchResultsTotal = Integer.parseInt(searchResultsTotalElem
                        .text().replaceAll("[^\\d]", ""));
                page++;


                System.out.println("Progress: " + searchResultsEnd + " of "
                        + searchResultsTotal);
                searchResultsEnd = 10 * page;
                Thread.sleep(1000);
            }

        } catch (IOException e) {
            e.printStackTrace();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }

    }
}

这是该网页的源代码:(您会注意到没有指向下一页的链接的 URL - 要查看网页的该部分，请搜索“searchResults_total”。按钮设置就在这之后，但没有什么可以让我 Hook ...)。请注意，我删除了一些我认为对于减少字符数没有用的标题部分...

<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8" />
<title>Steam Community Market :: Showing results for: &quot;m4a1&quot;</title>

<!-- main body -->
<div id="BG_bottom">
    <div class="market_search_results_header">
        <h2 class="market_search_results_title">Showing results for:</h2>
                    <div style="display: inline-block; margin: 10px 0;">
                                    <a href="http://steamcommunity.com/market/search?" class="market_searchedForTerm">
                        &quot;m4a1&quot;<span class="removeIcon"></span>
                    </a>

                            </div>
            </div>
    <div id="mainContents">
        <div id="sideBar">
            <div id="findItems">
                <div class="market_search_sidebar_contents">
                    <h2 class="market_section_title">
                        Find Items                  </h2>
                    <div class="marker_search_sidebar_section_tip">
                        Looking for a specific item?                    </div>
                    <div class="market_search_box_container">
                        <form id="market_search" accept-charset="UTF8" action="http://steamcommunity.com/market/search" method="GET" >
                                                        <span class="game_button">
                                <span class="game_button_edge game_button_left"></span>
                                <span class="game_button_contents">
                                    <input class="filter_search_box market_search_sidebar_search_box" type="text" id="findItemsSearchBox" value="m4a1" placeholder="Search" name="q" autocomplete="off" tabindex="1" />
                                    <input class="market_search_submit_button" id="findItemsSearchSubmit" type="submit" value="Submit" tabindex="3" />
                                </span>
                                <span class="game_button_edge game_button_right"></span>
                                <span class="game_button_preload"></span>
                            </span>
                            <br>
                            <div style="text-align: right; padding-right: 15px; margin-top: 0.25em;">
                                <div id="market_search_advanced_show" class="btn_black btn_details btn_small" onclick="ShowAdvancedSearchOptions();">
                                    <span>
                                        <span style="float: left;">Show advanced options...</span>
                                        <span class="btn_details_arrow down"></span>
                                    </span>
                                </div>
                            </div>
                        </form>
                    </div>
                </div>
            </div>
        </div>
        <div id="searchResults" class="market_page_left">
            <div id="searchResultsTable" class="market_content_block market_home_listing_table market_home_main_listing_table market_listing_table market_listing_table_active">
                <div id="searchResultsRows">
                        <div class="market_listing_table_header">
        <div class="market_listing_right_cell market_listing_their_price market_sortable_column" data-sorttype="price">PRICE<span class="market_sort_arrow" style="display:none;"></span></div>
        <div class="market_listing_right_cell market_listing_num_listings market_sortable_column" data-sorttype="quantity">QUANTITY<span class="market_sort_arrow" style="display:none;"></span></div>
                <div class="market_sortable_column" data-sorttype="name"><span class="market_listing_header_namespacer"></span>NAME<span class="market_sort_arrow" style="display:none;"></span></div>
    </div>

<a class="market_listing_row_link" href="http://steamcommunity.com/market/listings/730/M4A1-S%20%7C%20Nitro%20%28Battle-Scarred%29?filter=m4a1" id="resultlink_0">
    <div class="market_listing_row market_recent_listing_row market_listing_searchresult" id="result_0">
                <img id="result_0_image" src="http://steamcommunity-a.akamaihd.net/economy/image/fWFc82js0fmoRAP-qOIPu5THSWqfSmTELLqcUywGkijVjZYMUrsm1j-9xgEObwgfEh_nvjlWhNzZCveCDfIBj98xqodQ2CZknz52YOLkDyRufgHMAqVMY-Y1ywf-CCk17fhiV9Sz-a8VFlS65cSPLbZ9ONofScfWXKCOY1uu6xluiKhcLpff9C7m3yvsPjtcDRru_2IHyuGZ-uw8OqQ-aC8/62fx62f" style="border-color: #D2D2D2;" class="market_listing_item_img" alt="" />      
                <div class="market_listing_right_cell market_listing_their_price">
            <span class="market_table_value">
                Starting at:<br/>
                <span style="color:white">&#36;3.05</span>
            </span>
            <span class="market_arrow_down" style="display: none"></span>
            <span class="market_arrow_up" style="display: none"></span>
        </div>
        <div class="market_listing_right_cell market_listing_num_listings">
            <span class="market_table_value">
                <span class="market_listing_num_listings_qty">65</span>
            </span>
        </div>

                <div class="market_listing_item_name_block">
            <span id="result_0_name" class="market_listing_item_name" style="color: #D2D2D2;">M4A1-S | Nitro (Battle-Scarred)</span>
            <br/>
            <span class="market_listing_game_name">Counter-Strike: Global Offensive</span>
        </div>
    </div>
</a>

<a class="market_listing_row_link" href="http://steamcommunity.com/market/listings/730/M4A1-S%20%7C%20Nitro%20%28Field-Tested%29?filter=m4a1" id="resultlink_1">
    <div class="market_listing_row market_recent_listing_row market_listing_searchresult" id="result_1">
                <img id="result_1_image" src="http://steamcommunity-a.akamaihd.net/economy/image/fWFc82js0fmoRAP-qOIPu5THSWqfSmTELLqcUywGkijVjZYMUrsm1j-9xgEObwgfEh_nvjlWhNzZCveCDfIBj98xqodQ2CZknz52YOLkDyRufgHMAqVMY-Y1ywf-CCk17fhiV9Sz-a8VFlG64NuDbq4qNtpKGpHSX_TSbw36v01s0_deep3Y9Cntj3i9OG0DXhPo8mgBkeaOpPI11cbT2t9i/62fx62f" style="border-color: #D2D2D2;" class="market_listing_item_img" alt="" />     
                <div class="market_listing_right_cell market_listing_their_price">
            <span class="market_table_value">
                Starting at:<br/>
                <span style="color:white">&#36;3.58</span>
            </span>
            <span class="market_arrow_down" style="display: none"></span>
            <span class="market_arrow_up" style="display: none"></span>
        </div>
        <div class="market_listing_right_cell market_listing_num_listings">
            <span class="market_table_value">
                <span class="market_listing_num_listings_qty">183</span>
            </span>
        </div>

                <div class="market_listing_item_name_block">
            <span id="result_1_name" class="market_listing_item_name" style="color: #D2D2D2;">M4A1-S | Nitro (Field-Tested)</span>
            <br/>
            <span class="market_listing_game_name">Counter-Strike: Global Offensive</span>
        </div>
    </div>
</a>

<a class="market_listing_row_link" href="http://steamcommunity.com/market/listings/730/M4A1-S%20%7C%20Cyrex%20%28Field-Tested%29?filter=m4a1" id="resultlink_2">
    <div class="market_listing_row market_recent_listing_row market_listing_searchresult" id="result_2">
                <img id="result_2_image" src="http://steamcommunity-a.akamaihd.net/economy/image/fWFc82js0fmoRAP-qOIPu5THSWqfSmTELLqcUywGkijVjZYMUrsm1j-9xgEObwgfEh_nvjlWhNzZCveCDfIBj98xqodQ2CZknz52YOLkDyRufgHMAqVMY_YvywW4CHYh18R6RtKuyLYDLVWq6ZzFZrUqMItNHpbRUqCOYg-rvEMx1qdcLseIpy253Cy4Mz9fUxDi-ToHhqbZ7WBrCDQm/62fx62f" style="border-color: #D2D2D2;" class="market_listing_item_img" alt="" />     
                <div class="market_listing_right_cell market_listing_their_price">
            <span class="market_table_value">
                Starting at:<br/>
                <span style="color:white">&#36;58.34</span>
            </span>
            <span class="market_arrow_down" style="display: none"></span>
            <span class="market_arrow_up" style="display: none"></span>
        </div>
        <div class="market_listing_right_cell market_listing_num_listings">
            <span class="market_table_value">
                <span class="market_listing_num_listings_qty">19</span>
            </span>
        </div>

                <div class="market_listing_item_name_block">
            <span id="result_2_name" class="market_listing_item_name" style="color: #D2D2D2;">M4A1-S | Cyrex (Field-Tested)</span>
            <br/>
            <span class="market_listing_game_name">Counter-Strike: Global Offensive</span>
        </div>
    </div>
</a>

<a class="market_listing_row_link" href="http://steamcommunity.com/market/listings/730/M4A1-S%20%7C%20Nitro%20%28Well-Worn%29?filter=m4a1" id="resultlink_3">
    <div class="market_listing_row market_recent_listing_row market_listing_searchresult" id="result_3">
                <img id="result_3_image" src="http://steamcommunity-a.akamaihd.net/economy/image/fWFc82js0fmoRAP-qOIPu5THSWqfSmTELLqcUywGkijVjZYMUrsm1j-9xgEObwgfEh_nvjlWhNzZCveCDfIBj98xqodQ2CZknz52YOLkDyRufgHMAqVMY-Y1ywf-CCk17fhiV9Sz-a8VFlG64NuDbq4qNtpKGpHSX_TSbw36v01s0_deep3Y9Cntj3i9OG0DXhPo8mgBkeaOpPI11cbT2t9i/62fx62f" style="border-color: #D2D2D2;" class="market_listing_item_img" alt="" />     
                <div class="market_listing_right_cell market_listing_their_price">
            <span class="market_table_value">
                Starting at:<br/>
                <span style="color:white">&#36;3.17</span>
            </span>
            <span class="market_arrow_down" style="display: none"></span>
            <span class="market_arrow_up" style="display: none"></span>
        </div>
        <div class="market_listing_right_cell market_listing_num_listings">
            <span class="market_table_value">
                <span class="market_listing_num_listings_qty">64</span>
            </span>
        </div>

                <div class="market_listing_item_name_block">
            <span id="result_3_name" class="market_listing_item_name" style="color: #D2D2D2;">M4A1-S | Nitro (Well-Worn)</span>
            <br/>
            <span class="market_listing_game_name">Counter-Strike: Global Offensive</span>
        </div>
    </div>
</a>

<a class="market_listing_row_link" href="http://steamcommunity.com/market/listings/730/M4A1-S%20%7C%20Guardian%20%28Minimal%20Wear%29?filter=m4a1" id="resultlink_4">
    <div class="market_listing_row market_recent_listing_row market_listing_searchresult" id="result_4">
                <img id="result_4_image" src="http://steamcommunity-a.akamaihd.net/economy/image/fWFc82js0fmoRAP-qOIPu5THSWqfSmTELLqcUywGkijVjZYMUrsm1j-9xgEObwgfEh_nvjlWhNzZCveCDfIBj98xqodQ2CZknz52YOLkDyRufgHMAqVMY_YvywW4CHZ_-_hmWNKx9rUSFlC249qCLbF9Y4oeH8DWWvCCbgn06xprhaJfe8CPqHm5iSrrOjwNCkfi8msBnuKZ-uw8ccIhRqM/62fx62f" style="border-color: #D2D2D2;" class="market_listing_item_img" alt="" />      
                <div class="market_listing_right_cell market_listing_their_price">
            <span class="market_table_value">
                Starting at:<br/>
                <span style="color:white">&#36;4.61</span>
            </span>
            <span class="market_arrow_down" style="display: none"></span>
            <span class="market_arrow_up" style="display: none"></span>
        </div>
        <div class="market_listing_right_cell market_listing_num_listings">
            <span class="market_table_value">
                <span class="market_listing_num_listings_qty">151</span>
            </span>
        </div>

                <div class="market_listing_item_name_block">
            <span id="result_4_name" class="market_listing_item_name" style="color: #D2D2D2;">M4A1-S | Guardian (Minimal Wear)</span>
            <br/>
            <span class="market_listing_game_name">Counter-Strike: Global Offensive</span>
        </div>
    </div>
</a>

<a class="market_listing_row_link" href="http://steamcommunity.com/market/listings/730/M4A1-S%20%7C%20Guardian%20%28Field-Tested%29?filter=m4a1" id="resultlink_5">
    <div class="market_listing_row market_recent_listing_row market_listing_searchresult" id="result_5">
                <img id="result_5_image" src="http://steamcommunity-a.akamaihd.net/economy/image/fWFc82js0fmoRAP-qOIPu5THSWqfSmTELLqcUywGkijVjZYMUrsm1j-9xgEObwgfEh_nvjlWhNzZCveCDfIBj98xqodQ2CZknz52YOLkDyRufgHMAqVMY_YvywW4CHZ_-_hmWNKx9rUSFlG64NuDbq4qN4seF8jZWvXSYgqru0g-hKlfJpKNp3npiym6aDtZUxa_rmIMmeXWpPI11YiABWgk/62fx62f" style="border-color: #D2D2D2;" class="market_listing_item_img" alt="" />     
                <div class="market_listing_right_cell market_listing_their_price">
            <span class="market_table_value">
                Starting at:<br/>
                <span style="color:white">&#36;3.97</span>
            </span>
            <span class="market_arrow_down" style="display: none"></span>
            <span class="market_arrow_up" style="display: none"></span>
        </div>
        <div class="market_listing_right_cell market_listing_num_listings">
            <span class="market_table_value">
                <span class="market_listing_num_listings_qty">135</span>
            </span>
        </div>

                <div class="market_listing_item_name_block">
            <span id="result_5_name" class="market_listing_item_name" style="color: #D2D2D2;">M4A1-S | Guardian (Field-Tested)</span>
            <br/>
            <span class="market_listing_game_name">Counter-Strike: Global Offensive</span>
        </div>
    </div>
</a>

<a class="market_listing_row_link" href="http://steamcommunity.com/market/listings/730/M4A1-S%20%7C%20Guardian%20%28Factory%20New%29?filter=m4a1" id="resultlink_6">
    <div class="market_listing_row market_recent_listing_row market_listing_searchresult" id="result_6">
                <img id="result_6_image" src="http://steamcommunity-a.akamaihd.net/economy/image/fWFc82js0fmoRAP-qOIPu5THSWqfSmTELLqcUywGkijVjZYMUrsm1j-9xgEObwgfEh_nvjlWhNzZCveCDfIBj98xqodQ2CZknz52YOLkDyRufgHMAqVMY_YvywW4CHZ_-_hmWNKx9rUSFlC249qCLbF9Y4oeH8DWWvCCbgn06xprhaJfe8CPqHm5iSrrOjwNCkfi8msBnuKZ-uw8ccIhRqM/62fx62f" style="border-color: #D2D2D2;" class="market_listing_item_img" alt="" />      
                <div class="market_listing_right_cell market_listing_their_price">
            <span class="market_table_value">
                Starting at:<br/>
                <span style="color:white">&#36;6.30</span>
            </span>
            <span class="market_arrow_down" style="display: none"></span>
            <span class="market_arrow_up" style="display: none"></span>
        </div>
        <div class="market_listing_right_cell market_listing_num_listings">
            <span class="market_table_value">
                <span class="market_listing_num_listings_qty">118</span>
            </span>
        </div>

                <div class="market_listing_item_name_block">
            <span id="result_6_name" class="market_listing_item_name" style="color: #D2D2D2;">M4A1-S | Guardian (Factory New)</span>
            <br/>
            <span class="market_listing_game_name">Counter-Strike: Global Offensive</span>
        </div>
    </div>
</a>

<a class="market_listing_row_link" href="http://steamcommunity.com/market/listings/730/M4A1-S%20%7C%20Nitro%20%28Factory%20New%29?filter=m4a1" id="resultlink_7">
    <div class="market_listing_row market_recent_listing_row market_listing_searchresult" id="result_7">
                <img id="result_7_image" src="http://steamcommunity-a.akamaihd.net/economy/image/fWFc82js0fmoRAP-qOIPu5THSWqfSmTELLqcUywGkijVjZYMUrsm1j-9xgEObwgfEh_nvjlWhNzZCveCDfIBj98xqodQ2CZknz52YOLkDyRufgHMAqVMY-Y1ywf-CCk17fhiV9Sz-a8VFlC249qCLbYuNY5JF5OECfOAYgir7k840_ILLcPcqCzpji7oP2xYXhvr-T8FzuSZ-uw8GGkTN28/62fx62f" style="border-color: #D2D2D2;" class="market_listing_item_img" alt="" />      
                <div class="market_listing_right_cell market_listing_their_price">
            <span class="market_table_value">
                Starting at:<br/>
                <span style="color:white">&#36;13.26</span>
            </span>
            <span class="market_arrow_down" style="display: none"></span>
            <span class="market_arrow_up" style="display: none"></span>
        </div>
        <div class="market_listing_right_cell market_listing_num_listings">
            <span class="market_table_value">
                <span class="market_listing_num_listings_qty">31</span>
            </span>
        </div>

                <div class="market_listing_item_name_block">
            <span id="result_7_name" class="market_listing_item_name" style="color: #D2D2D2;">M4A1-S | Nitro (Factory New)</span>
            <br/>
            <span class="market_listing_game_name">Counter-Strike: Global Offensive</span>
        </div>
    </div>
</a>

<a class="market_listing_row_link" href="http://steamcommunity.com/market/listings/730/M4A1-S%20%7C%20Nitro%20%28Minimal%20Wear%29?filter=m4a1" id="resultlink_8">
    <div class="market_listing_row market_recent_listing_row market_listing_searchresult" id="result_8">
                <img id="result_8_image" src="http://steamcommunity-a.akamaihd.net/economy/image/fWFc82js0fmoRAP-qOIPu5THSWqfSmTELLqcUywGkijVjZYMUrsm1j-9xgEObwgfEh_nvjlWhNzZCveCDfIBj98xqodQ2CZknz52YOLkDyRufgHMAqVMY-Y1ywf-CCk17fhiV9Sz-a8VFlC249qCLbYuNY5JF5OECfOAYgir7k840_ILLcPcqCzpji7oP2xYXhvr-T8FzuSZ-uw8GGkTN28/62fx62f" style="border-color: #D2D2D2;" class="market_listing_item_img" alt="" />      
                <div class="market_listing_right_cell market_listing_their_price">
            <span class="market_table_value">
                Starting at:<br/>
                <span style="color:white">&#36;6.90</span>
            </span>
            <span class="market_arrow_down" style="display: none"></span>
            <span class="market_arrow_up" style="display: none"></span>
        </div>
        <div class="market_listing_right_cell market_listing_num_listings">
            <span class="market_table_value">
                <span class="market_listing_num_listings_qty">85</span>
            </span>
        </div>

                <div class="market_listing_item_name_block">
            <span id="result_8_name" class="market_listing_item_name" style="color: #D2D2D2;">M4A1-S | Nitro (Minimal Wear)</span>
            <br/>
            <span class="market_listing_game_name">Counter-Strike: Global Offensive</span>
        </div>
    </div>
</a>

<a class="market_listing_row_link" href="http://steamcommunity.com/market/listings/730/M4A1-S%20%7C%20Guardian%20%28Well-Worn%29?filter=m4a1" id="resultlink_9">
    <div class="market_listing_row market_recent_listing_row market_listing_searchresult" id="result_9">
                <img id="result_9_image" src="http://steamcommunity-a.akamaihd.net/economy/image/fWFc82js0fmoRAP-qOIPu5THSWqfSmTELLqcUywGkijVjZYMUrsm1j-9xgEObwgfEh_nvjlWhNzZCveCDfIBj98xqodQ2CZknz52YOLkDyRufgHMAqVMY_YvywW4CHZ_-_hmWNKx9rUSFlG64NuDbq4qN4seF8jZWvXSYgqru0g-hKlfJpKNp3npiym6aDtZUxa_rmIMmeXWpPI11YiABWgk/62fx62f" style="border-color: #D2D2D2;" class="market_listing_item_img" alt="" />     
                <div class="market_listing_right_cell market_listing_their_price">
            <span class="market_table_value">
                Starting at:<br/>
                <span style="color:white">&#36;4.52</span>
            </span>
            <span class="market_arrow_down" style="display: none"></span>
            <span class="market_arrow_up" style="display: none"></span>
        </div>
        <div class="market_listing_right_cell market_listing_num_listings">
            <span class="market_table_value">
                <span class="market_listing_num_listings_qty">33</span>
            </span>
        </div>

                <div class="market_listing_item_name_block">
            <span id="result_9_name" class="market_listing_item_name" style="color: #D2D2D2;">M4A1-S | Guardian (Well-Worn)</span>
            <br/>
            <span class="market_listing_game_name">Counter-Strike: Global Offensive</span>
        </div>
    </div>
</a>
                </div>

        <div id="searchResults_ctn" class="market_paging" style="">
            <div class="market_paging_summary">
                Showing <span id="searchResults_start">1</span>-<span id="searchResults_end">10</span> of <span id="searchResults_total">364</span> results         </div>
            <div class="market_paging_controls" id="searchResults_controls">
                <span id="searchResults_btn_prev" class="pagebtn">&lt;</span>
                <span id="searchResults_links"></span>
                <span id="searchResults_btn_next" class="pagebtn">&gt;</span>
            </div>
            <div style="clear: both;"></div>
        </div>
                </div>
        </div>
        <div style="clear:both"></div>
    </div>
    <br />
</div></div>
<div id="market_advancedsearch_dialog" class="market_modal_dialog" style="display: none;">
    <div>
        <form id="market_advanced_search" accept-charset="UTF8" action="http://steamcommunity.com/market/search" method="GET" >
            <div id="market_advancedsearch_left">
                    <input class="filter_search_box market_search_sidebar_search_box" type="text" id="advancedSearchBox" value="" placeholder="Search" name="q" autocomplete="off" tabindex="1" />
                    <input class="market_search_submit_button" id="advancedSearchSubmit" type="submit" value="Submit" tabindex="3" />
                    <div id="market_advancedsearch_options">
                        <input class="market_search_checkbox" id="market_advanced_searchdescriptions_checkbox" type="checkbox" name="descriptions" value="1" title="By default, the Community Market will only look for matches in the items' names and types. If you check this box, all of the text in the items' descriptions will be searched too. Searching for matches in descriptions may cause your search to take more time to complete." tabindex="2" />
                        <label for="market_advanced_searchdescriptions_checkbox" title="By default, the Community Market will only look for matches in the items' names and types. If you check this box, all of the text in the items' descriptions will be searched too. Searching for matches in descriptions may cause your search to take more time to complete.">Include descriptions in search</label>
                    </div>
            </div>
            <div style="overflow: hidden">
                <div class="market_app_selector">
                    <div class="appselect" id="market_advancedsearch_appselect" onclick="ShowMenu( 'market_advancedsearch_appselect', 'market_advancedsearch_appselect_options', 'left' );">
                        <div id="market_advancedsearch_appselect_activeapp" class="option">
                            <div id="app_option_0_selected" class="popup_item market_advancedsearch_appname">
                                <span>All Games</span>
                            </div>
                        </div>
                        <div id="appselect_arrow"></div>
                    </div>
                    <div id="market_advancedsearch_appselect_options" class="popup_block_new" style="display: none;">
                        <div id="market_advancedsearch_appselect_options_apps" class="popup_body popup_menu">
                            <div id="app_option_0" class="popup_item popup_menu_item market_advancedsearch_appname">
                                <span>All Games</span>
                            </div>
                                                            <div id="app_option_238460" class="popup_item popup_menu_item market_advancedsearch_appname" data-appid="238460">
                                    <img src="http://cdn.akamai.steamstatic.com/steamcommunity/public/images/apps/238460/2f258aaff583d797812cdcf24830d5992f48733b.jpg" alt="BattleBlock Theater" /><span>BattleBlock Theater</span>
                                </div>
                                                            <div id="app_option_730" class="popup_item popup_menu_item market_advancedsearch_appname" data-appid="730">
                                    <img src="http://cdn.akamai.steamstatic.com/steamcommunity/public/images/apps/730/69f7ebe2735c366c65c0b33dae00e12dc40edbe4.jpg" alt="Counter-Strike: Global Offensive" /><span>Counter-Strike: Global Offensive</span>
                                </div>
                                                            <div id="app_option_570" class="popup_item popup_menu_item market_advancedsearch_appname" data-appid="570">
                                    <img src="http://cdn.akamai.steamstatic.com/steamcommunity/public/images/apps/570/0bbb630d63262dd66d2fdd0f7d37e8661a410075.jpg" alt="Dota 2" /><span>Dota 2</span>
                                </div>
                                                            <div id="app_option_238960" class="popup_item popup_menu_item market_advancedsearch_appname" data-appid="238960">
                                    <img src="http://cdn.akamai.steamstatic.com/steamcommunity/public/images/apps/238960/1110764aac57ac28d7ffd8c43071c75d5482a9c9.jpg" alt="Path of Exile" /><span>Path of Exile</span>
                                </div>
                                                            <div id="app_option_251970" class="popup_item popup_menu_item market_advancedsearch_appname" data-appid="251970">
                                    <img src="http://cdn.akamai.steamstatic.com/steamcommunity/public/images/apps/251970/cee4aeadcf73b7b00018c535fa3b850bfcdb9982.jpg" alt="Sins of a Dark Age" /><span>Sins of a Dark Age</span>
                                </div>
                                                            <div id="app_option_753" class="popup_item popup_menu_item market_advancedsearch_appname" data-appid="753">
                                    <img src="http://cdn.akamai.steamstatic.com/steamcommunity/public/images/apps/753/135dc1ac1cd9763dfc8ad52f4e880d2ac058a36c.jpg" alt="Steam" /><span>Steam</span>
                                </div>
                                                            <div id="app_option_440" class="popup_item popup_menu_item market_advancedsearch_appname" data-appid="440">
                                    <img src="http://cdn.akamai.steamstatic.com/steamcommunity/public/images/apps/440/e3f595a92552da3d664ad00277fad2107345f743.jpg" alt="Team Fortress 2" /><span>Team Fortress 2</span>
                                </div>
                                                            <div id="app_option_230410" class="popup_item popup_menu_item market_advancedsearch_appname" data-appid="230410">
                                    <img src="http://cdn.akamai.steamstatic.com/steamcommunity/public/images/apps/230410/2e271fb410bb8be5536a24d4114b42f9e5c91176.jpg" alt="Warframe" /><span>Warframe</span>
                                </div>
                                                    </div>
                    </div>
                </div>
                <div id="market_advancedsearch_filters" class="market_advancedsearch_filters">
                    Pick a game from the drop-down list above to see the available filters.             </div>
            </div>
        </form>
    </div>

    <div class="market_advancedsearch_bottombuttons">
        <a href="javascript:ResetSearchFilters();">Reset to default</a>
        <a href="javascript:document.forms['market_advanced_search'].submit();" class="item_market_action_button item_market_action_button_green nodisable">
            <span class="item_market_action_button_edge item_market_action_button_left"></span>
            <span class="item_market_action_button_contents">
                Search          </span>
            <span class="item_market_action_button_edge item_market_action_button_right"></span>
            <span class="item_market_action_button_preload"></span>
        </a>
    </div>
</div>

谢谢!

最佳答案

url 中的哈希值告诉您 url 后面的部分实际上并未传输到服务器，它只是站点内部引用。所以你这样就不走运了。但是，如果您查看该站点并分析一下站点结构和网络流量(打开开发人员工具 -> 网络面板并触发下一页的加载)，您会发现下一个结果是使用以下 url 获取的:

steamcommunity.com/market/search/render/?query=m4a1&start=10&count=10&search_descriptions=0&sort_column=default&sort_dir=desc

这会返回一个 json 字符串，您可以使用 json 库解析该字符串。我没有深入研究这一点，但这是应该可行的路线。

关于java - 如何使用 JSoup 抓取 Steam 市场，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/25472391/

文章推荐： ajax - 客户端 AJAX 调用错误的一般处理

文章推荐： ruby-on-rails-4 - 使用 Rails ar-octopus gem 执行事务

文章推荐： java - Jsoup - 它需要多少 KB？

css - 从浏览器中提取光标图像(抓取、抓取...)
如本answer所述，如果浏览器不支持 e，可以设置后备游标。 G。光标:抓取；。我现在的问题是获取这些图像。在我的驱动器上本地搜索“.cur”只给了我系统光标，其中 grab.cur 和 grab
python - Instagram 抓取
以下代码在计算机上运行以从 Instagram 帐户中抓取数据。当我尝试在 VPS 服务器上使用它时，我被重定向到 Instagram 登录页面，因此脚本不起作用。为什么当我在电脑上或服务器上时，I
ruby - 抓取，我的查询参数错误
我在使用 Ruby 和 Mechanize 将 POST 查询传递到站点的网站上。访问站点的查询基于 firebug，如下所示 param.PrdNo=-1¶m.Type=Prop¶m
ruby - 抓取，跟随分页链接
我正在尝试抓取一个具有多个页面结果的网站，例如“1、2、3、4、5...”。每个分页号都是到另一个页面的链接，我需要抓取每个页面。到目前为止，我想出了这个: while lien = page.l
c# - 抓取 htmlagilitypack
我正在使用 HtmlAgilityPack 在 C# Asp.Net 中执行 Scraping，到目前为止，我在从多个 Web 执行 Scratch 时没有遇到问题，但是，尝试弹出以下代码时出现错误
javascript - 抓取 :after content
如果我有一个 css 文件做这样的事情 #foo:after{content:"bar;} ，有没有办法用 javascript 获取 :after 的内容？获取父元素的内容只返回 #foo 元素的内
javascript - 客户端页面调用/抓取？
问题是这样的: 我有一个 Web 应用程序 - 一个经常更改的通知系统 - 在一系列本地计算机上运行。该应用程序每隔几秒刷新一次以显示新信息。计算机仅显示信息，没有键盘或任何输入设备。问题是，如果与
c# - 抓取/模拟浏览帮助
我想制作一个程序来模拟用户浏览网站和点击链接。必须启用 Cookie 和 javascript。我已经在 python 中成功地做到了这一点，但我想把它写成一种可编译的语言(python ide 不会
python - 抓取/忽略空项目
我制作了这个小机器人，它通过搜索参数列表进行处理。它工作正常，直到页面上有几个结果: product_prices_euros 给出了一半为空的项目列表。因此，当我与 product_prices_c
python - 查找并统计网站上单词的匹配项 - 抓取
我需要找到一个单词的匹配项，例如: 在网上找到所有单词“学习”https://www.georgetown.edu/(结果:4个字)(您可以看到它按CTRL + F并搜索) 我有我的 Python 代
python - 抓取\蜘蛛防护
有一个站点\资源提供一些一般统计信息以及搜索工具的界面。这种搜索操作成本高昂，因此我想限制频繁且连续(即自动)的搜索请求(来自人，而不是来自搜索引擎)。我相信有很多现有的技术和框架可以执行一些情报抓
Python HTML 抓取
这并不是真正的抓取，我只是想在网页中找到类具有特定值的 URL。例如: 我想获取 href 值。关于如何做到这一点的任何想法？也许正则表达式？你能发布一些示例代码吗？我猜 html 抓取库，比如 B
python - 如何从无限滚动网站上抓取所有内容？抓取
我正在使用 scrapy。我正在使用的网站具有无限滚动功能。该网站有很多帖子，但我只抓取了 13 个。如何抓取剩余的帖子？这是我的代码: class exampleSpider(scrapy.
python - 无法通过 BeautifulSoup 抓取
我正在尝试从这个 website 中抓取图像和新闻 url .我定义的标签是 root_tag=["div", {"class":"ngp_col ngp_col-bottom-gutter-2 ng
java - 抓取 Cloudflare 站点
关闭。这个问题需要更多focused .它目前不接受答案。想改进这个问题吗？更新问题，使其只关注一个问题 editing this post . 关闭上个月。 Improve this ques
python - 对本地文件夹中的所有文件重复 BeautifulSoup 抓取
我在几个文件夹中有数千个 html 文件，我想从评论中提取数据并将其放入 csv 文件中。这将允许我为项目格式化和清理它。例如，我在这个文件夹中有 640 个 html 文件: D:\My Web S
c# - 通过 VIEWSTATE 抓取
我在编写用于抓取网页的实用程序时遇到了一个问题。我正在发送 POST 请求来检索数据，我模仿我正在抓取的网络行为(根据使用 fiddler 收集的信息)。我已经能够自动替换我的 POST 中除 V
ajax - Yandex AJAX 抓取
对于 Googlebot 的 AJAX 抓取，我在我的网站中使用“_escaped_fragment_”参数。现在我查看了 Yandex 对我网站的搜索结果。我看到搜索结果中不存在 AJAX 响应
Ruby Mechanize 抓取 ResponseCodeError
我正在尝试抓取网站的所有结果页面，它可以工作，但有时脚本会停止并显示此错误: 502 => Net::HTTPBadGateway for https://website.com/id/12/ --
python - 为什么这个网站不能用 bs4 抓取？
我是一个学习网络爬虫的初学者，由于某种原因我无法爬网this地点。当我在 Chrome 中检查它时，代码看起来不错，但是当我用 BeautifulSoup 阅读它时，它不再是可刮的。汤提到“谷歌分析”

行者123

个人简介

我是一名优秀的程序员,十分优秀！

作者热门文章

滴滴打车优惠券免费领取

全站热门文章

首页

博学

6Ren·AI

商城

java - 如何使用 JSoup 抓取 Steam 市场