python - 通过导入时间加快漂亮汤的速度(抓取太多不相关的数据)

Question

我的程序将正常运行并打印所需的输出，但是运行需要一分多钟的时间。它抓取整个页面的数据，然后在数据中搜索所需的信息。延迟在于抓取数据。基本上我正在寻找的是一种只抓取包含所需信息的页面的数据的方法，而不是从整个页面抓取所有不相关的数据并大大减慢进程。

我需要的信息是包含“$”和“LDK2-ENY10”的第一 block 数据，变量z用于仅抓取第一 block 数据，而不是后面所有不相关的数据。

import requests
from time import time
from bs4 import BeautifulSoup     
z = 0;
link = "http://yugiohprices.com/get_card_prices/Dark+Magician?_={}"
r = requests.get(link.format(int(time())))
soup = BeautifulSoup(r.content, "lxml")
rawdata = soup.find_all("td")
for thing in rawdata:    
  if "LDK2-ENY10" in str(thing) and "$" in str(thing) and z == 0:
    print thing; z = 1;

这是当前的输出，只要抓取这个输出就可能足够快了，这似乎有点多余，因为我只需要第 10 行( <b>$0.33</b> )、第 18 行( $0.77 )和第 29 行的信息( $5.28 )，但是此时我不在乎，只想让程序在不花 2-5 分钟的情况下运行..(30 秒或更短就太棒了)

如果我的帖子中有任何遗漏，请告诉我，我会更新。

<td style="width: 206px; padding-right: 10px" valign="top">
<table border="1" id="item_stats" style="margin-bottom: 10px">
<tr>
<td class="key" style="border: 1px solid #000; width: 94px">
              Lowest
              <a alt="How are lowest prices calculated?" href="http://blog.yugiohprices.com/post/90183367316/lowest-card-price-is-now-picked-using-ebay-listings" target="_blank" title="How are lowest prices calculated?">(?)</a>
</td>
<td style="border: 1px solid #000; text-align: center">
<p style="margin: 2px">
<b>$0.33</b>
</p>
</td>
</tr>
<tr>
<td class="key" style="border: 1px solid #000; width: 94px">Highest</td>
<td style="border: 1px solid #000; text-align: center">
<p style="margin: 2px">
                $5.28
              </p>
</td>
</tr>
<tr>
<td class="key" style="border: 1px solid #000; width: 94px">
              Average
              <a alt="How are average prices calculated?" href="http://blog.yugiohprices.com/post/54460976914/how-are-average-prices-calculated" target="_blank" title="How are average prices calculated?">(?)</a>
</td>
<td style="border: 1px solid #000; text-align: center">
<p style="margin: 2px">
                $0.77
              </p>
</td>
</tr>
</table>
<table border="1" id="item_stats">
<tr style="height: 22px">
<td class="key" style="border: 1px solid #000; width: 80px">
<p style="margin-top: -6px; margin-bottom: -4px; margin-left: 0px; margin-right: -6px; font-weight: normal">
<b>
                  Shift
                </b>
<br/>
                (24 Hours)
              </p>
</td>
<td style="border: 1px solid #000; text-align: center">
<p style="margin: 6px; margin-bottom: 9px">
<b style="color: red">
                  -9.41%
                </b>
</p>
</td>
</tr>
<tr>
<td class="key" style="border: 1px solid #000; width: 80px">
<p style="margin-top: -6px; margin-bottom: -4px; margin-left: 0px; margin-right: -6px; font-weight: normal">
<b>Shift</b>
<br/>
                (3 Days)
              </p>
</td>
<td style="border: 1px solid #000; text-align: center">
<p style="margin: 6px; margin-bottom: 9px">
<b style="color: red">
                  -2.53%
                </b>
</p>
</td>
</tr>
<tr>
<td class="key" style="border: 1px solid #000; width: 80px">
<p style="margin-top: -6px; margin-bottom: -4px; margin-left: 0px; margin-right: -6px; font-weight: normal">
<b>Shift</b>
<br/>
                (1 Week)
              </p>
</td>
<td style="border: 1px solid #000; text-align: center">
<p style="margin: 6px; margin-bottom: 9px">
<b style="color: red">
                  -9.41%
                </b>
</p>
</td>
</tr>
<tr>
<td class="key" style="border: 1px solid #000; width: 94px">
<p style="margin-top: -6px; margin-bottom: -4px; margin-left: 0px; margin-right: -6px; font-weight: normal">
<b>Shift</b>
<br/>
                (3 Weeks)
              </p>
</td>
<td style="border: 1px solid #000; text-align: center">
<p style="margin: 6px; margin-bottom: 9px">
<b>
                  0%
                </b>
</p>
</td>
</tr>
<tr>
<td class="key" style="border: 1px solid #000; width: 94px">
<p style="margin-top: -6px; margin-bottom: -4px; margin-left: 0px; margin-right: -6px; font-weight: normal">
<b>Shift</b>
<br/>
                (30 Days)
              </p>
</td>
<td style="border: 1px solid #000; text-align: center">
<p style="margin: 6px; margin-bottom: 9px">
<b>
                  0%
                </b>
</p>
</td>
</tr>
<tr>
<td class="key" style="border: 1px solid #000; width: 94px">
<p style="margin-top: -6px; margin-bottom: -4px; margin-left: 0px; margin-right: -6px; font-weight: normal">
<b>Shift</b>
<br/>
                (3 Months)
              </p>
</td>
<td style="border: 1px solid #000; text-align: center">
<p style="margin: 6px; margin-bottom: 9px">
<b>
                  0%
                </b>
</p>
</td>
</tr>
<tr>
<td class="key" style="border: 1px solid #000; width: 94px">
<p style="margin-top: -6px; margin-bottom: -4px; margin-left: 0px; margin-right: -6px; font-weight: normal">
<b>Shift</b>
<br/>
                (6 Months)
              </p>
</td>
<td style="border: 1px solid #000; text-align: center">
<p style="margin: 6px; margin-bottom: 9px">
<b>
                  0%
                </b>
</p>
</td>
</tr>
<tr>
<td class="key" style="border: 1px solid #000; width: 94px">
<p style="margin-top: -6px; margin-bottom: -4px; margin-left: 0px; margin-right: -6px; font-weight: normal">
<b>Shift</b>
<br/>
                (1 Year)
              </p>
</td>
<td style="border: 1px solid #000; text-align: center">
<p style="margin: 6px; margin-bottom: 9px">
<b>
                  0%
                </b>
</p>
</td>
</tr>
<tr>
<td class="key" colspan="2" style="text-align: center; border: 1px solid #000">
<a href="/price_history/LDK2-ENY10?rarity=Common" target="_blank">View History</a>
</td>
</tr>
</table>
<br/>
<div align="center">
<script async="" src="//pagead2.googlesyndication.com/pagead/js/adsbygoogle.js"></script>
<!-- Yugioh Prices Skyscraper -->
<ins class="adsbygoogle" data-ad-client="ca-pub-7333610178228936" data-ad-slot="9136249004" style="display:inline-block;width:160px;height:600px"></ins>
<script>
            (adsbygoogle = window.adsbygoogle || []).push({});
          </script>
</div>
</td>

Answer 1

首先，解析成为瓶颈的几率非常低 - 请重新检查该假设。

<小时/>

第一个合乎逻辑的事情是摆脱 z 变量，并在找到所需信息后简单地打破循环。这应该会对执行时间产生重大影响:

for thing in rawdata:    
    thing_html = str(thing)  # avoiding calling str() two times per iteration
    if "LDK2-ENY10" in thing_html and "$" in thing_html:
        print(thing)
        break

<小时/>

或者/和，您可以避免解析完整的页面源代码和 parse only a part of the document with the SoupStrainer 。大致如下:

from bs4 import BeautifulSoup, SoupStrainer

td_only = SoupStrainer("td")
soup = BeautifulSoup(r.content, "lxml", parse_only=td_only)

不过，鉴于树的大小相对较小，我怀疑“仅解析”方法是否会产生重大影响。

<小时/>

您可以尝试的另一件事是使用 PyPy 运行脚本解释器而不是常规的CPython。您需要从 lxml 切换到 html.parser 或 html5lib (或者您可以 install lxml from the fork )，但是，经过几次快速测试后，我可以看到，其性能优于 CPython + lxml。