python-3.x - 使用请求从登录后面的网页中提取表？

Question

我正在练习 Requests 库，在使用 session 首次登录后尝试从网页中获取表格时遇到了问题。

我正在尝试从 this 这样的页面获取“交易历史”表格, 但为了查看表格中的实际交易，您必须先登录页面。我有登录凭据，我尝试使用下面的代码，它似乎可以使用 200 状态代码正常登录，但是当我尝试从该 URL 中提取表格时，我的请求输出仍然说我需要登录以查看表格。

我使用的代码如下所示:

import requests
import requests
from bs4 import BeautifulSoup

login_url = "https://www.mql5.com/en/auth_login"
payload = {
    'Login': "MY_USERNAME_HERE",
    'Password': "MY_PASSWORD_HERE"
}
table_url = "https://www.mql5.com/en/signals/811921?source=Site+Signals+MT5+Table#!tab=history"

# Start the session
session = requests.Session()

# Post the payload to the site to log in
s = session.post(login_url, data=payload)
print(s.status_code)

# Navigate to the next page and scrape the data
html = session.get(table_url)
print(html.status_code)

# Get the soup of the page
soup = BeautifulSoup(html.text, 'html.parser')

# Find the table
table = soup.find("div", {"id": "tab_content_history"})
table = table.find("div", {"class": "tab__content__item"})
print(table)

我的打印输出如下所示:

200
200
<div class="tab__content__item">
<div class="need-auth">To see trades in realtime, please <strong><a href="https://www.mql5.com/en/auth_login">log in</a></strong> or <strong><a href="https://www.mql5.com/en/auth_register">register</a></strong></div>
</div>

Login 和Password 是登录元素的name。让 session 正确登录我还缺少什么？我没有看到任何需要添加到有效负载中的标记，但我也可能是错的。我还能尝试什么？如果您没有登录此站点也没关系，如果您有使用请求 session 的经验，只需寻找其他尝试尝试的想法。可以从this page查看登录表单的页面检查.我知道网络抓取是不允许的，我只是将此站点用作公共(public)示例，因为我需要对工作项目使用类似的逻辑。谢谢!

Answer 1

试试这个:

import random
import time
import pandas as pd
from bs4 import BeautifulSoup
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import NoSuchElementException
from selenium.common.exceptions import StaleElementReferenceException

user_agent_list = [
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36',
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:90.0) Gecko/20100101 Firefox/90.0',
    'Mozilla/5.0 (Macintosh; Intel Mac OS X 11.5; rv:90.0) Gecko/20100101 Firefox/90.0',
    'Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36',
    'Mozilla/5.0 (Macintosh; Intel Mac OS X 11_5_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36',
    'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:90.0) Gecko/20100101 Firefox/90.0',
    'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36'
]

data = []

# set url
url = 'https://www.mql5.com/en/signals/811921?source=Site+Signals+MT5+Table#!tab=history'

# call open browser function
chrome_options = Options()
chrome_options.add_argument("--window-size=1920,1080")
chrome_options.headless = True
chrome_options.add_argument("--disable-dev-shm-usage")
chrome_options.add_argument("--no-sandbox")
prefs = {"profile.managed_default_content_settings.images": 2}
chrome_options.add_experimental_option("prefs", prefs)
userAgent = random.choice(user_agent_list)
chrome_options.add_argument(f'user-agent={userAgent}')
driver = webdriver.Chrome(ChromeDriverManager().install(), options=chrome_options)
driver.get(url)

# login to website
sign_in_link = driver.find_element_by_link_text('Log in')
sign_in_link.click()

username = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.ID, 'Login')))
password = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.ID, 'Password')))
username.send_keys('your_username')
password.send_keys('your_password')

driver.find_element_by_id("loginSubmit").click()
time.sleep(2)

innerHTML = driver.page_source
soup = BeautifulSoup(innerHTML, "html.parser")

# Find the table
table = soup.find("div", {"id": "tab_content_history"})
table = table.find("div", {"class": "tab__content__item"})
table_body = table.find('tbody')

rows = table_body.find_all('tr')

for row in rows:
    cols = row.find_all('td')
    cols = [ele.text.strip() for ele in cols]
    data.append([ele for ele in cols])


df = pd.DataFrame(data[1:])
df.columns = ['Time', 'Type', 'Volume', 'Symbol', 'Price', 'Time', 'Price', 'Commission', 'Swap', 'Profit']
print(df)

driver.quit()

---

输出:

====== WebDriver manager ======
Current google-chrome version is 93.0.4577
Get LATEST driver version for 93.0.4577
Driver [/home/user/.wdm/drivers/chromedriver/linux64/93.0.4577.63/chromedriver] found in cache
                Time  Type Volume  Symbol    Price              Time    Price Commission   Swap Profit
0   2021.09.17 09:56   Buy   0.03  NZDCAD  0.89502  2021.09.17 14:54  0.89623      -0.12          2.85
1   2021.09.17 02:58   Buy   0.03  NZDCAD  0.89570  2021.09.17 06:36  0.89625      -0.12          1.30
2   2021.09.16 14:17   Buy   0.03  NZDCAD  0.89626  2021.09.16 19:03  0.89749      -0.12          2.91
3   2021.09.16 12:48   Buy   0.03  NZDCAD  0.89714  2021.09.16 19:03  0.89749      -0.12          0.83
4   2021.09.16 04:56  Sell   0.03  NZDCAD  0.89857  2021.09.16 12:38  0.89777      -0.12          1.90
5   2021.09.16 01:31  Sell   0.03  AUDCAD  0.92665  2021.09.16 02:11  0.92650      -0.12          0.36
6   2021.09.15 14:32   Buy   0.03  NZDCAD  0.89842  2021.09.15 22:45  0.89896      -0.12  -0.51   1.28
7   2021.09.15 12:49   Buy   0.03  NZDCAD  0.89995  2021.09.15 22:45  0.89896      -0.12  -0.51  -2.35
8   2021.09.15 04:16  Sell   0.03  NZDCAD  0.89992  2021.09.15 12:35  0.89977      -0.12          0.35
9   2021.09.14 13:07  Sell   0.03  NZDCAD  0.90053  2021.09.14 14:12  0.90043      -0.12          0.24
10  2021.09.14 08:08  Sell   0.03  NZDCAD  0.90040  2021.09.14 10:16  0.89976      -0.12          1.52
11  2021.09.13 07:44  Sell   0.02  NZDCAD  0.90165  2021.09.13 09:42  0.90094      -0.08          1.12
12  2021.09.13 01:10  Sell   0.03  NZDCAD  0.90096  2021.09.13 06:30  0.89982      -0.12          2.70
13  2021.09.10 12:32  Sell   0.05  NZDCAD  0.90067  2021.09.10 12:58  0.90042      -0.20          0.99
14  2021.09.09 07:50  Sell   0.03  NZDCAD  0.90334  2021.09.09 16:59  0.89919      -0.12          9.86
15  2021.09.08 11:16  Sell   0.03  NZDCAD  0.90021  2021.09.09 16:59  0.89920      -0.12  -0.30   2.40
16  2021.09.07 16:20  Sell   0.03  NZDCAD  0.89673  2021.09.09 16:59  0.89914      -0.12  -0.40  -5.72
17  2021.09.07 11:07  Sell   0.03  NZDCAD  0.89637  2021.09.07 13:54  0.89522      -0.12          2.74
18  2021.09.03 16:54  Sell   0.03  NZDCAD  0.89541  2021.09.06 14:26  0.89438      -0.12  -0.10   2.47
19  2021.09.03 14:03  Sell   0.03  NZDCAD  0.89447  2021.09.03 14:49  0.89342      -0.12          2.52
20  2021.09.02 12:52  Sell   0.03  NZDCAD  0.89466  2021.09.02 16:43  0.89307      -0.12          3.80
21  2021.09.02 09:45  Sell   0.03  NZDCAD  0.89297  2021.09.02 16:43  0.89307      -0.12         -0.24
22  2021.09.02 06:21  Sell   0.03  NZDCAD  0.89244  2021.09.02 08:00  0.89140      -0.12          2.48
23  2021.08.31 11:29  Sell   0.03  NZDCAD  0.88831  2021.09.01 12:40  0.88704      -0.12  -0.10   3.03
24  2021.08.31 06:32  Sell   0.03  NZDCAD  0.88773  2021.09.01 12:40  0.88707      -0.12  -0.10   1.57
25  2021.08.31 03:02  Sell   0.03  NZDCAD  0.88654  2021.09.01 12:40  0.88707      -0.12  -0.10  -1.26
26  2021.08.30 02:31  Sell   0.03  NZDCAD  0.88391  2021.08.30 06:12  0.88335      -0.12          1.33
27  2021.08.27 16:47  Sell   0.03  NZDCAD  0.88475  2021.08.29 22:04  0.88437      -0.12  -0.10   0.90
28  2021.08.27 12:31  Sell   0.03  NZDCAD  0.88189  2021.08.27 13:22  0.88120      -0.12          1.63
29  2021.08.27 01:10  Sell   0.03  NZDCAD  0.88193  2021.08.27 03:31  0.88013      -0.12          4.26
30  2021.08.26 15:55  Sell   0.03  NZDCAD  0.88039  2021.08.27 03:31  0.88013      -0.12  -0.10   0.61
31  2021.08.26 14:42  Sell   0.03  NZDCAD  0.87928  2021.08.27 03:31  0.88013      -0.12  -0.10  -2.01
32  2021.08.25 15:13  Sell   0.02  NZDCAD  0.87859  2021.08.26 09:05  0.87712      -0.08  -0.20   2.33
33  2021.08.25 10:09  Sell   0.03  NZDCAD  0.87745  2021.08.26 09:05  0.87710      -0.12  -0.30   0.83
34  2021.08.25 07:58  Sell   0.03  NZDCAD  0.87632  2021.08.26 09:05  0.87712      -0.12  -0.30  -1.90
35  2021.08.24 08:45  Sell   0.03  NZDCAD  0.87599  2021.08.24 17:02  0.87533      -0.12          1.57
36  2021.08.23 04:20   Buy   0.03  NZDCAD  0.87396  2021.08.23 12:44  0.87461      -0.12          1.53
37  2021.08.20 13:20  Sell   0.03  NZDCAD  0.87990  2021.08.20 13:47  0.87896      -0.12          2.19
38  2021.08.20 12:08  Sell   0.03  NZDCAD  0.88049  2021.08.20 13:16  0.88043      -0.12          0.14
39  2021.08.20 04:32  Sell   0.03  NZDCAD  0.87886  2021.08.20 06:40  0.87781      -0.12          2.45
40  2021.08.18 13:30   Buy   0.03  NZDCAD  0.86860  2021.08.18 15:31  0.87046      -0.12          4.42
41  2021.08.18 12:35   Buy   0.03  NZDCAD  0.87085  2021.08.18 15:31  0.87049      -0.12         -0.85
42  2021.08.18 11:37  Sell   0.03  AUDNZD  1.05070  2021.08.18 12:00  1.05023      -0.12          0.97
43  2021.08.18 10:14   Buy   0.03  NZDCAD  0.87009  2021.08.18 11:28  0.87071      -0.12          1.47
44  2021.08.18 07:11   Buy   0.03  EURUSD  1.17214  2021.08.18 07:39  1.17277      -0.18          1.89
45  2021.08.18 05:13   Buy   0.03  EURUSD  1.17161  2021.08.18 06:59  1.17172      -0.18          0.33
46  2021.08.18 05:01  Sell   0.03  NZDCAD  0.87438  2021.08.18 05:55  0.87325      -0.12          2.69
47  2021.08.17 12:59   Buy   0.03  NZDCAD  0.87254  2021.08.18 01:13  0.87490      -0.12  -0.17   5.61

---

Make sure have selenium and BeautifulSoup installed in your system