python - 抓取带有非结构化数据的 href 链接的表格

Question

我试图从出现在 webpage 上的表格中提取一些信息。 .我能够根据需要提取所有数据，但卡在抓取里面的 pdf 链接上。
这是我的源代码:
这是我目前工作的代码:

    import pandas as pd
    import requests
    from bs4 import BeautifulSoup
    
    hdr={'User-Agent':'Chrome/84.0.4147.135'}
    
    dfs=[]
    
    for page_number in range(0,10):
        http= "https://example.com/portal/ap_list_all.php?sPage={}".format(page_number)
    
        print('Downloading page %s...' % http)
        url= requests.get(http,headers=hdr)
        soup = BeautifulSoup(url.text, 'html.parser')
        table = soup.find('table')
        df_list= pd.read_html(url.text)
        df = pd.concat(df_list)
        df['Link'] = [link.get('href') for link in table.find_all('a')]
        dfs.append(df)
    
    final_df = pd.concat(dfs)
    final_df.to_csv('myscraper.csv',index=False,encoding='utf-8-sig')

但是，这就是我得到的:

我想要的结果如下，再加上附加的pdf链接列:

任何帮助，将不胜感激。非常感谢你。

Answer 1

您可以使用此示例如何将数据抓取到 Pandas 的数据框中并将其保存到 CSV 文件中:

import requests
import pandas as pd
from bs4 import BeautifulSoup


url = "https://ejudgment.kehakiman.gov.my/portal/ap_list_all.php?sPage={page}&sRecordsPerPage=20&sSortField=aph_ap_date&sSortOrder=DESC"

data = []
for page in range(0, 10):
    print("Getting page {}..".format(page))

    soup = BeautifulSoup(
        requests.get(url.format(page=page)).content, "html.parser"
    )

    headers = [
        h.get_text(strip=True)
        for h in soup.select("thead tr#trchk2 td, thead tr#trchk2 th")
    ]

    for row in soup.select('tbody tr[id^="trchk"]'):
        dic = {}
        for h, d in zip(headers, row.select("td")):
            dic[h] = d.get_text(strip=True, separator="\n")
        dic["Link"] = "https:" + row.find("a", target="_blank")["href"]
        data.append(dic)

df = pd.DataFrame(data)
print(df)
df.to_csv("data.csv", index=False)

打印:

        #                                         Nombor Kes                                              Pihak                                         Kata Kunci Tarikh Alasan Penghakiman                                  Hakim / Majistret                                            Dokumen                                               Link
0      1.  WA-42(R)(A)-4-11/2018; WA-42R-13-11/2018; WA-4...  PERAYU  : \r\n\t\t1. MOHD ARIF BIN AB RAHMAN\n...  Appellants found guilty of corruption in Sessi...                28/04/2021                           YA DATUK ASLAM ZAINUDDIN  Alasan Penghakiman\n1)\n28042021 judgment case...  https://ejudgment.kehakiman.gov.my/ks_builtin/...
1      1.                                   JA-22C-3-02/2020  PLAINTIF  : \r\n\t\tYPJ PLANTATIONS SDN BHD\n(...  Civil procedure - Application to set aside JID...                28/04/2021                      YA PUAN EVROL MARIETTE PETERS  Alasan Penghakiman\n1)\nja-22c-3-02-2020 (ypj ...  https://ejudgment.kehakiman.gov.my/ks_builtin/...
2      1.                             W-02(C)(A)-163-01/2020  PERAYU  : \r\n\t\tAHMAD ZAKI SDN BHD\n(COMPANY...                                         Ahmad Zaki                27/04/2021  YA DATO' ABU BAKAR BIN JAIS\nKorum:\nYA DATO' ...     Alasan Penghakiman\n1)\nahmad zaki sdn bhd.pdf  https://ejudgment.kehakiman.gov.my/ks_builtin/...
3      1.                                  WA-45B-24-01/2018  PENDAKWARAYA  : \r\n\t\tPN. ZAHIDA BINTI ZAKAR...                                             GROUND                26/04/2021                   YA TUAN COLLIN LAWRENCE SEQUERAH  Alasan Penghakiman\n1)\npp v muhammad khairuan...  https://ejudgment.kehakiman.gov.my/ks_builtin/...
4      1.                                      62-01-02/2019  PENDAKWARAYA  : \r\n\t\tPENDAKWARAYA\nTERTUDUH...  -seksyen 326 Kanun Keseksaan- cedera parah- kr...                26/04/2021                                TN MUSYIRI BIN PEET  Alasan Penghakiman\n1)\nalasan penghakiman thu...  https://ejudgment.kehakiman.gov.my/ks_builtin/...
5      1.                                   JA-25-40-12/2020  PEMOHON  : \r\n\t\tKIND ACTION (M) SDN BHD\n(N...                Leave for judicial review - granted                25/04/2021             YA DATO' FREDRICK INDRAN X.A. NICHOLAS  Alasan Penghakiman\n1)\nmtjbs2-ja-25-40-12-202...  https://ejudgment.kehakiman.gov.my/ks_builtin/...
6      1.                               WA-22NCC-618-12/2020  PLAINTIF  : \r\n\t\tTAN MAN HIN (NRIC NO.55082...  Summary Judgment application dismissed - wheth...                24/04/2021                              YA TUAN ONG CHEE KWAN  Alasan Penghakiman\n1)\nwa-22ncc-618-12-2020 t...  https://ejudgment.kehakiman.gov.my/ks_builtin/...
7      1.                               WA-22NCC-167-04/2020  PLAINTIF  : \r\n\t\tSMEB ASSET MANAGEMENT SDN....                                               O.14                23/04/2021            YA TUAN WAN MUHAMMAD AMIN BIN WAN YAHYA  Alasan Penghakiman\n1)\njudgment [sgd] (smeb) ...  https://ejudgment.kehakiman.gov.my/ks_builtin/...
8      1.                               WA-22NCC-401-08/2020  PLAINTIF  : \r\n\t\tKHEE SAN BERHAD &AMP 1 LAG...                                                  -                22/04/2021                         YA PUAN LIZA CHAN SOW KENG  Alasan Penghakiman\n1)\nkhee san dan 1 lagi v ...  https://ejudgment.kehakiman.gov.my/ks_builtin/...
9      1.                                  BA-12B-24-02/2020  PERAYU  : \r\n\t\tLETCHUMY A/P KRISHNAN\nRESPO...                          Letchumy v. Khairil Azraf                22/04/2021                                   TN TEE GEOK HOCK  Alasan Penghakiman\n1)\ngoj - [final] khairil ...  https://ejudgment.kehakiman.gov.my/ks_builtin/...
10     1.                                  BA-12B-92-06/2019  PERAYU  : \r\n\t\tKHAIRIL AZRAF BIN MOHD SEDEK...                           Khairil Azraf v Letchumy                22/04/2021                                   TN TEE GEOK HOCK  Alasan Penghakiman\n1)\ngoj - [final] khairil ...  https://ejudgment.kehakiman.gov.my/ks_builtin/...
11     1.                               JA-41(ORS)-2-11/2020  PERAYU  : \r\n\t\tMUHAMMAD BIN JAAFAR\nRESPOND...                 Seksyen 413 Kanun Tatacara Jenayah                22/04/2021                      YA DR. SHAHNAZ BINTI SULAIMAN    Alasan Penghakiman\n1)\nmuhammad bin jaafar.pdf  https://ejudgment.kehakiman.gov.my/ks_builtin/...

...

并保存 data.csv (来自 LibreOffice 的截图):