python - 将抓取列表添加到 Pandas Dataframe

Question

我正在尝试返回公司名称列表的 URL，然后更新 pandas 数据框以包含返回的 URL。

我正在导入所有依赖项并读取 .csv 文件:

import logging
import os
import pandas as pd
import re
from googlesearch import search

df = pd.read_csv('Building_Contractors_Stephen_V1.csv')

df.head()

df.head() results

然后我定义了一个获取所有 URL 的函数:

def get_urls(tag, n, language):
    urls = [url for url in search(tag, stop=n, lang=language)][:n]
    return urls

然后我在一个 url 上测试它:

test_return = get_urls(df.Hospital_Building_Contractors[0], 10, 'en') 
test_return     

返回 URL 列表:

['https://www.turnerconstruction.com/',
 'http://www.turnerconstruction.com/careers',
 'http://www.turnerconstruction.com/office-network',
 'http://www.turnerconstruction.com/about-us',
 'http://www.turnerconstruction.com/turner-university',
 'http://www.turnerconstruction.com/careers/jobs',
 'http://www.turnerconstruction.com/about-us/where-we-work',
 'https://en.wikipedia.org/wiki/Turner_Construction',
 'https://en.wikipedia.org/wiki/Turner_Construction#History',
 'https://en.wikipedia.org/wiki/Turner_Construction#Early_years']

我似乎无法弄清楚的是如何遍历列表中的所有项目，并将它们添加到数据框中的新列中。

这是我的代码:

i = 0
num = len(df.Hospital_Building_Contractors)
while i < num:
    get_urls(df.Hospital_Building_Contractors[i], 1, 'en')
    df.insert(1, "URL", urls, allow_duplicates=True) 
    i += 1

返回此错误:

---

NameError Traceback (most recent call last) in 17 while i < num: 18 get_urls(df.Hospital_Building_Contractors[i], 1, 'en') ---> 19 df.insert(1, "URL", urls, allow_duplicates=True) 20 i += 1 21

NameError: name 'urls' is not defined

我确信这是一个很容易解决的问题，但我很难过。我在 get_urls() 函数中定义了“urls”；所以我不确定为什么会收到此错误。

理想情况下，我会有这样的解决方案:

a = 0
num = len(df.Hospital_Building_Contractors)
while a < num:
    get_urls(df.Hospital_Building_Contractors[a], 1, 'en')
    df.insert(1, "URL", urls, allow_duplicates=True) 
    a += 1


b = 0
num = len(df.University_Building_Contractors)
while b < num:
    get_urls(df.University_Building_Contractors[b], 1, 'en')
    df.insert(3, "URL", urls, allow_duplicates=True) 
    b += 1


c = 0
num = len(df.Hospital_Building_Contractors)
while c < num:
    get_urls(df.Hospital_Building_Contractors[c], 1, 'en')
    df.insert(5, "URL", urls, allow_duplicates=True) 
    c += 1

这将遍历每个列表，找到 URL 并将它们添加到数据框。

Answer 1

将自定义 lambda 函数 与过滤器一起使用以仅处理字符串:

from googlesearch import search

df = pd.read_csv('Building_Contractors_Stephen_V1.csv')
#print (df)


def get_urls(tag, n, language):
    urls = [url for url in search(tag, stop=n, lang=language)][:n]
    return urls

#for only one top1 value
f = lambda x: next(iter(get_urls(x, 1, 'en') if isinstance(x, str) else []), 'no value')
#for multiple top values, eg. top3
#f = lambda x: get_urls(x, 3, 'en') if isinstance(x, str) else []
df['a'] = df.Hospital_Building_Contractors.apply(f)
df['b'] = df.University_Building_Contractors.apply(f)
df['c'] = df.Military_Contractors.apply(f)

---

print (df.tail())
    Hospital_Building_Contractors University_Building_Contractors  \
104                Progressive AE                             NaN   
105           Hellas Construction                             NaN   
106               Wight & Company                             NaN   
107               PWI Engineering                             NaN   
108   Cordogan Clark & Associates                             NaN   

    Military_Contractors                                                a  \
104                  NaN  https://www.linkedin.com/company/progressive-ae   
105                  NaN               http://www.hellasconstruction.com/   
106                  NaN                         https://www.wightco.com/   
107                  NaN                            http://www.pwius.com/   
108                  NaN                    http://www.cordoganclark.com/   

            b         c  
104  no value  no value  
105  no value  no value  
106  no value  no value  
107  no value  no value  
108  no value  no value