python - 使用 set() 删除重复的 url python BeautifulSoup 会破坏 url

Question

在 python 中，我使用 BeautifulSoup 从项目的网站上抓取 url，一切都很好，直到我尝试通过将标签传递到设置对象中来删除重复项。标签被“炸散”。这是我的代码和打印结果的示例。

file = open('parsed_data.csv', 'w')

for link in soup.find_all('a', attrs={'href': re.compile("^http")}):

    soup_link = str(link)
    if soup_link.endswith('/') or soup_link.endswith('#'):
        soup_link = soup_link[-1]

    soup_link_unique = str(set(soup_link))

    print (soup_link)
    print (soup_link_unique)

    file.write(soup_link_unique)
    file.flush()
    file.close
```
Before passing into set object:
<a href="https://www.census.gov/en.html" onfocus="CensusSearchTypeahead.onSearchFocusBlur(false);" tabindex="2">
<img alt="United States Census Bureau" class="uscb-nav-image" src="https://www.census.gov/etc/designs/census/images/USCENSUS_IDENTITY_SOLO_White_2in_TM.svg" title="U.S. Census Bureau"/>
</a>

After passing into a set object:
{'I', 'S', '\n', 'C', '>', 'u', '"', '-', 'i', 'Y', 'L', 'M', 'p', '.', 'c', ')', 'B', '2', 't', 'N', '<', ' ', 'b', 'w', 'e', 'E', '/', 'O', ':', 'U', 'x', 'o', 'W', 'f', '(', 'l', 'D', 'F', 'g', 'd', '_', '=', 'n', 's', 'h', 'a', 'T', 'v', 'r', ';', 'm', 'y'}

Answer 1

在for循环之前创建一个集合，并使用方法add()将新元素添加到集合中:

soup_link_unique = set()

for link in soup.find_all('a', attrs={'href': re.compile("^http")}):    
    soup_link = str(link)
    if soup_link.endswith('/') or soup_link.endswith('#'):
        soup_link = soup_link[-1]   
    soup_link_unique.add(soup_link)

示例:

my_set = set('ABCDE')
print(my_set)
# {'E', 'D', 'C', 'B', 'A'}

与

my_set = set()
my_set.add('ABCDE')
print(my_set)
# {'ABCDE'}