Python scrapy 蜘蛛

Question

我想使用这个过滤器从网站 http://www.quoka.de/immobilien/bueros-gewerbeflaechen 抓取数据:

<a class="t-bld" rel="nofollow" href="javascript:qsn.set('classtype','of',1);">nur Angebote</a>

如何使用scrapy设置这个过滤器？

Answer 1

您可以使用Beautifulsoup和urllib2解析特定的网站。这是您想要根据您编写的过滤器解析或抓取的数据的 python 实现。

from BeautifulSoup import BeautifulSoup
import urllib2

def main1(website):
    data_list = []
    web =urllib2.urlopen(website).read()
    soup = BeautifulSoup(web)
    description = soup.findAll('a', attrs={'rel':'nofollow'})
    for de in description:
        data_list.append(de.text)
    return data_list

print main1("http://www.quoka.de/immobilien/bueros-gewerbeflaechen")

如果你想解析其他数据，比如下面的描述:

def main(website):
    data_list = []
    web =urllib2.urlopen(website).read()
    soup = BeautifulSoup(web)
    description = soup.findAll('div', attrs={'class':'description'})
    for de in description:
        data_list.append(de.text)
    return data_list

print main("http://www.quoka.de/immobilien/bueros-gewerbeflaechen") #this is the data of each section