python - 如何使用python icrawler爬取多个关键词

Question

我有一个包含很多关键字的数组:

array = ['table', 'chair', 'pen']

我想使用 python icrawler 从 Google 图片搜索中抓取 array 中每个项目的 5 张图片

这里是初始化:

from icrawler.builtin import GoogleImageCrawler

google_crawler = GoogleImageCrawler(
  parser_threads=2, 
  downloader_threads=4,
  storage={ 'root_dir': 'images' }
)

我使用循环来抓取每个项目:

for item in array:
  google_crawler.crawl(
    keyword=item, 
    offset=0, 
    max_num=5,
    min_size=(500, 500)
  )

但是，我收到错误日志:

  File "crawler.py", line 20, in <module>
    min_size=(500, 500)
  File "/home/user/opt/miniconda3/envs/pak/lib/python3.6/site-packages/icrawler/builtin/google.py", line 83, in crawl
    feeder_kwargs=feeder_kwargs, downloader_kwargs=downloader_kwargs)
  File "/home/user/opt/miniconda3/envs/pak/lib/python3.6/site-packages/icrawler/crawler.py", line 166, in crawl
    self.feeder.start(**feeder_kwargs)                                   
  File "/home/user/opt/miniconda3/envs/pak/lib/python3.6/site-packages/icrawler/utils/thread_pool.py", line 66, in start
    worker.start()                                                       
  File "/home/user/opt/miniconda3/envs/pak/lib/python3.6/threading.py", line 842, in start
    raise RuntimeError("threads can only be started once")
RuntimeError: threads can only be started once

看来我不能多次使用google_crawler.crawl。我该如何解决这个问题？

Answer 1

在最新版本中，你可以这样使用。

from icrawler.builtin import GoogleImageCrawler

google_crawler = GoogleImageCrawler(
    parser_threads=2,
    downloader_threads=4,
    storage={'root_dir': 'images'}
)

for keyword in ['cat', 'dog']:
    google_crawler.crawl(
        keyword=keyword, max_num=5, min_size=(500, 500), file_idx_offset='auto')
    # set `file_idx_offset` to 'auto' will prevent naming the 5 images
    # of dog from 000001.jpg to 000005.jpg, but naming it from 000006.jpg.

或者，如果您想将这些图像下载到不同的文件夹，您只需创建两个 GoogleImageCrawler 实例即可。

from icrawler.builtin import GoogleImageCrawler

for keyword in ['cat', 'dog']:
    google_crawler = GoogleImageCrawler(
        parser_threads=2,
        downloader_threads=4,
        storage={'root_dir': 'images/{}'.format(keword)}
    )
    google_crawler.crawl(
        keyword=keyword, max_num=5, min_size=(500, 500))