python - 在 Django 中同时运行多个网络爬虫

Question

我为 Django 创建了一个名为crawl.py的自定义管理命令

我想让它同时运行两个爬虫。我的爬虫都是一个带有crawl()函数的对象，该函数运行无限循环并与orm交互。

def crawl(self):
    current_page = self.start_page
    while True:
        page_response = requests.get('http://magdeleine.co/browse/page/{}/'.format(current_page))
        page_soup = BeautifulSoup(page_response.text)
        image_links = [link["href"] for link in page_soup.find_all('a', {'class': 'photo-link'})]


        for image_link in image_links:
            response = requests.get(image_link)
            image_page_soup = BeautifulSoup(response.text)
            print('getting image source link')
            image_source_link = image_page_soup.find('a',{'class': 'download'})['href']

            #Get Tags
            print('getting tags')
            ul = image_page_soup.find('ul', {'class': 'tags'})
            tag_links = ul.find_all('a', {'rel':'tag'})
            tag_names = [tag_link.string for tag_link in tag_links]
            try:
                tag_names.remove('editor\'s pick')
            except:
                pass

            if not Image.objects.filter(url=image_source_link).exists():
                image = Image(url=image_source_link, origin="MG")
                print('creating thumbnail')
                image.create_thumb()
                image.save()
                # get or create a new tag for every element in the list
                for tag_name in tag_names:
                    tag, created = Tag.objects.get_or_create(name=tag_name)
                    image.tags.add(tag)

        current_page+=1
        print("end page")

在我的管理命令中

class Command(BaseCommand):

    def handle(self, *args, **options):
        pexel_crawler = PexelCrawler()
        pexel_crawler.crawl()

        magdeleine_crawler = MagdeleineCrawler()
        magdeleine_crawler.crawl()

我想同时运行两个 .crawl() 命令。另外，如果有人可以详细说明如何在生产与开发中完成此操作以及最佳解决方案。

Answer 1

您应该能够使用 gevent 来完成此操作，例如:

import gevent
from django.core.management.base import BaseCommand

class Command(BaseCommand):

    def handle(self, *args, **options):
        pexel_crawler = PexelCrawler()
        magdeleine_crawler = MagdeleineCrawler()
        pexel_job = gevent.spawn(pexel_crawler.crawl)
        magdeleine_job = gevent.spawn(magdeleine_crawler.crawl)
        gevent.joinall([pexel_job, magdeleine_job])

我相信这会起作用，并且只要两个爬网程序都在运行，管理命令就会在前台运行。不过我会很小心，因为如果这按预期工作，它将真正成为一个无限循环并且永远不会停止。