python - Twisted同时执行10个线程并等待结果

Question

编写一个程序来验证电子邮件语法列表和 MX 记录，因为阻塞编程非常耗时，我想异步或通过线程执行此操作，这是我的代码:

with open(file_path) as f:
    # check the status of file, if away then file pointer will be at the last index
    if (importState.status == ImportStateFile.STATUS_AWAY):
        f.seek(importState.fileIndex, 0)

    while True:
        # the number of emails to process is configurable 10 or 20
        emails = list(islice(f, app.config['NUMBER_EMAILS_TO_PROCESS']))
        if len(emails) == 0:
            break;

        importState.fileIndex = importState.fileIndex + len(''.join(emails))

        for email in emails:
            email = email.strip('''<>;,'\r\n ''').lower()
            d = threads.deferToThread(check_email, email)
            d.addCallback(save_email_status, email, importState)

        # set the number of emails processed 
        yield set_nbrs_emails_process(importState)

        # do an insert of all emails
        yield reactor.callFromThread(db.session.commit)

# set file status as success
yield finalize_import_file_state(importState)
reactor.callFromThread(reactor.stop)

查看电子邮件功能:

def check_email(email):
    pipe = subprocess.Popen(["./check_email", '--email=%s' % email], stdout=subprocess.PIPE)
    status = pipe.stdout.read()
    try:
        status = int(status)
    except ValueError:
        status = -1

    return status

我需要的是同时处理 10 封电子邮件并等待结果。

Answer 1

我不确定为什么您的示例代码中涉及线程。您不需要线程通过 Twisted 与电子邮件交互，也不需要同时执行此操作。

如果您有一个返回Deferred的异步函数，您只需调用它十次，十个不同的工作流就会并行进行:

for i in range(10):
    async_check_email_returning_deferred()

如果您想知道所有十个结果何时可用，您可以使用gatherResults:

from twisted.internet.defer import gatherResults
...
email_results = []
for i in range(10):
    email_results.append(async_check_mail_returning_deferred())
all_results = gatherResults(email_results)

all_results 是一个 Deferred，当 email_results 中的所有 Deferreds 已触发时(或者当其中第一个触发失败)。