python - 如何在 Flask 中并行化任务？

Question

我正在向我的 Flask 服务器发送一个 XHR 请求，以便在网络上执行多个 ping

资源

def get(self, site_id):
    …
    for printer in printers:
        hostname = printer['hostname']
        response[site_id][hostname] = network_utils.ping(hostname)

    return response

平

在 shell.execute 下面我使用 subprocess.check_output运行 native ping:

def ping(hostname):
    command = ['ping', '-c', '1', '-W', '1', '-q', hostname]

    response = shell.execute(command)
    return output_parser.ping(response['results'])

输出

{
    "test-site": {
        "avg": 0.093, "max": 0.093, "mdev": 0.0, "min": 0.093,
        "1.1.1.1": { "avg": null, "max": null, "mdev": null, "min": null},
        "1.2.3.4": { "avg": null, "max": null, "mdev": null, "min": null},
        "127.0.0.1": { "avg": 0.061, "max": 0.061, "mdev": 0.0, "min": 0.061}
    }
}

问题

ping 是按顺序运行的，这使得请求非常慢(几十秒，我怎样才能加快速度？

Answer 1

听起来最好的选择是线程，因为您的问题是I/O 限制。我正在使用 Semaphore限制为 5 个线程。

我将响应字典发送到 ping 字典是线程安全的，但你应该阅读 this如果您考虑更复杂的事情。

def get(self, site_id):
    …
    semaphore = threading.Semaphore(5)
    threads = []

    for printer in printers:
        hostname = printer['hostname']
        threads.append(threading.Thread(target=network_utils.ping,
                          args=(semaphore, response, site_id, hostname)))

    # Start and wait to all threads to finish
    map(lambda t: t.start(), threads)
    map(lambda t: t.join(), threads)

    return response

def ping(semaphore, response, site_id, hostname):
    semaphore.acquire()

    command = ['ping', '-c', '1', '-W', '1', '-q', hostname]
    response = shell.execute(command)
    ping_data = output_parser.ping(response['results'])

    response[site_id][hostname] = ping_data

    semaphore.release()