python - 如何制作一个具有并行运行命令行工具功能的 python 模块(不使用 if __name__ == '__main__' : so it's importable)?

Question

我想制作一个带有便利函数的 python 模块，以便在 Windows 上使用 Python 3.7 并行运行命令。 (对于 az cli 命令)

我想要一个函数:

• 易于使用:只需将命令列表作为字符串传递，然后让它们并行执行。
• 让我看看命令生成的输出。
• 在 python 库中使用构建
• 在 Windows 和 Linux 上同样有效(Python 多重处理使用 fork()，而 Windows 没有 fork()，因此有时多重处理代码可以在 Linux 上运行，但不能在 Windows 上运行。)
• 可以制作成可导入的模块以提供更大的便利。

这出奇地困难，我想也许这在旧版本的 python 中是不可能的？ (我看到几个 2-8 岁的问答说你必须使用 if __name__==__main__: 来完成并行处理，但我发现在制作可导入模块时，它并不能以一致可预测的方式工作。

def removeExtraLinesFromString(inputstring):
    stringtoreturn = ""
    for line in inputstring.split("\n"):
        if len(line.strip()) > 0: #Only add non empty lines to the stringtoreturn
            stringtoreturn = stringtoreturn + line
    return stringtoreturn


def runCmd(cmd): #string of a command passed in here
    from subprocess import run, PIPE
    stringtoreturn = str( run(cmd, shell=True, stdout=PIPE).stdout.decode('utf-8') )
    stringtoreturn = removeExtraLinesFromString(stringtoreturn)
    return stringtoreturn


def exampleOfParrallelCommands():
    if __name__ == '__main__': #I don't like this method, because it doesn't work when imported, refractoring attempts lead to infinite loops and unexpected behavior.
        from multiprocessing import Pool
        cmd = "python -c \"import time;time.sleep(5);print('5 seconds have passed')\""
        cmds = []
        for i in range(12):  #If this were running in series it'd take at least a minute to sleep 5 seconds 12 times
            cmds.append(cmd)
        with Pool(processes=len(cmds)) as pool:
            results = pool.map(runCmd, cmds) #results is a list of cmd output
        print(results[0])
        print(results[1])
        return results

当我尝试将其作为模块导入时，它不起作用(因为 if 语句而导致)，所以我尝试重写代码以移动 if 语句，我想我删除了它一次，这导致我的计算机进入循环直到我关闭程序。还有一次，我能够将模块导入到另一个 python 程序中，但为了使其工作，我必须添加 __name__ == '__main__'这非常直观。

我几乎放弃了，但是经过 2 天的搜索大量 python 网站和 SO 帖子后，在这个问答中看到用户 jfs 的代码后，我终于想出了如何做到这一点( Python: execute cat subprocess in parallel )，我修改了他的代码，所以它' d 更适合我的问题的答案。

Answer 1

工具箱.py

def removeExtraLinesFromString(inputstring):
    stringtoreturn = ""
    for line in inputstring.split("\n"):
        if len(line.strip()) > 0: #Only add non empty lines to the stringtoreturn
            stringtoreturn = stringtoreturn + line
    return stringtoreturn


def runCmd(cmd): #string of a command passed in here
    from subprocess import run, PIPE
    stringtoreturn = str( run(cmd, shell=True, stdout=PIPE).stdout.decode('utf-8') )
    stringtoreturn = removeExtraLinesFromString(stringtoreturn)
    return stringtoreturn


def runParallelCmds(listofcommands): 
    from multiprocessing.dummy import Pool #thread pool
    from subprocess import Popen, PIPE, STDOUT
    listofprocesses = [Popen(listofcommands[i], shell=True,stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True) for i in range(len(listofcommands))] 
    #Python calls this list comprehension, it's a way of making a list
    def get_outputs(process): #MultiProcess Thread Pooling require you to map to a function, thus defining a function.
        return process.communicate()[0] #process is object of type subprocess.Popen
    outputs = Pool(len(listofcommands)).map(get_outputs, listofprocesses) #outputs is a list of bytes (which is a type of string)
    listofoutputstrings = []
    for i in range( len(listofcommands) ):
        outputasstring = removeExtraLinesFromString(  outputs[i].decode('utf-8')  ) #.decode('utf-8') converts bytes to string
        listofoutputstrings.append( outputasstring )
    return listofoutputstrings

main.py

from toolbox import runCmd #(cmd)
from toolbox import runParallelCmds #(listofcommands)

listofcommands = []
cmd = "ping -n 2 localhost"
listofcommands.append(cmd)
cmd = "python -c \"import time;time.sleep(5);print('5 seconds have passed')\""
for i in range(12):
    listofcommands.append(cmd) # If 12 processes each sleep 5 seconds, this taking less than 1 minute proves parrallel processing

outputs = runParallelCmds(listofcommands)
print(outputs[0])
print(outputs[1])

输出:

Pinging neokylesPC [::1] with 32 bytes of data: Reply from ::1: time<1ms Reply from ::1: time<1ms Ping statistics for ::1: Packets: Sent = 2, Received = 2, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms

5 seconds have passed