python - 如何使用子进程拦截 "pipenv install"类似微调器的输出消息？

Question

我正在尝试读取“pipenv install”(pipenv==2018.11.26，Python 3.6.0)命令的输出，该命令在生成输出时通过 subprocess.Popen 运行，即不是在整个过程结束时，因为根据必须下载的数据量、连接速度等，可能需要很长时间。

我正在打印所有 stdout 和 stderr 消息并为它们添加前缀，但我仍然无法理解这种“微调器”消息:“[==] 正在创建虚拟环境...”来自何处.

这是我正在运行的完整代码

import subprocess
import threading

cmd = ['pipenv', 'install','--ignore-pipfile']
cwd = 'test'

def run_cmd(cmd,cwd):
    popen = subprocess.Popen(cmd, cwd=cwd, stdout=subprocess.PIPE, stderr=subprocess.PIPE,bufsize=1)

    a = threading.Thread(target=printTh,args=("stdout",iter(popen.stdout.readline, b"")))
    a.start()

    b = threading.Thread(target=printTh,args=("stderr",iter(popen.stderr.readline, b"")))
    b.start()

    while popen.poll() is not None:
        a.join()
        b.join()

def printTh(pipe_name,iter):
    for line in iter:
        print(pipe_name+"->"+line.rstrip().decode("utf-8"), end = "\r\n",flush =True)


run_cmd(cmd,cwd)

在我的 gui 控制台上，我可以看到除了微调器“[===] Creating virtual environment..”之外的所有内容都有一个前缀，该消息既不属于 stderr 也不属于 stdout:

控制台打印:

stderr->Creating a virtualenv for this project…
stderr->Pipfile: C:\Users\ahadu\test\Pipfile
stderr->Using C:/Program Files (x86)/Anaconda3/python.exe (3.6.0) to create virtualenv…
[ ===] Creating virtual environment...Running virtualenv with interpreter C:/Program Files (x86)/Anaconda3/python.exe
stderr->Already using interpreter C:\Program Files (x86)\Anaconda3\python.exe
stderr->Using base prefix 'C:\\Program Files (x86)\\Anaconda3'
stderr->  No LICENSE.txt / LICENSE found in source
stderr->New python executable in C:\Users\ahadu\test\.venv\Scripts\python.exe
stderr->Installing setuptools, pip, wheel...
stderr->done.
stderr->
stderr-Successfully created virtual environment!
stderr->Virtualenv location: C:\Users\ahadu\test\.venv
stdout->Installing dependencies from Pipfile.lock (d34422)…
stdout->To activate this project's virtualenv, run pipenv shell.
stdout->Alternatively, run a command inside the virtualenv with pipenv run.

Process finished with exit code 0

这条消息是从哪里来的？这也是可能需要一些时间的地方，因此在生成此消息时无法阅读它会破坏整个工作。

换句话说，我期望在控制台上看到以下打印:

some-source->[= ] Creating virtual environment...
some-source->[ =] Creating virtual environment...
some-source->[= ] Creating virtual environment...
some-source->[ =] Creating virtual environment...

直到它完成。

有谁知道这个问题的原因以及如何解决？

Answer 1

我相信这是因为，对于那些微调输出，pipenv 在写入下一行之前将流 (stdout/stderr) 重置回行首。因此，您的前缀被清除。

最初我以为你只需要设置 PIPENV_NOSPIN和 PIPENV_HIDE_EMOJIS运行脚本之前的环境变量。虽然这会禁用微调器和表情符号，但某些行仍然没有前缀。 (此外，您不会看到任何需要时间的操作的进展):

temp$ export PIPENV_NOSPIN=1
temp$ export PIPENV_HIDE_EMOJIS=1
temp$ python3.8 test.py
...
stderr>>>>>Pipfile.lock not found, creating...
stderr>>>>>Locking [dev-packages] dependencies...
stderr>>>>>Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
Success!>>>
stderr>>>>>Updated Pipfile.lock (49bf85)!
stdout>>>>>Installing dependencies from Pipfile.lock (49bf85)...
stdout>>>>>To activate this project's virtualenv, run pipenv shell.
stdout>>>>>Alternatively, run a command inside the virtualenv with pipenv run.

如果我们按照“解决依赖关系...”日志到 venv_resolve_deps function in utils.py , 然后按照 write method in spin.py ，我们会看到这样的几行:

stdout.write(decode_output(u"\r", target_stream=stdout))

现在编写 "\r" 可能会导致这样的结果:

>>> print('abc\rdef')
def

清除先前输出的位置。由于您的前缀在开头，因此也会被清除。请注意，我在这里可能错了，这就是原因，无法真正理解这些行是如何打印出来的，但这是我能找到/想到的唯一解释。

现在，我已经通过简单地将 text=True 传递给 Popen 来让您的脚本正常工作。调用，这意味着:

stdin, stdout and stderr will be opened in text mode using the encoding and errors specified in the call or the defaults for io.TextIOWrapper

def run_cmd(cmd, cwd):
    popen = subprocess.Popen(
        cmd, cwd=cwd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, bufsize=1,
        text=True)  # <==== add param here

    a = threading.Thread(
        target=printTh,
        args=("stdout", iter(popen.stdout.readline, "")))  # <===== remove 'b'
    a.start()

    b = threading.Thread(
        target=printTh,
        args=("stderr", iter(popen.stderr.readline, "")))  # <===== remove 'b'
    b.start()

    while popen.poll() is not None:
        a.join()
        b.join()

def printTh(pipe_name, iter):
    for line in iter:
        #      =================== remove .decode ==============
        print(pipe_name + ">>>>>" + line.rstrip(), end="\r\n", flush=True)

run_cmd(cmd, cwd)

结果似乎是你想要的:

stderr>>>>>⠋ Creating virtual environment...
stderr>>>>>⠙ Creating virtual environment...
stderr>>>>>⠹ Creating virtual environment...
stderr>>>>>⠸ Creating virtual environment...
stderr>>>>>⠼ Creating virtual environment...
...
stderr>>>>>
stderr>>>>✔ Successfully created virtual environment!
stderr>>>>>Virtualenv location: /Users/me/.venvs/temp-9JdZcEvf
stderr>>>>>Creating a Pipfile for this project...
stdout>>>>>Installing flask...
stderr>>>>>
stderr>>>>>⠋ Installing...
stderr>>>>>⠙ Installing flask...
stderr>>>>>⠹ Installing flask...
stderr>>>>>⠋ Installing flask...
stderr>>>>>⠙ Installing flask...
stderr>>>>>⠹ Installing flask...
stderr>>>>>⠸ Installing flask...
stderr>>>>>⠼ Installing flask...
stderr>>>>>Adding flask to Pipfile's [packages]...
stderr>>>>✔ Installation Succeeded
stderr>>>>>Pipfile.lock not found, creating...
stderr>>>>>Locking [dev-packages] dependencies...
stderr>>>>>Locking [packages] dependencies...
stderr>>>>>
stderr>>>>>⠋ Locking...
stderr>>>>>Building requirements...
stderr>>>>>
stderr>>>>>Resolving dependencies...
stderr>>>>>
stderr>>>>>⠙ Locking...
stderr>>>>>⠹ Locking...
stderr>>>>>⠸ Locking...
stderr>>>>>⠙ Locking...
stderr>>>>>⠦ Locking...
stderr>>>>>⠧ Locking...
stderr>>>>>⠇ Locking..✔ Success!
stderr>>>>>Updated Pipfile.lock (9536c4)!
stdout>>>>>Installing dependencies from Pipfile.lock (9536c4)...
stdout>>>>>To activate this project's virtualenv, run pipenv shell.
stdout>>>>>Alternatively, run a command inside the virtualenv with pipenv run.

测试:

• pipenv，版本 2020.11.15
• python 3.8
• macOS 10.15.7，带有 bash 5.x 的终端