python - 如何使用 Python 将标准输入/标准输出通过管道传输到 Perl 脚本

Question

这段 Python 代码可以很好地通过 Perl 脚本传输数据。

import subprocess
kw = {}
kw['executable'] = None
kw['shell'] = True
kw['stdin'] = None
kw['stdout'] = subprocess.PIPE
kw['stderr'] = subprocess.PIPE
args = ' '.join(['/usr/bin/perl','-w','/path/script.perl','<','/path/mydata'])
subproc = subprocess.Popen(args,**kw)
for line in iter(subproc.stdout.readline, ''):
    print line.rstrip().decode('UTF-8')

但是，它要求我首先将缓冲区保存到磁盘文件 (/path/mydata)。在 Python 代码中循环遍历数据并像这样逐行传递给子进程会更清晰:

import subprocess
kw = {}
kw['executable'] = '/usr/bin/perl'
kw['shell'] = False
kw['stderr'] = subprocess.PIPE
kw['stdin'] = subprocess.PIPE
kw['stdout'] = subprocess.PIPE
args = ['-w','/path/script.perl',]
subproc = subprocess.Popen(args,**kw)
f = codecs.open('/path/mydata','r','UTF-8')
for line in f:
    subproc.stdin.write('%s\n'%(line.strip().encode('UTF-8')))
    print line.strip()  ### code hangs after printing this ###
    for line in iter(subproc.stdout.readline, ''):
        print line.rstrip().decode('UTF-8')
subproc.terminate()
f.close()

在将第一行发送到子进程后，代码与 readline 一起挂起。我有其他可执行文件完美地使用了这个完全相同的代码。

我的数据文件可能非常大 (1.5 GB) 有没有办法在不保存到文件的情况下完成数据管道传输？我不想为了与其他系统兼容而重写 perl 脚本。

Answer 1

您的代码在以下行阻塞:

for line in iter(subproc.stdout.readline, ''):

因为此迭代可以终止的唯一方式是到达 EOF(文件结尾)时，这将在子进程终止时发生。您不想等到进程终止，但是，您只想等到它处理完发送给它的行。

此外，正如 Chris Morgan 已经指出的那样，您遇到了缓冲问题。另一个question on stackoverflow讨论如何使用子进程进行非阻塞读取。我已经针对您的问题对该问题的代码进行了快速而肮脏的改编:

def enqueue_output(out, queue):
    for line in iter(out.readline, ''):
        queue.put(line)
    out.close()

kw = {}
kw['executable'] = '/usr/bin/perl'
kw['shell'] = False
kw['stderr'] = subprocess.PIPE
kw['stdin'] = subprocess.PIPE
kw['stdout'] = subprocess.PIPE
args = ['-w','/path/script.perl',]
subproc = subprocess.Popen(args, **kw)
f = codecs.open('/path/mydata','r','UTF-8')
q = Queue.Queue()
t = threading.Thread(target = enqueue_output, args = (subproc.stdout, q))
t.daemon = True
t.start()
for line in f:
    subproc.stdin.write('%s\n'%(line.strip().encode('UTF-8')))
    print "Sent:", line.strip()  ### code hangs after printing this ###
    try:
        line = q.get_nowait()
    except Queue.Empty:
        pass
    else:
        print "Received:", line.rstrip().decode('UTF-8')

subproc.terminate()
f.close()

您很可能需要修改这段代码，但至少它不会阻塞。