Python subprocess.popen 和 rdiff-backup

Question

我想在 python 中创建一个 rdiff-backup 包装程序，用于将 windows 机器备份到 linux 服务器。

我想在包装程序中处理 rdiff-backup 的输出。但是当使用子进程模块执行 rdiff-backup 并将 stdout 和 stderr 输送到包装器时，stdout 总是在管道的末端结束。

在不通过管道传输 stdout 和 stderr 的情况下使用子进程时，会在控制台中以正确的顺序打印。

我还注意到，当在本地使用 rdiff-backup 而不是使用 ssh 管道时，stdout 和 stderr 的顺序是正确的。 Rdiff-backup 还使用 subprocess.popen 打开 ssh session 并将数据通过管道传输到服务器。我怀疑由于某种原因，stdout 被阻塞，直到 ssh session 关闭。

这是我的代码，这是真实程序使用线程来监听标准输出的简化版本:

import sys
import subprocess

class Rdiffbackup(object):
    def __init__(self):
        #self.io_q = Queue()
        self.exe = 'F:\\workspace\\pysubprocess\\bin\\rdiff-backup\\rdiff-backup.exe'

        self.verbosity = '-v5'

        self.ssh_exe = './bin/openssh/bin/ssh'
        self.quiet = '-q'
        self.compression = '-C'
        self.port = '-p 5555'
        self.key = '-i ./keys/rdiffbackup'
        self.options = '-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null'
        self.remote_schema = self.ssh_exe +  ' ' + self.quiet + ' ' + self.compression + ' ' + self.port + ' ' + self.key + ' ' + self.options + ' %s rdiff-backup --server'  

    def start(self,source,dest):
        with subprocess.Popen([self.exe,self.verbosity,'--remote-schema',self.remote_schema,source,dest],stdout=subprocess.PIPE,stderr=subprocess.STDOUT) as self.proc:     
            for line in self.proc.stdout:
                sys.stdout.write(line.decode("utf-8"))      

if __name__ == '__main__':
    rdb = Rdiffbackup()
    source = "C:/Users/vdrmrt/Desktop/data"    
    dest = "vdrmrt@hostname::backup"
    rdb.start(source,dest)

输出:

-----------------------------------------------------------------
Detected abilities for source (read only) file system:
  Access control lists                         Off
  Extended attributes                          Off
  Windows access control lists                 On
  Case sensitivity                             Off
  Escape DOS devices                           Off
  Escape trailing spaces                       Off
  Mac OS X style resource forks                Off
  Mac OS X Finder information                  Off
-----------------------------------------------------------------
Unable to import win32security module. Windows ACLs
not supported by filesystem at backup/rdiff-backup-data/rdiff-backup.tmp.0
escape_dos_devices not required by filesystem at backup/rdiff-backup-data/rdiff-backup.tmp.0
-----------------------------------------------------------------
Detected abilities for destination (read/write) file system:
  Ownership changing                           Off
  Hard linking                                 On
  fsync() directories                          On
  Directory inc permissions                    On
  High-bit permissions                         On
  Symlink permissions                          Off
  Extended filenames                           On
  Windows reserved filenames                   Off
  Access control lists                         On
  Extended attributes                          On
  Windows access control lists                 Off
  Case sensitivity                             On
  Escape DOS devices                           Off
  Escape trailing spaces                       Off
  Mac OS X style resource forks                Off
  Mac OS X Finder information                  Off
-----------------------------------------------------------------
Backup: must_escape_dos_devices = 0
Using rdiff-backup version 1.2.8
Executing ./bin/openssh/bin/ssh -q -C -p 5555 -i ./keys/rdiffbackup -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null vdrmrt@hostname rdiff-backup --server
Hardlinks disabled by default on Windows
Unable to import module xattr.
Extended attributes not supported on filesystem at C:/Users/vdrmrt/Desktop/data
Unable to import module posix1e from pylibacl package.
POSIX ACLs not supported on filesystem at C:/Users/vdrmrt/Desktop/data
escape_dos_devices not required by filesystem at C:/Users/vdrmrt/Desktop/data
Symbolic links excluded by default on Windows
Starting increment operation C:/Users/vdrmrt/Desktop/data to backup

正确的输出:

Using rdiff-backup version 1.2.8
Executing ./bin/openssh/bin/ssh -q -C -p 5555 -i ./keys/rdiffbackup -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null vdrmrt@hostname rdiff-backup --server
Hardlinks disabled by default on Windows
Unable to import module xattr.
Extended attributes not supported on filesystem at C:/Users/vdrmrt/Desktop/data
Unable to import module posix1e from pylibacl package.
POSIX ACLs not supported on filesystem at C:/Users/vdrmrt/Desktop/data
escape_dos_devices not required by filesystem at C:/Users/vdrmrt/Desktop/data
-----------------------------------------------------------------
Detected abilities for source (read only) file system:
  Access control lists                         Off
  Extended attributes                          Off
  Windows access control lists                 On
  Case sensitivity                             Off
  Escape DOS devices                           Off
  Escape trailing spaces                       Off
  Mac OS X style resource forks                Off
  Mac OS X Finder information                  Off
-----------------------------------------------------------------
Unable to import win32security module. Windows ACLs
not supported by filesystem at backup/rdiff-backup-data/rdiff-backup.tmp.0
escape_dos_devices not required by filesystem at backup/rdiff-backup-data/rdiff-backup.tmp.0
-----------------------------------------------------------------
Detected abilities for destination (read/write) file system:
  Ownership changing                           Off
  Hard linking                                 On
  fsync() directories                          On
  Directory inc permissions                    On
  High-bit permissions                         On
  Symlink permissions                          Off
  Extended filenames                           On
  Windows reserved filenames                   Off
  Access control lists                         On
  Extended attributes                          On
  Windows access control lists                 Off
  Case sensitivity                             On
  Escape DOS devices                           Off
  Escape trailing spaces                       Off
  Mac OS X style resource forks                Off
  Mac OS X Finder information                  Off
-----------------------------------------------------------------
Backup: must_escape_dos_devices = 0
Symbolic links excluded by default on Windows
Starting increment operation C:/Users/vdrmrt/Desktop/data to backup

Answer 1

我终于解决了这个问题。
Rdiff-backup 在通过管道传输到其他程序时缓冲 stdout 和 stderr。解决方案是使用额外的 py2exe 选项重建 rdiff-backup:'unbuffered': True
我还必须添加一个额外的选项以使其在使用 Windows 7 构建时能够正常工作。
Py2exe - win32api.pyc ImportError DLL load failed

这是来自 rdiff-backup 构建的最终 setup.py 的片段。

if '--single-file' in sys.argv[1:]:
            sys.argv.remove('--single-file')
            extra_options.update({
                'options': {'py2exe': {'bundle_files': 1,
                                       'unbuffered': True,
                                       'dll_excludes': [ "mswsock.dll", "powrprof.dll" ]}},
                'zipfile': None
            })