python - Popen/Wait - 等待永远不会结束

Question

import sys
import os
from subprocess import Popen, PIPE, STDOUT

# Transfer Database
print ('Transferring from ' + mysql_source_database)
mysql = Popen(f"mysql -h {mysql_dest_host} -P 3306 -u {mysql_dest_username} -p{mysql_dest_pw} {mysql_dest_database}".split(), stdin=PIPE, stdout=PIPE)
dbnamerewrite = Popen(f"sed s/{mysql_source_database}/{mysql_dest_database}/g".split(), stdin=PIPE, stdout=mysql.stdin)
mysqldump = Popen(f"mysqldump --set-gtid-purged=OFF --column-statistics=0 -h {mysql_source_host} -P 3306 -u {mysql_source_username} -p{mysql_source_pw} {mysql_source_database}".split(), stdout=dbnamerewrite.stdin)
mysql_stdout = mysql.communicate()[0]
mysqldump.wait()

上面的代码完成了我想要的，但从未停止等待。有谁知道如何解决等待。如果我在 SQL 工作完成后按 ctrl-c，这是给定的错误:

^CERROR 1064 (42000) at line 3829: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '' at line 3
Traceback (most recent call last):
  File "test.py", line 19, in <module>
    mysql_stdout = mysql.communicate()[0]
  File "/usr/lib/python3.8/subprocess.py", line 1028, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.8/subprocess.py", line 1868, in _communicate
    ready = selector.select(timeout)
  File "/usr/lib/python3.8/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
KeyboardInterrupt

Answer 1

一件事是您应该放弃对 mysqldump.wait() 的显式调用。根据docs :

Note: This will deadlock when using stdout=PIPE or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use Popen.communicate() when using pipes to avoid that.

mysql.communicate在这种情况下就足够了，因为它不会收到 EOF，直到管道上的所有元素都发送一个 EOF。所以mysql.communicate()直接返回意味着其他两个过程已经完成。

另一个问题是，根据您拥有的进程的顺序，您将不得不以相反的顺序对所有进程调用 communicate 以使数据流过管道。一种解决方案就是这样做:

db_param = ['-h', mysql_dest_host, '-P', '3306', '-u', mysql_dest_username, f'p{mysql_dest_pw}', mysql_dest_database]

mysql = Popen(['mysql'] + db_param,
              stdin=PIPE, stdout=PIPE)
dbnamerewrite = Popen(['sed', f's/{mysql_source_database}/{mysql_dest_database}/g'],
                      stdin=PIPE, stdout=mysql.stdin)
mysqldump = Popen(['mysqldump', '--set-gtid-purged=OFF', '--column-statistics=0'] + db_param,
                  stdout=dbnamerewrite.stdin)

mysqldump.communicate()
dbnamerewrite.communicate()
mysql_stdout = mysql.communicate()[0]

另一种选择是以相反的顺序设置你的管道，在这种情况下你只需要与最后一个进程通信:

db_param = ['-h', mysql_dest_host, '-P', '3306', '-u', mysql_dest_username, f'p{mysql_dest_pw}', mysql_dest_database]

mysqldump = Popen(['mysqldump', '--set-gtid-purged=OFF', '--column-statistics=0'] + db_param,
                  stdout=PIPE)
dbnamerewrite = Popen(['sed', f's/{mysql_source_database}/{mysql_dest_database}/g'],
                      stdin=mysqldump.stdout, stdout=PIPE)
mysql = Popen(['mysql'] + db_param, stdin=dbnamerewrite.stdout, stdout=PIPE)

mysql_stdout = mysql.communicate()[0]