python - 使用子进程输出到 HDFS 中的文件

Question

我有一个脚本，它逐行读取文本，稍微修改该行，然后将该行输出到文件。我可以很好地将文本读入文件，问题是我无法输出文本。这是我的代码。

cat = subprocess.Popen(["hadoop", "fs", "-cat", "/user/test/myfile.txt"], stdout=subprocess.PIPE)
for line in cat.stdout:
    line = line+"Blah";
    subprocess.Popen(["hadoop", "fs", "-put", "/user/test/moddedfile.txt"], stdin=line)

这是我遇到的错误。

AttributeError: 'str' object has no attribute 'fileno'
cat: Unable to write to output stream.

Answer 1

stdin 参数不接受字符串。它应该是 PIPE、None 或现有文件(具有有效 .fileno() 或整数文件描述符的文件)。

from subprocess import Popen, PIPE

cat = Popen(["hadoop", "fs", "-cat", "/user/test/myfile.txt"],
            stdout=PIPE, bufsize=-1)
put = Popen(["hadoop", "fs", "-put", "-", "/user/test/moddedfile.txt"],
            stdin=PIPE, bufsize=-1)
for line in cat.stdout:
    line += "Blah"
    put.stdin.write(line)

cat.stdout.close()
cat.wait()
put.stdin.close()
put.wait()