python - 将 MosesTokenizer 与 Python 结合使用时出现错误 "stdbuf was not found; communication with perl may hang due to stdio buffering."

Question

在 Windows 平台(尝试了多台 Windows 机器)上使用 mosestokenizer(https://github.com/luismsgomes/mosestokenizer，通过 pypi 安装 https://pypi.org/project/mosestokenizer/)时出现问题。

虽然上个月没有新版本发布，但之前它运行良好。我相信这与 python 和 perl 之间的通信有关。

>> MosesSentenceSplitter('en')
stdbuf was not found; communication with perl may hang due to stdio buffering.
Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "...\.venv\lib\site-packages\mosestokenizer\sentsplitter.py",
    line 82, in __init__
        super().__init__(argv)
    File "...\.venv\lib\site-packages\toolwrapper.py",
    line 64, in __init__
        self.start()
    File "...\.venv\lib\site-packages\toolwrapper.py",
    line 102, in start
    self.proc = subprocess.Popen(
    File "C:\Users\MyUser\AppData\Local\Programs\Python\Python38-32\lib\subprocess.py",
    line 854, in __init__
        self._execute_child(args, executable, preexec_fn, close_fds,
    File "C:\Users\MyUser\AppData\Local\Programs\Python\Python38-32\lib\subprocess.py",
    line 1307, in _execute_child
        hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified

为了重现它，只需创建一些 moses* 类对象:

from mosestokenizer import MosesSentenceSplitter
splitsents = MosesSentenceSplitter('en')  # and here error appear.

Answer 1

安装 perl 修复了错误。

另外，正如@ikegami 提到的，This应该有 $| = 1，否则你会收到 stdbuf 警告。