SLURM，使用 srun 打印输出

Question

我正在使用 srun 来运行我的程序，但是它无法打印输出。

me@home:~$ srun -p K80q --gres=gpu:1 -N 1 python3 main.py 
2019-05-15 19:56:43.305156: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-05-15 19:56:43.543516: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1392] Found device 0 with properties: 
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:85:00.0
totalMemory: 11.17GiB freeMemory: 11.10GiB
2019-05-15 19:56:43.543567: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1471] Adding visible gpu devices: 0
2019-05-15 19:56:43.900189: I tensorflow/core/common_runtime/gpu/gpu_device.cc:952] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-05-15 19:56:43.900248: I tensorflow/core/common_runtime/gpu/gpu_device.cc:958]      0 
2019-05-15 19:56:43.900257: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0:   N 
2019-05-15 19:56:43.900619: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10761 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:85:00.0, compute capability: 3.7)

我只得到了上面的输出，并不能打印出我期望的信息。我该如何解决？

顺便定义一个测试代码

import tensorflow 

if __name__ == '__main__':
    for i in range(10):
        print('Hello')

它可以打印 Hello 10 次。

更新:

20 分钟后，它输出了一些我预期的信息。如何让它立即输出？

Answer 1

试试 srun 的 -u 选项:

-u, --unbuffered By default the connection between slurmstepd and the user launched application is over a pipe. The stdio output written by the application is buffered by the glibc until it is flushed or the output is set as unbuffered. See setbuf(3). If this option is specified the tasks are executed with a pseudo terminal so that the application output is unbuffered.