python - 这是使用 Python 3 unittest 测试 stdout 的正确方法吗？

Question

假设我有一个提交文件，fileFromStudent.py，其中唯一的内容是:

print("hello world")

我想测试标准输出，看看学生是否正确写出了打印语句。根据我所读到的内容，我已经能够创建以下代码:

from io import StringIO
from unittest.mock import patch
import unittest, importlib, sys

class TestStringMethods(unittest.TestCase):
    def setUp(self):
        studentSubmission = 'fileFromStudent'

        ## Stores output from print() in fakeOutput
        with patch('sys.stdout', new=StringIO()) as self.fakeOutput:
            ## Loads submission on first test, reloads on subsequent tests
            if studentSubmission in sys.modules:
                importlib.reload(sys.modules[ studentSubmission ] )
            else:
                importlib.import_module( studentSubmission )

    ## Test Cases
    def test_print_passes(self):
        test_case = "Checking Output Statement - Will Pass"
        self.output = self.fakeOutput.getvalue().strip()
        self.assertEqual(self.output, 'hello world', msg=test_case)

    def test_print_fails(self):
        test_case = "Checking Output Statement - Will Fail"
        self.output = self.fakeOutput.getvalue().strip()
        self.assertEqual(self.output, 'hell world', msg=test_case)

if __name__ == '__main__':
    suite = unittest.TestLoader().loadTestsFromTestCase(TestStringMethods)
    testResult = unittest.TextTestRunner(verbosity=2).run(suite)

上面的方法有效，但是我的处理方式正确吗？我添加的内容之一是调用 import.reload() 来重新加载学生的程序。这是因为在最初几周，我会让学生使用 print() 作为最终输出(直到我们进入函数)。

我知道它看起来模棱两可，或者为什么我应该打扰它，因为它有效，但是上面的代码是构建此代码的正确方法还是我完全错过了使这一切变得简单的东西？

Answer 1

我花了几个星期的时间来解决这个问题，取得了一定的成功，但也遇到了一些麻烦，还需要 Google。我没有走 Popen 路线的原因之一是我想捕获学生是否提交了立即崩溃的错误代码。不管你信不信，入门类(class)的前几周就是这样。由于我发现的所有内容都是 2011 年至 2012 年的，因此我想我应该发布此内容，以便 future 的 Google 员工可以找到它。

扩展我上面写的内容，让我们假设下一个任务是获取输入并说“嗨”

name = input("What's your name? ")
print("Hi " + name)

现在，我想自动化测试，看看是否可以输入 "Adam" 并返回 "Hi Adam"。为此，我选择使用 StringIO 作为我的标准输入 (sys.stdin = StringIO("Adam"))。这使我能够控制文本流的来源和去向。此外，我不想看到学生可能发生的所有错误 (sys.stderr = StringIO())。

正如我所提到的，我选择使用 importlib 而不是 Popen。我想确保，如果学生提交了虚假代码，而不是破坏所有内容，而是让我正在运行的任何测试失败。我尝试了 subprocess 和 py.test，虽然它们可能更好、更干净，但我找不到任何对我来说有意义的关于如何让它移动的东西正确地。

下面是我最新版本测试的副本:

from io import StringIO
from unittest.mock import patch
import unittest, importlib, sys, os
from time import sleep

# setup the environment
backup = sys.stderr

class TestTypingExercise(unittest.TestCase):
    def __init__(self, test_name, filename, inputs):
        super(TestTypingExercise, self).__init__(test_name)
        self.library = filename.split('.')[0]
        self.inputs = inputs

    def setUp(self):
        sys.stdin = StringIO(self.inputs[0])
        try:
            ## Stores output from print() in fakeOutput
            with patch('sys.stdout', new=StringIO()) as self.fakeOutput:
                ## Loads submission on first test, reloads on subsequent tests
                if self.library in sys.modules:
                    importlib.reload(sys.modules[ self.library ] )
                else:
                    importlib.import_module( self.library )
        except Exception as e:
            self.fail("Failed to Load - {0}".format(str(e)))

    ## Test Cases
    def test_code_runs(self):
        test_case = "Checking to See if code can run"
        self.assertTrue(True, msg=test_case)

    def test_says_hello(self):
        test_case = "Checking to See if code said 'Hi Adam'"
        # Regex might be cleaner, but this typically solves most cases
        self.output = self.fakeOutput.getvalue().strip().lower()
        self.assertTrue('hi adam' in self.output, msg=test_case)

if __name__ == '__main__':
    ignore_list = ["grader.py"]

    # Run Through Each Submitted File
    directory = os.listdir('.')
    for filename in sorted(directory):
        if (filename.split('.')[-1] != 'py') or (filename in ignore_list):
            continue
        #print("*"*15, filename, "*"*15)

        # 'Disables' stderr, so I don't have to see all their errors
        sys.stderr = StringIO()     # capture output

        # Run Tests Across Student's Submission
        suite = unittest.TestSuite()
        suite.addTest(TestTypingExercise('test_code_runs', filename, 'Adam'))
        suite.addTest(TestTypingExercise('test_says_hello', filename, 'Adam'))
        results = unittest.TextTestRunner().run(suite)

        # Reset stderr
        out = sys.stderr.getvalue() # release output
        sys.stderr.close()  # close the stream 
        sys.stderr = backup # restore original stderr

        # Display Test Results
        print(filename,"Test Results - ", end='')
        if not results.wasSuccessful():
            print("Failed (test cases that failed):")
            for error in results.failures:
                print('\t',error[1].split('\n')[-2])
        else:
            print("Pass!")
        sleep(0.05)

最终结果如下:

StudentSubmission01.py Test Results - Failed (test cases that failed):
     AssertionError: Failed to Load - EOL while scanning string literal (StudentSubmission01.py, line 23)
     AssertionError: Failed to Load - EOL while scanning string literal (StudentSubmission01.py, line 23)
StudentSubmission02.py Test Results - Pass!
StudentSubmission03.py Test Results - Pass!
StudentSubmission04.py Test Results - Pass!
StudentSubmission05.py Test Results - Pass!
StudentSubmission06.py Test Results - Pass!
StudentSubmission07.py Test Results - Pass!
StudentSubmission08.py Test Results - Pass!
StudentSubmission09.py Test Results - Pass!
StudentSubmission10.py Test Results - Pass!
StudentSubmission11.py Test Results - Pass!
StudentSubmission12.py Test Results - Pass!
StudentSubmission13.py Test Results - Pass!
[Finished in 0.9s]

如果我想测试多个不同的输入，我可能需要移动一些东西，但目前这是可行的。