python - Pytest 运行所有测试两次并比较模拟结果和真实结果

Question

我为数据库编写了一些插件——这可能会改变从数据库接收到的结果，但大多数情况下这不是预期的。我想知道它什么时候发生。

我有几十个测试，我为任何功能添加了更多，我希望有一个系统，在没有这个插件的情况下，所有的测试都运行一次，然后使用插件并可以选择比较结果。我需要它准备好进行更多测试。

目前，如果数据库在有或没有插件的情况下运行，我可以更改 fixture 。当每次使用不同的 fixture 运行时，是否有任何选项可以使测试运行两次？

Answer 1

除非我误解了您的问题，否则您可以定义一个参数化 fixture ，它将根据当前参数(真实或模拟)选择特定的实现。下面是一个将 sqlalchemy 与 SQLite 数据库和 alchemy-mock 结合使用的工作示例:

import pytest
from unittest import mock

from sqlalchemy import create_engine, Column, String, Integer
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker

from alchemy_mock.mocking import UnifiedAlchemyMagicMock

Base = declarative_base()

class Item(Base):
    __tablename__ = 'items'
    id = Column(Integer, primary_key=True)
    name = Column(String)


@pytest.fixture
def real_db_session():
    engine = create_engine('sqlite:///real.db')

    with engine.connect() as conn:
        Session = sessionmaker(bind=conn)
        Base.metadata.create_all(engine)
        session = Session()

        sample_item = Item(name='fizz')
        session.add(sample_item)
        session.commit()

        yield session


@pytest.fixture
def mocked_db_session():
    session = UnifiedAlchemyMagicMock()
    session.add(Item(name='fizz'))
    return session


@pytest.fixture(params=('real', 'mock'))
def db_session(request, real_db_session, mocked_db_session):
    backend_type = request.param
    if backend_type == 'real':
        return real_db_session
    elif backend_type == 'mock':
        return mocked_db_session

测试示例:

def test_fizz(db_session):
    assert db_session.query(Item).one().name == 'fizz'

执行 yield :

$ pytest -v 
======================================= test session starts ========================================
platform linux -- Python 3.6.8, pytest-4.4.2, py-1.8.0, pluggy-0.11.0
cachedir: .pytest_cache
rootdir: /home/hoefling/projects/private/stackoverflow/so-56558823
plugins: xdist-1.28.0, forked-1.0.2, cov-2.7.1
collected 2 items                                                                                  

test_spam.py::test_fizz[real] PASSED                                                         [ 50%]
test_spam.py::test_fizz[mock] PASSED                                                         [100%]

===================================== 2 passed in 0.18 seconds =====================================

示例:自定义执行顺序

您将需要实现自定义 pytest_collection_modifyitems Hook ，您可以在其中使用收集的测试列表。例如，要先运行 real 测试，然后再运行其余测试:

# conftest.py

def pytest_collection_modifyitems(session, config, items):
    items.sort(key=lambda item: 'real' in item.name, reverse=True)

示例:收集和评估测试结果

这个例子是基于my answer问题How can I access the overall test result of a pytest test run during runtime? .松散地跟随它:

# conftest.py

def pytest_sessionstart(session):
    session.results = dict()


@pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item, call):
    outcome = yield
    result = outcome.get_result()

    if result.when == 'call':
        item.session.results[item] = result


@pytest.fixture(scope='session', autouse=True)
def compare_results(request):
    yield  # wait for all tests to finish

    results = request.session.results

    # partition test results into reals and mocks
    def partition(pred, coll):
        first, second = itertools.tee(coll)
        return itertools.filterfalse(pred, first), filter(pred, second)

    mocks, reals = partition(lambda item: item.name.endswith('[real]'), results.keys())
    # process test results in pairs
    by_name = operator.attrgetter('name')
    for real, mock in zip(sorted(reals, key=by_name), sorted(mocks, key=by_name)):
        if results[real].outcome != results[mock].outcome:
            pytest.fail(
                'A pair of tests has different outcomes:\n'
                f'outcome of {real.name} is {results[real].outcome}\n'
                f'outcome of {mock.name} is {results[mock].outcome}'
            )

当然，这只是一个 stub ；例如，比较将在第一对具有不同结果的测试中失败，如果您有未参数化的测试，结果字典键的分区将为 reals 和 mocks 生成不均匀的列表等