Full pytest documentation

Download latest version as PDF

Installation and Getting Started

Pythons: Python 3.5, 3.6, 3.7, PyPy3

Platforms: Linux and Windows

PyPI package name: pytest

Documentation as PDF: download latest

pytest is a framework that makes building simple and scalable tests easy. Tests are expressive and readable—no boilerplate code required. Get started in minutes with a small unit test or complex functional test for your application or library.

Install pytest

1. Run the following command in your command line:

pip install -U pytest

2. Check that you installed the correct version:

$ pytest --version
This is pytest version 5.x.y, imported from $PYTHON_PREFIX/lib/python3.8/site-packages/pytest/__init__.py

Create your first test

Create a simple test function with just four lines of code:

# content of test_sample.py
def func(x):
    return x + 1


def test_answer():
    assert func(3) == 5

That’s it. You can now execute the test function:

$ pytest
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 1 item

test_sample.py F                                                     [100%]

================================= FAILURES =================================
_______________________________ test_answer ________________________________

    def test_answer():
>       assert func(3) == 5
E       assert 4 == 5
E        +  where 4 = func(3)

test_sample.py:6: AssertionError
============================ 1 failed in 0.12s =============================

This test returns a failure report because func(3) does not return 5.

Note

You can use the assert statement to verify test expectations. pytest’s Advanced assertion introspection will intelligently report intermediate values of the assert expression so you can avoid the many names of JUnit legacy methods.

Run multiple tests

pytest will run all files of the form test_*.py or *_test.py in the current directory and its subdirectories. More generally, it follows standard test discovery rules.

Assert that a certain exception is raised

Use the raises helper to assert that some code raises an exception:

# content of test_sysexit.py
import pytest


def f():
    raise SystemExit(1)


def test_mytest():
    with pytest.raises(SystemExit):
        f()

Execute the test function with “quiet” reporting mode:

$ pytest -q test_sysexit.py
.                                                                    [100%]
1 passed in 0.12s

Group multiple tests in a class

Once you develop multiple tests, you may want to group them into a class. pytest makes it easy to create a class containing more than one test:

# content of test_class.py
class TestClass:
    def test_one(self):
        x = "this"
        assert "h" in x

    def test_two(self):
        x = "hello"
        assert hasattr(x, "check")

pytest discovers all tests following its Conventions for Python test discovery, so it finds both test_ prefixed functions. There is no need to subclass anything. We can simply run the module by passing its filename:

$ pytest -q test_class.py
.F                                                                   [100%]
================================= FAILURES =================================
____________________________ TestClass.test_two ____________________________

self = <test_class.TestClass object at 0xdeadbeef>

    def test_two(self):
        x = "hello"
>       assert hasattr(x, "check")
E       AssertionError: assert False
E        +  where False = hasattr('hello', 'check')

test_class.py:8: AssertionError
1 failed, 1 passed in 0.12s

The first test passed and the second failed. You can easily see the intermediate values in the assertion to help you understand the reason for the failure.

Request a unique temporary directory for functional tests

pytest provides Builtin fixtures/function arguments to request arbitrary resources, like a unique temporary directory:

# content of test_tmpdir.py
def test_needsfiles(tmpdir):
    print(tmpdir)
    assert 0

List the name tmpdir in the test function signature and pytest will lookup and call a fixture factory to create the resource before performing the test function call. Before the test runs, pytest creates a unique-per-test-invocation temporary directory:

$ pytest -q test_tmpdir.py
F                                                                    [100%]
================================= FAILURES =================================
_____________________________ test_needsfiles ______________________________

tmpdir = local('PYTEST_TMPDIR/test_needsfiles0')

    def test_needsfiles(tmpdir):
        print(tmpdir)
>       assert 0
E       assert 0

test_tmpdir.py:3: AssertionError
--------------------------- Captured stdout call ---------------------------
PYTEST_TMPDIR/test_needsfiles0
1 failed in 0.12s

More info on tmpdir handling is available at Temporary directories and files.

Find out what kind of builtin pytest fixtures exist with the command:

pytest --fixtures   # shows builtin and custom fixtures

Note that this command omits fixtures with leading _ unless the -v option is added.

Continue reading

Check out additional pytest resources to help you customize tests for your unique workflow:

• “Calling pytest through python -m pytest” for command line invocation examples
• “Using pytest with an existing test suite” for working with pre-existing tests
• “Marking test functions with attributes” for information on the pytest.mark mechanism
• “pytest fixtures: explicit, modular, scalable” for providing a functional baseline to your tests
• “Writing plugins” for managing and writing plugins
• “Good Integration Practices” for virtualenv and test layouts

Usage and Invocations

Calling pytest through python -m pytest

You can invoke testing through the Python interpreter from the command line:

python -m pytest [...]

This is almost equivalent to invoking the command line script pytest [...] directly, except that calling via python will also add the current directory to sys.path.

Possible exit codes

Running pytest can result in six different exit codes:

Exit code 0:All tests were collected and passed successfully Exit code 1:Tests were collected and run but some of the tests failed Exit code 2:Test execution was interrupted by the user Exit code 3:Internal error happened while executing tests Exit code 4:pytest command line usage error Exit code 5:No tests were collected

They are represented by the _pytest.main.ExitCode enum. The exit codes being a part of the public API can be imported and accessed directly using:

from pytest import ExitCode

Note

If you would like to customize the exit code in some scenarios, specially when no tests are collected, consider using the pytest-custom_exit_code plugin.

Getting help on version, option names, environment variables

pytest --version   # shows where pytest was imported from
pytest --fixtures  # show available builtin function arguments
pytest -h | --help # show help on command line and config file options

Stopping after the first (or N) failures

To stop the testing process after the first (N) failures:

pytest -x           # stop after first failure
pytest --maxfail=2  # stop after two failures

Specifying tests / selecting tests

Pytest supports several ways to run and select tests from the command-line.

Run tests in a module

pytest test_mod.py

Run tests in a directory

pytest testing/

Run tests by keyword expressions

pytest -k "MyClass and not method"

This will run tests which contain names that match the given string expression, which can include Python operators that use filenames, class names and function names as variables. The example above will run TestMyClass.test_something but not TestMyClass.test_method_simple.

Run tests by node ids

Each collected test is assigned a unique nodeid which consist of the module filename followed by specifiers like class names, function names and parameters from parametrization, separated by :: characters.

To run a specific test within a module:

pytest test_mod.py::test_func

Another example specifying a test method in the command line:

pytest test_mod.py::TestClass::test_method

Run tests by marker expressions

pytest -m slow

Will run all tests which are decorated with the @pytest.mark.slow decorator.

For more information see marks.

Run tests from packages

pytest --pyargs pkg.testing

This will import pkg.testing and use its filesystem location to find and run tests from.

Modifying Python traceback printing

Examples for modifying traceback printing:

pytest --showlocals # show local variables in tracebacks
pytest -l           # show local variables (shortcut)

pytest --tb=auto    # (default) 'long' tracebacks for the first and last
                     # entry, but 'short' style for the other entries
pytest --tb=long    # exhaustive, informative traceback formatting
pytest --tb=short   # shorter traceback format
pytest --tb=line    # only one line per failure
pytest --tb=native  # Python standard library formatting
pytest --tb=no      # no traceback at all

The --full-trace causes very long traces to be printed on error (longer than --tb=long). It also ensures that a stack trace is printed on KeyboardInterrupt (Ctrl+C). This is very useful if the tests are taking too long and you interrupt them with Ctrl+C to find out where the tests are hanging. By default no output will be shown (because KeyboardInterrupt is caught by pytest). By using this option you make sure a trace is shown.

Detailed summary report

The -r flag can be used to display a “short test summary info” at the end of the test session, making it easy in large test suites to get a clear picture of all failures, skips, xfails, etc.

Example:

# content of test_example.py
import pytest


@pytest.fixture
def error_fixture():
    assert 0


def test_ok():
    print("ok")


def test_fail():
    assert 0


def test_error(error_fixture):
    pass


def test_skip():
    pytest.skip("skipping this test")


def test_xfail():
    pytest.xfail("xfailing this test")


@pytest.mark.xfail(reason="always xfail")
def test_xpass():
    pass

$ pytest -ra
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 6 items

test_example.py .FEsxX                                               [100%]

================================== ERRORS ==================================
_______________________ ERROR at setup of test_error _______________________

    @pytest.fixture
    def error_fixture():
>       assert 0
E       assert 0

test_example.py:6: AssertionError
================================= FAILURES =================================
________________________________ test_fail _________________________________

    def test_fail():
>       assert 0
E       assert 0

test_example.py:14: AssertionError
========================= short test summary info ==========================
SKIPPED [1] $REGENDOC_TMPDIR/test_example.py:22: skipping this test
XFAIL test_example.py::test_xfail
  reason: xfailing this test
XPASS test_example.py::test_xpass always xfail
ERROR test_example.py::test_error - assert 0
FAILED test_example.py::test_fail - assert 0
== 1 failed, 1 passed, 1 skipped, 1 xfailed, 1 xpassed, 1 error in 0.12s ===

The -r options accepts a number of characters after it, with a used above meaning “all except passes”.

Here is the full list of available characters that can be used:

• f - failed
• E - error
• s - skipped
• x - xfailed
• X - xpassed
• p - passed
• P - passed with output
• a - all except pP
• A - all

More than one character can be used, so for example to only see failed and skipped tests, you can execute:

$ pytest -rfs
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 6 items

test_example.py .FEsxX                                               [100%]

================================== ERRORS ==================================
_______________________ ERROR at setup of test_error _______________________

    @pytest.fixture
    def error_fixture():
>       assert 0
E       assert 0

test_example.py:6: AssertionError
================================= FAILURES =================================
________________________________ test_fail _________________________________

    def test_fail():
>       assert 0
E       assert 0

test_example.py:14: AssertionError
========================= short test summary info ==========================
FAILED test_example.py::test_fail - assert 0
SKIPPED [1] $REGENDOC_TMPDIR/test_example.py:22: skipping this test
== 1 failed, 1 passed, 1 skipped, 1 xfailed, 1 xpassed, 1 error in 0.12s ===

Using p lists the passing tests, whilst P adds an extra section “PASSES” with those tests that passed but had captured output:

$ pytest -rpP
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 6 items

test_example.py .FEsxX                                               [100%]

================================== ERRORS ==================================
_______________________ ERROR at setup of test_error _______________________

    @pytest.fixture
    def error_fixture():
>       assert 0
E       assert 0

test_example.py:6: AssertionError
================================= FAILURES =================================
________________________________ test_fail _________________________________

    def test_fail():
>       assert 0
E       assert 0

test_example.py:14: AssertionError
================================== PASSES ==================================
_________________________________ test_ok __________________________________
--------------------------- Captured stdout call ---------------------------
ok
========================= short test summary info ==========================
PASSED test_example.py::test_ok
== 1 failed, 1 passed, 1 skipped, 1 xfailed, 1 xpassed, 1 error in 0.12s ===

Dropping to PDB (Python Debugger) on failures

Python comes with a builtin Python debugger called PDB. pytest allows one to drop into the PDB prompt via a command line option:

pytest --pdb

This will invoke the Python debugger on every failure (or KeyboardInterrupt). Often you might only want to do this for the first failing test to understand a certain failure situation:

pytest -x --pdb   # drop to PDB on first failure, then end test session
pytest --pdb --maxfail=3  # drop to PDB for first three failures

Note that on any failure the exception information is stored on sys.last_value, sys.last_type and sys.last_traceback. In interactive use, this allows one to drop into postmortem debugging with any debug tool. One can also manually access the exception information, for example:

>>> import sys
>>> sys.last_traceback.tb_lineno
42
>>> sys.last_value
AssertionError('assert result == "ok"',)

Dropping to PDB (Python Debugger) at the start of a test

pytest allows one to drop into the PDB prompt immediately at the start of each test via a command line option:

pytest --trace

This will invoke the Python debugger at the start of every test.

Setting breakpoints

To set a breakpoint in your code use the native Python import pdb;pdb.set_trace() call in your code and pytest automatically disables its output capture for that test:

• Output capture in other tests is not affected.
• Any prior test output that has already been captured and will be processed as such.
• Output capture gets resumed when ending the debugger session (via the continue command).

Using the builtin breakpoint function

Python 3.7 introduces a builtin breakpoint() function. Pytest supports the use of breakpoint() with the following behaviours:

• When breakpoint() is called and PYTHONBREAKPOINT is set to the default value, pytest will use the custom internal PDB trace UI instead of the system default Pdb.
• When tests are complete, the system will default back to the system Pdb trace UI.
• With --pdb passed to pytest, the custom internal Pdb trace UI is used with both breakpoint() and failed tests/unhandled exceptions.
• --pdbcls can be used to specify a custom debugger class.

Profiling test execution duration

To get a list of the slowest 10 test durations:

pytest --durations=10

By default, pytest will not show test durations that are too small (<0.01s) unless -vv is passed on the command-line.

Fault Handler

New in version 5.0.

The faulthandler standard module can be used to dump Python tracebacks on a segfault or after a timeout.

The module is automatically enabled for pytest runs, unless the -p no:faulthandler is given on the command-line.

Also the faulthandler_timeout=X configuration option can be used to dump the traceback of all threads if a test takes longer than X seconds to finish (not available on Windows).

Note

This functionality has been integrated from the external pytest-faulthandler plugin, with two small differences:

• To disable it, use -p no:faulthandler instead of --no-faulthandler: the former can be used with any plugin, so it saves one option.
• The --faulthandler-timeout command-line option has become the faulthandler_timeout configuration option. It can still be configured from the command-line using -o faulthandler_timeout=X.

Creating JUnitXML format files

To create result files which can be read by Jenkins or other Continuous integration servers, use this invocation:

pytest --junitxml=path

to create an XML file at path.

To set the name of the root test suite xml item, you can configure the junit_suite_name option in your config file:

[pytest]
junit_suite_name = my_suite

New in version 4.0.

JUnit XML specification seems to indicate that "time" attribute should report total test execution times, including setup and teardown (1, 2). It is the default pytest behavior. To report just call durations instead, configure the junit_duration_report option like this:

[pytest]
junit_duration_report = call

record_property

If you want to log additional information for a test, you can use the record_property fixture:

def test_function(record_property):
    record_property("example_key", 1)
    assert True

This will add an extra property example_key="1" to the generated testcase tag:

<testcase classname="test_function" file="test_function.py" line="0" name="test_function" time="0.0009">
  <properties>
    <property name="example_key" value="1" />
  </properties>
</testcase>

Alternatively, you can integrate this functionality with custom markers:

# content of conftest.py


def pytest_collection_modifyitems(session, config, items):
    for item in items:
        for marker in item.iter_markers(name="test_id"):
            test_id = marker.args[0]
            item.user_properties.append(("test_id", test_id))

And in your tests:

# content of test_function.py
import pytest


@pytest.mark.test_id(1501)
def test_function():
    assert True

Will result in:

<testcase classname="test_function" file="test_function.py" line="0" name="test_function" time="0.0009">
  <properties>
    <property name="test_id" value="1501" />
  </properties>
</testcase>

Warning

Please note that using this feature will break schema verifications for the latest JUnitXML schema. This might be a problem when used with some CI servers.

record_xml_attribute

To add an additional xml attribute to a testcase element, you can use record_xml_attribute fixture. This can also be used to override existing values:

def test_function(record_xml_attribute):
    record_xml_attribute("assertions", "REQ-1234")
    record_xml_attribute("classname", "custom_classname")
    print("hello world")
    assert True

Unlike record_property, this will not add a new child element. Instead, this will add an attribute assertions="REQ-1234" inside the generated testcase tag and override the default classname with "classname=custom_classname":

<testcase classname="custom_classname" file="test_function.py" line="0" name="test_function" time="0.003" assertions="REQ-1234">
    <system-out>
        hello world
    </system-out>
</testcase>

Warning

record_xml_attribute is an experimental feature, and its interface might be replaced by something more powerful and general in future versions. The functionality per-se will be kept, however.

Using this over record_xml_property can help when using ci tools to parse the xml report. However, some parsers are quite strict about the elements and attributes that are allowed. Many tools use an xsd schema (like the example below) to validate incoming xml. Make sure you are using attribute names that are allowed by your parser.

Below is the Scheme used by Jenkins to validate the XML report:

<xs:element name="testcase">
    <xs:complexType>
        <xs:sequence>
            <xs:element ref="skipped" minOccurs="0" maxOccurs="1"/>
            <xs:element ref="error" minOccurs="0" maxOccurs="unbounded"/>
            <xs:element ref="failure" minOccurs="0" maxOccurs="unbounded"/>
            <xs:element ref="system-out" minOccurs="0" maxOccurs="unbounded"/>
            <xs:element ref="system-err" minOccurs="0" maxOccurs="unbounded"/>
        </xs:sequence>
        <xs:attribute name="name" type="xs:string" use="required"/>
        <xs:attribute name="assertions" type="xs:string" use="optional"/>
        <xs:attribute name="time" type="xs:string" use="optional"/>
        <xs:attribute name="classname" type="xs:string" use="optional"/>
        <xs:attribute name="status" type="xs:string" use="optional"/>
    </xs:complexType>
</xs:element>

Warning

Please note that using this feature will break schema verifications for the latest JUnitXML schema. This might be a problem when used with some CI servers.

record_testsuite_property

New in version 4.5.

If you want to add a properties node at the test-suite level, which may contains properties that are relevant to all tests, you can use the record_testsuite_property session-scoped fixture:

The record_testsuite_property session-scoped fixture can be used to add properties relevant to all tests.

import pytest


@pytest.fixture(scope="session", autouse=True)
def log_global_env_facts(record_testsuite_property):
    record_testsuite_property("ARCH", "PPC")
    record_testsuite_property("STORAGE_TYPE", "CEPH")


class TestMe:
    def test_foo(self):
        assert True

The fixture is a callable which receives name and value of a <property> tag added at the test-suite level of the generated xml:

<testsuite errors="0" failures="0" name="pytest" skipped="0" tests="1" time="0.006">
  <properties>
    <property name="ARCH" value="PPC"/>
    <property name="STORAGE_TYPE" value="CEPH"/>
  </properties>
  <testcase classname="test_me.TestMe" file="test_me.py" line="16" name="test_foo" time="0.000243663787842"/>
</testsuite>

name must be a string, value will be converted to a string and properly xml-escaped.

The generated XML is compatible with the latest xunit standard, contrary to record_property and record_xml_attribute.

Creating resultlog format files

To create plain-text machine-readable result files you can issue:

pytest --resultlog=path

and look at the content at the path location. Such files are used e.g. by the PyPy-test web page to show test results over several revisions.

Warning

This option is rarely used and is scheduled for removal in pytest 6.0.

If you use this option, consider using the new pytest-reportlog plugin instead.

See the deprecation docs for more information.

Sending test report to online pastebin service

Creating a URL for each test failure:

pytest --pastebin=failed

This will submit test run information to a remote Paste service and provide a URL for each failure. You may select tests as usual or add for example -x if you only want to send one particular failure.

Creating a URL for a whole test session log:

pytest --pastebin=all

Currently only pasting to the http://bpaste.net service is implemented.

Changed in version 5.2.

If creating the URL fails for any reason, a warning is generated instead of failing the entire test suite.

Early loading plugins

You can early-load plugins (internal and external) explicitly in the command-line with the -p option:

pytest -p mypluginmodule

The option receives a name parameter, which can be:

• A full module dotted name, for example myproject.plugins. This dotted name must be importable.

• The entry-point name of a plugin. This is the name passed to setuptools when the plugin is registered. For example to early-load the pytest-cov plugin you can use:

  pytest -p pytest_cov
  

Disabling plugins

To disable loading specific plugins at invocation time, use the -p option together with the prefix no:.

Example: to disable loading the plugin doctest, which is responsible for executing doctest tests from text files, invoke pytest like this:

pytest -p no:doctest

Calling pytest from Python code

You can invoke pytest from Python code directly:

pytest.main()

this acts as if you would call “pytest” from the command line. It will not raise SystemExit but return the exitcode instead. You can pass in options and arguments:

pytest.main(["-x", "mytestdir"])

You can specify additional plugins to pytest.main:

# content of myinvoke.py
import pytest


class MyPlugin:
    def pytest_sessionfinish(self):
        print("*** test run reporting finishing")


pytest.main(["-qq"], plugins=[MyPlugin()])

Running it will show that MyPlugin was added and its hook was invoked:

$ python myinvoke.py
.FEsxX.                                                              [100%]*** test run reporting finishing

================================== ERRORS ==================================
_______________________ ERROR at setup of test_error _______________________

    @pytest.fixture
    def error_fixture():
>       assert 0
E       assert 0

test_example.py:6: AssertionError
================================= FAILURES =================================
________________________________ test_fail _________________________________

    def test_fail():
>       assert 0
E       assert 0

test_example.py:14: AssertionError

Note

Calling pytest.main() will result in importing your tests and any modules that they import. Due to the caching mechanism of python’s import system, making subsequent calls to pytest.main() from the same process will not reflect changes to those files between the calls. For this reason, making multiple calls to pytest.main() from the same process (in order to re-run tests, for example) is not recommended.

Using pytest with an existing test suite

Pytest can be used with most existing test suites, but its behavior differs from other test runners such as nose or Python’s default unittest framework.

Before using this section you will want to install pytest.

Running an existing test suite with pytest

Say you want to contribute to an existing repository somewhere. After pulling the code into your development space using some flavor of version control and (optionally) setting up a virtualenv you will want to run:

cd <repository>
pip install -e .  # Environment dependent alternatives include
                  # 'python setup.py develop' and 'conda develop'

in your project root. This will set up a symlink to your code in site-packages, allowing you to edit your code while your tests run against it as if it were installed.

Setting up your project in development mode lets you avoid having to reinstall every time you want to run your tests, and is less brittle than mucking about with sys.path to point your tests at local code.

Also consider using tox.

The writing and reporting of assertions in tests

Asserting with the assert statement

pytest allows you to use the standard python assert for verifying expectations and values in Python tests. For example, you can write the following:

# content of test_assert1.py
def f():
    return 3


def test_function():
    assert f() == 4

to assert that your function returns a certain value. If this assertion fails you will see the return value of the function call:

$ pytest test_assert1.py
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 1 item

test_assert1.py F                                                    [100%]

================================= FAILURES =================================
______________________________ test_function _______________________________

    def test_function():
>       assert f() == 4
E       assert 3 == 4
E        +  where 3 = f()

test_assert1.py:6: AssertionError
============================ 1 failed in 0.12s =============================

pytest has support for showing the values of the most common subexpressions including calls, attributes, comparisons, and binary and unary operators. (See Demo of Python failure reports with pytest). This allows you to use the idiomatic python constructs without boilerplate code while not losing introspection information.

However, if you specify a message with the assertion like this:

assert a % 2 == 0, "value was odd, should be even"

then no assertion introspection takes places at all and the message will be simply shown in the traceback.

See Assertion introspection details for more information on assertion introspection.

Assertions about expected exceptions

In order to write assertions about raised exceptions, you can use pytest.raises as a context manager like this:

import pytest


def test_zero_division():
    with pytest.raises(ZeroDivisionError):
        1 / 0

and if you need to have access to the actual exception info you may use:

def test_recursion_depth():
    with pytest.raises(RuntimeError) as excinfo:

        def f():
            f()

        f()
    assert "maximum recursion" in str(excinfo.value)

excinfo is a ExceptionInfo instance, which is a wrapper around the actual exception raised. The main attributes of interest are .type, .value and .traceback.

You can pass a match keyword parameter to the context-manager to test that a regular expression matches on the string representation of an exception (similar to the TestCase.assertRaisesRegexp method from unittest):

import pytest


def myfunc():
    raise ValueError("Exception 123 raised")


def test_match():
    with pytest.raises(ValueError, match=r".* 123 .*"):
        myfunc()

The regexp parameter of the match method is matched with the re.search function, so in the above example match='123' would have worked as well.

There’s an alternate form of the pytest.raises function where you pass a function that will be executed with the given *args and **kwargs and assert that the given exception is raised:

pytest.raises(ExpectedException, func, *args, **kwargs)

The reporter will provide you with helpful output in case of failures such as no exception or wrong exception.

Note that it is also possible to specify a “raises” argument to pytest.mark.xfail, which checks that the test is failing in a more specific way than just having any exception raised:

@pytest.mark.xfail(raises=IndexError)
def test_f():
    f()

Using pytest.raises is likely to be better for cases where you are testing exceptions your own code is deliberately raising, whereas using @pytest.mark.xfail with a check function is probably better for something like documenting unfixed bugs (where the test describes what “should” happen) or bugs in dependencies.

Assertions about expected warnings

You can check that code raises a particular warning using pytest.warns.

Making use of context-sensitive comparisons

pytest has rich support for providing context-sensitive information when it encounters comparisons. For example:

# content of test_assert2.py


def test_set_comparison():
    set1 = set("1308")
    set2 = set("8035")
    assert set1 == set2

if you run this module:

$ pytest test_assert2.py
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 1 item

test_assert2.py F                                                    [100%]

================================= FAILURES =================================
___________________________ test_set_comparison ____________________________

    def test_set_comparison():
        set1 = set("1308")
        set2 = set("8035")
>       assert set1 == set2
E       AssertionError: assert {'0', '1', '3', '8'} == {'0', '3', '5', '8'}
E         Extra items in the left set:
E         '1'
E         Extra items in the right set:
E         '5'
E         Use -v to get the full diff

test_assert2.py:6: AssertionError
============================ 1 failed in 0.12s =============================

Special comparisons are done for a number of cases:

• comparing long strings: a context diff is shown
• comparing long sequences: first failing indices
• comparing dicts: different entries

See the reporting demo for many more examples.

Defining your own explanation for failed assertions

It is possible to add your own detailed explanations by implementing the pytest_assertrepr_compare hook.

pytest_assertrepr_compare(config, op, left, right)

return explanation for comparisons in failing assert expressions.

Return None for no custom explanation, otherwise return a list of strings. The strings will be joined by newlines but any newlines in a string will be escaped. Note that all but the first line will be indented slightly, the intention is for the first line to be a summary.

Parameters:config (_pytest.config.Config) – pytest config object

As an example consider adding the following hook in a conftest.py file which provides an alternative explanation for Foo objects:

# content of conftest.py
from test_foocompare import Foo


def pytest_assertrepr_compare(op, left, right):
    if isinstance(left, Foo) and isinstance(right, Foo) and op == "==":
        return [
            "Comparing Foo instances:",
            "   vals: {} != {}".format(left.val, right.val),
        ]

now, given this test module:

# content of test_foocompare.py
class Foo:
    def __init__(self, val):
        self.val = val

    def __eq__(self, other):
        return self.val == other.val


def test_compare():
    f1 = Foo(1)
    f2 = Foo(2)
    assert f1 == f2

you can run the test module and get the custom output defined in the conftest file:

$ pytest -q test_foocompare.py
F                                                                    [100%]
================================= FAILURES =================================
_______________________________ test_compare _______________________________

    def test_compare():
        f1 = Foo(1)
        f2 = Foo(2)
>       assert f1 == f2
E       assert Comparing Foo instances:
E            vals: 1 != 2

test_foocompare.py:12: AssertionError
1 failed in 0.12s

Assertion introspection details

Reporting details about a failing assertion is achieved by rewriting assert statements before they are run. Rewritten assert statements put introspection information into the assertion failure message. pytest only rewrites test modules directly discovered by its test collection process, so asserts in supporting modules which are not themselves test modules will not be rewritten.

You can manually enable assertion rewriting for an imported module by calling register_assert_rewrite before you import it (a good place to do that is in your root conftest.py).

For further information, Benjamin Peterson wrote up Behind the scenes of pytest’s new assertion rewriting.

Assertion rewriting caches files on disk

pytest will write back the rewritten modules to disk for caching. You can disable this behavior (for example to avoid leaving stale .pyc files around in projects that move files around a lot) by adding this to the top of your conftest.py file:

import sys

sys.dont_write_bytecode = True

Note that you still get the benefits of assertion introspection, the only change is that the .pyc files won’t be cached on disk.

Additionally, rewriting will silently skip caching if it cannot write new .pyc files, i.e. in a read-only filesystem or a zipfile.

Disabling assert rewriting

pytest rewrites test modules on import by using an import hook to write new pyc files. Most of the time this works transparently. However, if you are working with the import machinery yourself, the import hook may interfere.

If this is the case you have two options:

• Disable rewriting for a specific module by adding the string PYTEST_DONT_REWRITE to its docstring.

• Disable rewriting for all modules by using --assert=plain.

  Add assert rewriting as an alternate introspection technique.

  Introduce the --assert option. Deprecate --no-assert and --nomagic.

  Removes the --no-assert and --nomagic options. Removes the --assert=reinterp option.

pytest fixtures: explicit, modular, scalable

The purpose of test fixtures is to provide a fixed baseline upon which tests can reliably and repeatedly execute. pytest fixtures offer dramatic improvements over the classic xUnit style of setup/teardown functions:

• fixtures have explicit names and are activated by declaring their use from test functions, modules, classes or whole projects.
• fixtures are implemented in a modular manner, as each fixture name triggers a fixture function which can itself use other fixtures.
• fixture management scales from simple unit to complex functional testing, allowing to parametrize fixtures and tests according to configuration and component options, or to re-use fixtures across function, class, module or whole test session scopes.

In addition, pytest continues to support classic xunit-style setup. You can mix both styles, moving incrementally from classic to new style, as you prefer. You can also start out from existing unittest.TestCase style or nose based projects.

Fixtures as Function arguments

Test functions can receive fixture objects by naming them as an input argument. For each argument name, a fixture function with that name provides the fixture object. Fixture functions are registered by marking them with @pytest.fixture. Let’s look at a simple self-contained test module containing a fixture and a test function using it:

# content of ./test_smtpsimple.py
import pytest


@pytest.fixture
def smtp_connection():
    import smtplib

    return smtplib.SMTP("smtp.gmail.com", 587, timeout=5)


def test_ehlo(smtp_connection):
    response, msg = smtp_connection.ehlo()
    assert response == 250
    assert 0  # for demo purposes

Here, the test_ehlo needs the smtp_connection fixture value. pytest will discover and call the @pytest.fixture marked smtp_connection fixture function. Running the test looks like this:

$ pytest test_smtpsimple.py
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 1 item

test_smtpsimple.py F                                                 [100%]

================================= FAILURES =================================
________________________________ test_ehlo _________________________________

smtp_connection = <smtplib.SMTP object at 0xdeadbeef>

    def test_ehlo(smtp_connection):
        response, msg = smtp_connection.ehlo()
        assert response == 250
>       assert 0  # for demo purposes
E       assert 0

test_smtpsimple.py:14: AssertionError
============================ 1 failed in 0.12s =============================

In the failure traceback we see that the test function was called with a smtp_connection argument, the smtplib.SMTP() instance created by the fixture function. The test function fails on our deliberate assert 0. Here is the exact protocol used by pytest to call the test function this way:

1. pytest finds the test_ehlo because of the test_ prefix. The test function needs a function argument named smtp_connection. A matching fixture function is discovered by looking for a fixture-marked function named smtp_connection.
2. smtp_connection() is called to create an instance.
3. test_ehlo(<smtp_connection instance>) is called and fails in the last line of the test function.

Note that if you misspell a function argument or want to use one that isn’t available, you’ll see an error with a list of available function arguments.

Note

You can always issue:

pytest --fixtures test_simplefactory.py

to see available fixtures (fixtures with leading _ are only shown if you add the -v option).

Fixtures: a prime example of dependency injection

Fixtures allow test functions to easily receive and work against specific pre-initialized application objects without having to care about import/setup/cleanup details. It’s a prime example of dependency injection where fixture functions take the role of the injector and test functions are the consumers of fixture objects.

conftest.py: sharing fixture functions

If during implementing your tests you realize that you want to use a fixture function from multiple test files you can move it to a conftest.py file. You don’t need to import the fixture you want to use in a test, it automatically gets discovered by pytest. The discovery of fixture functions starts at test classes, then test modules, then conftest.py files and finally builtin and third party plugins.

You can also use the conftest.py file to implement local per-directory plugins.

Sharing test data

If you want to make test data from files available to your tests, a good way to do this is by loading these data in a fixture for use by your tests. This makes use of the automatic caching mechanisms of pytest.

Another good approach is by adding the data files in the tests folder. There are also community plugins available to help managing this aspect of testing, e.g. pytest-datadir and pytest-datafiles.

Scope: sharing a fixture instance across tests in a class, module or session

Fixtures requiring network access depend on connectivity and are usually time-expensive to create. Extending the previous example, we can add a scope="module" parameter to the @pytest.fixture invocation to cause the decorated smtp_connection fixture function to only be invoked once per test module (the default is to invoke once per test function). Multiple test functions in a test module will thus each receive the same smtp_connection fixture instance, thus saving time. Possible values for scope are: function, class, module, package or session.

The next example puts the fixture function into a separate conftest.py file so that tests from multiple test modules in the directory can access the fixture function:

# content of conftest.py
import pytest
import smtplib


@pytest.fixture(scope="module")
def smtp_connection():
    return smtplib.SMTP("smtp.gmail.com", 587, timeout=5)

The name of the fixture again is smtp_connection and you can access its result by listing the name smtp_connection as an input parameter in any test or fixture function (in or below the directory where conftest.py is located):

# content of test_module.py


def test_ehlo(smtp_connection):
    response, msg = smtp_connection.ehlo()
    assert response == 250
    assert b"smtp.gmail.com" in msg
    assert 0  # for demo purposes


def test_noop(smtp_connection):
    response, msg = smtp_connection.noop()
    assert response == 250
    assert 0  # for demo purposes

We deliberately insert failing assert 0 statements in order to inspect what is going on and can now run the tests:

$ pytest test_module.py
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 2 items

test_module.py FF                                                    [100%]

================================= FAILURES =================================
________________________________ test_ehlo _________________________________

smtp_connection = <smtplib.SMTP object at 0xdeadbeef>

    def test_ehlo(smtp_connection):
        response, msg = smtp_connection.ehlo()
        assert response == 250
        assert b"smtp.gmail.com" in msg
>       assert 0  # for demo purposes
E       assert 0

test_module.py:7: AssertionError
________________________________ test_noop _________________________________

smtp_connection = <smtplib.SMTP object at 0xdeadbeef>

    def test_noop(smtp_connection):
        response, msg = smtp_connection.noop()
        assert response == 250
>       assert 0  # for demo purposes
E       assert 0

test_module.py:13: AssertionError
============================ 2 failed in 0.12s =============================

You see the two assert 0 failing and more importantly you can also see that the same (module-scoped) smtp_connection object was passed into the two test functions because pytest shows the incoming argument values in the traceback. As a result, the two test functions using smtp_connection run as quick as a single one because they reuse the same instance.

If you decide that you rather want to have a session-scoped smtp_connection instance, you can simply declare it:

@pytest.fixture(scope="session")
def smtp_connection():
    # the returned fixture value will be shared for
    # all tests needing it
    ...

Finally, the class scope will invoke the fixture once per test class.

Note

Pytest will only cache one instance of a fixture at a time. This means that when using a parametrized fixture, pytest may invoke a fixture more than once in the given scope.

package scope (experimental)

In pytest 3.7 the package scope has been introduced. Package-scoped fixtures are finalized when the last test of a package finishes.

Warning

This functionality is considered experimental and may be removed in future versions if hidden corner-cases or serious problems with this functionality are discovered after it gets more usage in the wild.

Use this new feature sparingly and please make sure to report any issues you find.

Dynamic scope

New in version 5.2.

In some cases, you might want to change the scope of the fixture without changing the code. To do that, pass a callable to scope. The callable must return a string with a valid scope and will be executed only once - during the fixture definition. It will be called with two keyword arguments - fixture_name as a string and config with a configuration object.

This can be especially useful when dealing with fixtures that need time for setup, like spawning a docker container. You can use the command-line argument to control the scope of the spawned containers for different environments. See the example below.

def determine_scope(fixture_name, config):
    if config.getoption("--keep-containers"):
        return "session"
    return "function"


@pytest.fixture(scope=determine_scope)
def docker_container():
    yield spawn_container()

Order: Higher-scoped fixtures are instantiated first

Within a function request for features, fixture of higher-scopes (such as session) are instantiated first than lower-scoped fixtures (such as function or class). The relative order of fixtures of same scope follows the declared order in the test function and honours dependencies between fixtures. Autouse fixtures will be instantiated before explicitly used fixtures.

Consider the code below:

import pytest

# fixtures documentation order example
order = []


@pytest.fixture(scope="session")
def s1():
    order.append("s1")


@pytest.fixture(scope="module")
def m1():
    order.append("m1")


@pytest.fixture
def f1(f3):
    order.append("f1")


@pytest.fixture
def f3():
    order.append("f3")


@pytest.fixture(autouse=True)
def a1():
    order.append("a1")


@pytest.fixture
def f2():
    order.append("f2")


def test_order(f1, m1, f2, s1):
    assert order == ["s1", "m1", "a1", "f3", "f1", "f2"]

The fixtures requested by test_order will be instantiated in the following order:

1. s1: is the highest-scoped fixture (session).
2. m1: is the second highest-scoped fixture (module).
3. a1: is a function-scoped autouse fixture: it will be instantiated before other fixtures within the same scope.
4. f3: is a function-scoped fixture, required by f1: it needs to be instantiated at this point
5. f1: is the first function-scoped fixture in test_order parameter list.
6. f2: is the last function-scoped fixture in test_order parameter list.

Fixture finalization / executing teardown code

pytest supports execution of fixture specific finalization code when the fixture goes out of scope. By using a yield statement instead of return, all the code after the yield statement serves as the teardown code:

# content of conftest.py

import smtplib
import pytest


@pytest.fixture(scope="module")
def smtp_connection():
    smtp_connection = smtplib.SMTP("smtp.gmail.com", 587, timeout=5)
    yield smtp_connection  # provide the fixture value
    print("teardown smtp")
    smtp_connection.close()

The print and smtp.close() statements will execute when the last test in the module has finished execution, regardless of the exception status of the tests.

Let’s execute it:

$ pytest -s -q --tb=no
FFteardown smtp

2 failed in 0.12s

We see that the smtp_connection instance is finalized after the two tests finished execution. Note that if we decorated our fixture function with scope='function' then fixture setup and cleanup would occur around each single test. In either case the test module itself does not need to change or know about these details of fixture setup.

Note that we can also seamlessly use the yield syntax with with statements:

# content of test_yield2.py

import smtplib
import pytest


@pytest.fixture(scope="module")
def smtp_connection():
    with smtplib.SMTP("smtp.gmail.com", 587, timeout=5) as smtp_connection:
        yield smtp_connection  # provide the fixture value

The smtp_connection connection will be closed after the test finished execution because the smtp_connection object automatically closes when the with statement ends.

Using the contextlib.ExitStack context manager finalizers will always be called regardless if the fixture setup code raises an exception. This is handy to properly close all resources created by a fixture even if one of them fails to be created/acquired:

# content of test_yield3.py

import contextlib

import pytest


@contextlib.contextmanager
def connect(port):
    ...  # create connection
    yield
    ...  # close connection


@pytest.fixture
def equipments():
    with contextlib.ExitStack() as stack:
        yield [stack.enter_context(connect(port)) for port in ("C1", "C3", "C28")]

In the example above, if "C28" fails with an exception, "C1" and "C3" will still be properly closed.

Note that if an exception happens during the setup code (before the yield keyword), the teardown code (after the yield) will not be called.

An alternative option for executing teardown code is to make use of the addfinalizer method of the request-context object to register finalization functions.

Here’s the smtp_connection fixture changed to use addfinalizer for cleanup:

# content of conftest.py
import smtplib
import pytest


@pytest.fixture(scope="module")
def smtp_connection(request):
    smtp_connection = smtplib.SMTP("smtp.gmail.com", 587, timeout=5)

    def fin():
        print("teardown smtp_connection")
        smtp_connection.close()

    request.addfinalizer(fin)
    return smtp_connection  # provide the fixture value

Here’s the equipments fixture changed to use addfinalizer for cleanup:

# content of test_yield3.py

import contextlib
import functools

import pytest


@contextlib.contextmanager
def connect(port):
    ...  # create connection
    yield
    ...  # close connection


@pytest.fixture
def equipments(request):
    r = []
    for port in ("C1", "C3", "C28"):
        cm = connect(port)
        equip = cm.__enter__()
        request.addfinalizer(functools.partial(cm.__exit__, None, None, None))
        r.append(equip)
    return r

Both yield and addfinalizer methods work similarly by calling their code after the test ends. Of course, if an exception happens before the finalize function is registered then it will not be executed.

Fixtures can introspect the requesting test context

Fixture functions can accept the request object to introspect the “requesting” test function, class or module context. Further extending the previous smtp_connection fixture example, let’s read an optional server URL from the test module which uses our fixture:

# content of conftest.py
import pytest
import smtplib


@pytest.fixture(scope="module")
def smtp_connection(request):
    server = getattr(request.module, "smtpserver", "smtp.gmail.com")
    smtp_connection = smtplib.SMTP(server, 587, timeout=5)
    yield smtp_connection
    print("finalizing {} ({})".format(smtp_connection, server))
    smtp_connection.close()

We use the request.module attribute to optionally obtain an smtpserver attribute from the test module. If we just execute again, nothing much has changed:

$ pytest -s -q --tb=no
FFfinalizing <smtplib.SMTP object at 0xdeadbeef> (smtp.gmail.com)

2 failed in 0.12s

Let’s quickly create another test module that actually sets the server URL in its module namespace:

# content of test_anothersmtp.py

smtpserver = "mail.python.org"  # will be read by smtp fixture


def test_showhelo(smtp_connection):
    assert 0, smtp_connection.helo()

Running it:

$ pytest -qq --tb=short test_anothersmtp.py
F                                                                    [100%]
================================= FAILURES =================================
______________________________ test_showhelo _______________________________
test_anothersmtp.py:6: in test_showhelo
    assert 0, smtp_connection.helo()
E   AssertionError: (250, b'mail.python.org')
E   assert 0
------------------------- Captured stdout teardown -------------------------
finalizing <smtplib.SMTP object at 0xdeadbeef> (mail.python.org)

voila! The smtp_connection fixture function picked up our mail server name from the module namespace.

Factories as fixtures

The “factory as fixture” pattern can help in situations where the result of a fixture is needed multiple times in a single test. Instead of returning data directly, the fixture instead returns a function which generates the data. This function can then be called multiple times in the test.

Factories can have parameters as needed:

@pytest.fixture
def make_customer_record():
    def _make_customer_record(name):
        return {"name": name, "orders": []}

    return _make_customer_record


def test_customer_records(make_customer_record):
    customer_1 = make_customer_record("Lisa")
    customer_2 = make_customer_record("Mike")
    customer_3 = make_customer_record("Meredith")

If the data created by the factory requires managing, the fixture can take care of that:

@pytest.fixture
def make_customer_record():

    created_records = []

    def _make_customer_record(name):
        record = models.Customer(name=name, orders=[])
        created_records.append(record)
        return record

    yield _make_customer_record

    for record in created_records:
        record.destroy()


def test_customer_records(make_customer_record):
    customer_1 = make_customer_record("Lisa")
    customer_2 = make_customer_record("Mike")
    customer_3 = make_customer_record("Meredith")

Parametrizing fixtures

Fixture functions can be parametrized in which case they will be called multiple times, each time executing the set of dependent tests, i. e. the tests that depend on this fixture. Test functions usually do not need to be aware of their re-running. Fixture parametrization helps to write exhaustive functional tests for components which themselves can be configured in multiple ways.

Extending the previous example, we can flag the fixture to create two smtp_connection fixture instances which will cause all tests using the fixture to run twice. The fixture function gets access to each parameter through the special request object:

# content of conftest.py
import pytest
import smtplib


@pytest.fixture(scope="module", params=["smtp.gmail.com", "mail.python.org"])
def smtp_connection(request):
    smtp_connection = smtplib.SMTP(request.param, 587, timeout=5)
    yield smtp_connection
    print("finalizing {}".format(smtp_connection))
    smtp_connection.close()

The main change is the declaration of params with @pytest.fixture, a list of values for each of which the fixture function will execute and can access a value via request.param. No test function code needs to change. So let’s just do another run:

$ pytest -q test_module.py
FFFF                                                                 [100%]
================================= FAILURES =================================
________________________ test_ehlo[smtp.gmail.com] _________________________

smtp_connection = <smtplib.SMTP object at 0xdeadbeef>

    def test_ehlo(smtp_connection):
        response, msg = smtp_connection.ehlo()
        assert response == 250
        assert b"smtp.gmail.com" in msg
>       assert 0  # for demo purposes
E       assert 0

test_module.py:7: AssertionError
________________________ test_noop[smtp.gmail.com] _________________________

smtp_connection = <smtplib.SMTP object at 0xdeadbeef>

    def test_noop(smtp_connection):
        response, msg = smtp_connection.noop()
        assert response == 250
>       assert 0  # for demo purposes
E       assert 0

test_module.py:13: AssertionError
________________________ test_ehlo[mail.python.org] ________________________

smtp_connection = <smtplib.SMTP object at 0xdeadbeef>

    def test_ehlo(smtp_connection):
        response, msg = smtp_connection.ehlo()
        assert response == 250
>       assert b"smtp.gmail.com" in msg
E       AssertionError: assert b'smtp.gmail.com' in b'mail.python.org\nPIPELINING\nSIZE 51200000\nETRN\nSTARTTLS\nAUTH DIGEST-MD5 NTLM CRAM-MD5\nENHANCEDSTATUSCODES\n8BITMIME\nDSN\nSMTPUTF8\nCHUNKING'

test_module.py:6: AssertionError
-------------------------- Captured stdout setup ---------------------------
finalizing <smtplib.SMTP object at 0xdeadbeef>
________________________ test_noop[mail.python.org] ________________________

smtp_connection = <smtplib.SMTP object at 0xdeadbeef>

    def test_noop(smtp_connection):
        response, msg = smtp_connection.noop()
        assert response == 250
>       assert 0  # for demo purposes
E       assert 0

test_module.py:13: AssertionError
------------------------- Captured stdout teardown -------------------------
finalizing <smtplib.SMTP object at 0xdeadbeef>
4 failed in 0.12s

We see that our two test functions each ran twice, against the different smtp_connection instances. Note also, that with the mail.python.org connection the second test fails in test_ehlo because a different server string is expected than what arrived.

pytest will build a string that is the test ID for each fixture value in a parametrized fixture, e.g. test_ehlo[smtp.gmail.com] and test_ehlo[mail.python.org] in the above examples. These IDs can be used with -k to select specific cases to run, and they will also identify the specific case when one is failing. Running pytest with --collect-only will show the generated IDs.

Numbers, strings, booleans and None will have their usual string representation used in the test ID. For other objects, pytest will make a string based on the argument name. It is possible to customise the string used in a test ID for a certain fixture value by using the ids keyword argument:

# content of test_ids.py
import pytest


@pytest.fixture(params=[0, 1], ids=["spam", "ham"])
def a(request):
    return request.param


def test_a(a):
    pass


def idfn(fixture_value):
    if fixture_value == 0:
        return "eggs"
    else:
        return None


@pytest.fixture(params=[0, 1], ids=idfn)
def b(request):
    return request.param


def test_b(b):
    pass

The above shows how ids can be either a list of strings to use or a function which will be called with the fixture value and then has to return a string to use. In the latter case if the function return None then pytest’s auto-generated ID will be used.

Running the above tests results in the following test IDs being used:

$ pytest --collect-only
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 10 items
<Module test_anothersmtp.py>
  <Function test_showhelo[smtp.gmail.com]>
  <Function test_showhelo[mail.python.org]>
<Module test_ids.py>
  <Function test_a[spam]>
  <Function test_a[ham]>
  <Function test_b[eggs]>
  <Function test_b[1]>
<Module test_module.py>
  <Function test_ehlo[smtp.gmail.com]>
  <Function test_noop[smtp.gmail.com]>
  <Function test_ehlo[mail.python.org]>
  <Function test_noop[mail.python.org]>

========================== no tests ran in 0.12s ===========================

Using marks with parametrized fixtures

pytest.param() can be used to apply marks in values sets of parametrized fixtures in the same way that they can be used with @pytest.mark.parametrize.

Example:

# content of test_fixture_marks.py
import pytest


@pytest.fixture(params=[0, 1, pytest.param(2, marks=pytest.mark.skip)])
def data_set(request):
    return request.param


def test_data(data_set):
    pass

Running this test will skip the invocation of data_set with value 2:

$ pytest test_fixture_marks.py -v
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y -- $PYTHON_PREFIX/bin/python
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collecting ... collected 3 items

test_fixture_marks.py::test_data[0] PASSED                           [ 33%]
test_fixture_marks.py::test_data[1] PASSED                           [ 66%]
test_fixture_marks.py::test_data[2] SKIPPED                          [100%]

======================= 2 passed, 1 skipped in 0.12s =======================

Modularity: using fixtures from a fixture function

You can not only use fixtures in test functions but fixture functions can use other fixtures themselves. This contributes to a modular design of your fixtures and allows re-use of framework-specific fixtures across many projects. As a simple example, we can extend the previous example and instantiate an object app where we stick the already defined smtp_connection resource into it:

# content of test_appsetup.py

import pytest


class App:
    def __init__(self, smtp_connection):
        self.smtp_connection = smtp_connection


@pytest.fixture(scope="module")
def app(smtp_connection):
    return App(smtp_connection)


def test_smtp_connection_exists(app):
    assert app.smtp_connection

Here we declare an app fixture which receives the previously defined smtp_connection fixture and instantiates an App object with it. Let’s run it:

$ pytest -v test_appsetup.py
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y -- $PYTHON_PREFIX/bin/python
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collecting ... collected 2 items

test_appsetup.py::test_smtp_connection_exists[smtp.gmail.com] PASSED [ 50%]
test_appsetup.py::test_smtp_connection_exists[mail.python.org] PASSED [100%]

============================ 2 passed in 0.12s =============================

Due to the parametrization of smtp_connection, the test will run twice with two different App instances and respective smtp servers. There is no need for the app fixture to be aware of the smtp_connection parametrization because pytest will fully analyse the fixture dependency graph.

Note that the app fixture has a scope of module and uses a module-scoped smtp_connection fixture. The example would still work if smtp_connection was cached on a session scope: it is fine for fixtures to use “broader” scoped fixtures but not the other way round: A session-scoped fixture could not use a module-scoped one in a meaningful way.

Automatic grouping of tests by fixture instances

pytest minimizes the number of active fixtures during test runs. If you have a parametrized fixture, then all the tests using it will first execute with one instance and then finalizers are called before the next fixture instance is created. Among other things, this eases testing of applications which create and use global state.

The following example uses two parametrized fixtures, one of which is scoped on a per-module basis, and all the functions perform print calls to show the setup/teardown flow:

# content of test_module.py
import pytest


@pytest.fixture(scope="module", params=["mod1", "mod2"])
def modarg(request):
    param = request.param
    print("  SETUP modarg", param)
    yield param
    print("  TEARDOWN modarg", param)


@pytest.fixture(scope="function", params=[1, 2])
def otherarg(request):
    param = request.param
    print("  SETUP otherarg", param)
    yield param
    print("  TEARDOWN otherarg", param)


def test_0(otherarg):
    print("  RUN test0 with otherarg", otherarg)


def test_1(modarg):
    print("  RUN test1 with modarg", modarg)


def test_2(otherarg, modarg):
    print("  RUN test2 with otherarg {} and modarg {}".format(otherarg, modarg))

Let’s run the tests in verbose mode and with looking at the print-output:

$ pytest -v -s test_module.py
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y -- $PYTHON_PREFIX/bin/python
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collecting ... collected 8 items

test_module.py::test_0[1]   SETUP otherarg 1
  RUN test0 with otherarg 1
PASSED  TEARDOWN otherarg 1

test_module.py::test_0[2]   SETUP otherarg 2
  RUN test0 with otherarg 2
PASSED  TEARDOWN otherarg 2

test_module.py::test_1[mod1]   SETUP modarg mod1
  RUN test1 with modarg mod1
PASSED
test_module.py::test_2[mod1-1]   SETUP otherarg 1
  RUN test2 with otherarg 1 and modarg mod1
PASSED  TEARDOWN otherarg 1

test_module.py::test_2[mod1-2]   SETUP otherarg 2
  RUN test2 with otherarg 2 and modarg mod1
PASSED  TEARDOWN otherarg 2

test_module.py::test_1[mod2]   TEARDOWN modarg mod1
  SETUP modarg mod2
  RUN test1 with modarg mod2
PASSED
test_module.py::test_2[mod2-1]   SETUP otherarg 1
  RUN test2 with otherarg 1 and modarg mod2
PASSED  TEARDOWN otherarg 1

test_module.py::test_2[mod2-2]   SETUP otherarg 2
  RUN test2 with otherarg 2 and modarg mod2
PASSED  TEARDOWN otherarg 2
  TEARDOWN modarg mod2


============================ 8 passed in 0.12s =============================

You can see that the parametrized module-scoped modarg resource caused an ordering of test execution that lead to the fewest possible “active” resources. The finalizer for the mod1 parametrized resource was executed before the mod2 resource was setup.

In particular notice that test_0 is completely independent and finishes first. Then test_1 is executed with mod1, then test_2 with mod1, then test_1 with mod2 and finally test_2 with mod2.

The otherarg parametrized resource (having function scope) was set up before and teared down after every test that used it.

Using fixtures from classes, modules or projects

Sometimes test functions do not directly need access to a fixture object. For example, tests may require to operate with an empty directory as the current working directory but otherwise do not care for the concrete directory. Here is how you can use the standard tempfile and pytest fixtures to achieve it. We separate the creation of the fixture into a conftest.py file:

# content of conftest.py

import os
import shutil
import tempfile

import pytest


@pytest.fixture()
def cleandir():
    newpath = tempfile.mkdtemp()
    os.chdir(newpath)
    yield
    shutil.rmtree(newpath)

and declare its use in a test module via a usefixtures marker:

# content of test_setenv.py
import os
import pytest


@pytest.mark.usefixtures("cleandir")
class TestDirectoryInit:
    def test_cwd_starts_empty(self):
        assert os.listdir(os.getcwd()) == []
        with open("myfile", "w") as f:
            f.write("hello")

    def test_cwd_again_starts_empty(self):
        assert os.listdir(os.getcwd()) == []

Due to the usefixtures marker, the cleandir fixture will be required for the execution of each test method, just as if you specified a “cleandir” function argument to each of them. Let’s run it to verify our fixture is activated and the tests pass:

$ pytest -q
..                                                                   [100%]
2 passed in 0.12s

You can specify multiple fixtures like this:

@pytest.mark.usefixtures("cleandir", "anotherfixture")
def test():
    ...

and you may specify fixture usage at the test module level, using a generic feature of the mark mechanism:

pytestmark = pytest.mark.usefixtures("cleandir")

Note that the assigned variable must be called pytestmark, assigning e.g. foomark will not activate the fixtures.

It is also possible to put fixtures required by all tests in your project into an ini-file:

# content of pytest.ini
[pytest]
usefixtures = cleandir

Warning

Note this mark has no effect in fixture functions. For example, this will not work as expected:

@pytest.mark.usefixtures("my_other_fixture")
@pytest.fixture
def my_fixture_that_sadly_wont_use_my_other_fixture():
    ...

Currently this will not generate any error or warning, but this is intended to be handled by #3664.

Autouse fixtures (xUnit setup on steroids)

Occasionally, you may want to have fixtures get invoked automatically without declaring a function argument explicitly or a usefixtures decorator. As a practical example, suppose we have a database fixture which has a begin/rollback/commit architecture and we want to automatically surround each test method by a transaction and a rollback. Here is a dummy self-contained implementation of this idea:

# content of test_db_transact.py

import pytest


class DB:
    def __init__(self):
        self.intransaction = []

    def begin(self, name):
        self.intransaction.append(name)

    def rollback(self):
        self.intransaction.pop()


@pytest.fixture(scope="module")
def db():
    return DB()


class TestClass:
    @pytest.fixture(autouse=True)
    def transact(self, request, db):
        db.begin(request.function.__name__)
        yield
        db.rollback()

    def test_method1(self, db):
        assert db.intransaction == ["test_method1"]

    def test_method2(self, db):
        assert db.intransaction == ["test_method2"]

The class-level transact fixture is marked with autouse=true which implies that all test methods in the class will use this fixture without a need to state it in the test function signature or with a class-level usefixtures decorator.

If we run it, we get two passing tests:

$ pytest -q
..                                                                   [100%]
2 passed in 0.12s

Here is how autouse fixtures work in other scopes:

• autouse fixtures obey the scope= keyword-argument: if an autouse fixture has scope='session' it will only be run once, no matter where it is defined. scope='class' means it will be run once per class, etc.
• if an autouse fixture is defined in a test module, all its test functions automatically use it.
• if an autouse fixture is defined in a conftest.py file then all tests in all test modules below its directory will invoke the fixture.
• lastly, and please use that with care: if you define an autouse fixture in a plugin, it will be invoked for all tests in all projects where the plugin is installed. This can be useful if a fixture only anyway works in the presence of certain settings e. g. in the ini-file. Such a global fixture should always quickly determine if it should do any work and avoid otherwise expensive imports or computation.

Note that the above transact fixture may very well be a fixture that you want to make available in your project without having it generally active. The canonical way to do that is to put the transact definition into a conftest.py file without using autouse:

# content of conftest.py
@pytest.fixture
def transact(request, db):
    db.begin()
    yield
    db.rollback()

and then e.g. have a TestClass using it by declaring the need:

@pytest.mark.usefixtures("transact")
class TestClass:
    def test_method1(self):
        ...

All test methods in this TestClass will use the transaction fixture while other test classes or functions in the module will not use it unless they also add a transact reference.

Overriding fixtures on various levels

In relatively large test suite, you most likely need to override a global or root fixture with a locally defined one, keeping the test code readable and maintainable.

Override a fixture on a folder (conftest) level

Given the tests file structure is:

tests/
    __init__.py

    conftest.py
        # content of tests/conftest.py
        import pytest

        @pytest.fixture
        def username():
            return 'username'

    test_something.py
        # content of tests/test_something.py
        def test_username(username):
            assert username == 'username'

    subfolder/
        __init__.py

        conftest.py
            # content of tests/subfolder/conftest.py
            import pytest

            @pytest.fixture
            def username(username):
                return 'overridden-' + username

        test_something.py
            # content of tests/subfolder/test_something.py
            def test_username(username):
                assert username == 'overridden-username'

As you can see, a fixture with the same name can be overridden for certain test folder level. Note that the base or super fixture can be accessed from the overriding fixture easily - used in the example above.

Override a fixture on a test module level

Given the tests file structure is:

tests/
    __init__.py

    conftest.py
        # content of tests/conftest.py
        import pytest

        @pytest.fixture
        def username():
            return 'username'

    test_something.py
        # content of tests/test_something.py
        import pytest

        @pytest.fixture
        def username(username):
            return 'overridden-' + username

        def test_username(username):
            assert username == 'overridden-username'

    test_something_else.py
        # content of tests/test_something_else.py
        import pytest

        @pytest.fixture
        def username(username):
            return 'overridden-else-' + username

        def test_username(username):
            assert username == 'overridden-else-username'

In the example above, a fixture with the same name can be overridden for certain test module.

Override a fixture with direct test parametrization

Given the tests file structure is:

tests/
    __init__.py

    conftest.py
        # content of tests/conftest.py
        import pytest

        @pytest.fixture
        def username():
            return 'username'

        @pytest.fixture
        def other_username(username):
            return 'other-' + username

    test_something.py
        # content of tests/test_something.py
        import pytest

        @pytest.mark.parametrize('username', ['directly-overridden-username'])
        def test_username(username):
            assert username == 'directly-overridden-username'

        @pytest.mark.parametrize('username', ['directly-overridden-username-other'])
        def test_username_other(other_username):
            assert other_username == 'other-directly-overridden-username-other'

In the example above, a fixture value is overridden by the test parameter value. Note that the value of the fixture can be overridden this way even if the test doesn’t use it directly (doesn’t mention it in the function prototype).

Override a parametrized fixture with non-parametrized one and vice versa

Given the tests file structure is:

tests/
    __init__.py

    conftest.py
        # content of tests/conftest.py
        import pytest

        @pytest.fixture(params=['one', 'two', 'three'])
        def parametrized_username(request):
            return request.param

        @pytest.fixture
        def non_parametrized_username(request):
            return 'username'

    test_something.py
        # content of tests/test_something.py
        import pytest

        @pytest.fixture
        def parametrized_username():
            return 'overridden-username'

        @pytest.fixture(params=['one', 'two', 'three'])
        def non_parametrized_username(request):
            return request.param

        def test_username(parametrized_username):
            assert parametrized_username == 'overridden-username'

        def test_parametrized_username(non_parametrized_username):
            assert non_parametrized_username in ['one', 'two', 'three']

    test_something_else.py
        # content of tests/test_something_else.py
        def test_username(parametrized_username):
            assert parametrized_username in ['one', 'two', 'three']

        def test_username(non_parametrized_username):
            assert non_parametrized_username == 'username'

In the example above, a parametrized fixture is overridden with a non-parametrized version, and a non-parametrized fixture is overridden with a parametrized version for certain test module. The same applies for the test folder level obviously.

Marking test functions with attributes

By using the pytest.mark helper you can easily set metadata on your test functions. There are some builtin markers, for example:

• skip - always skip a test function
• skipif - skip a test function if a certain condition is met
• xfail - produce an “expected failure” outcome if a certain condition is met
• parametrize to perform multiple calls to the same test function.

It’s easy to create custom markers or to apply markers to whole test classes or modules. Those markers can be used by plugins, and also are commonly used to select tests on the command-line with the -m option.

See Working with custom markers for examples which also serve as documentation.

Note

Marks can only be applied to tests, having no effect on fixtures.

Registering marks

You can register custom marks in your pytest.ini file like this:

[pytest]
markers =
    slow: marks tests as slow (deselect with '-m "not slow"')
    serial

Note that everything after the : is an optional description.

Alternatively, you can register new markers programmatically in a pytest_configure hook:

def pytest_configure(config):
    config.addinivalue_line(
        "markers", "env(name): mark test to run only on named environment"
    )

Registered marks appear in pytest’s help text and do not emit warnings (see the next section). It is recommended that third-party plugins always register their markers.

Raising errors on unknown marks

Unregistered marks applied with the @pytest.mark.name_of_the_mark decorator will always emit a warning in order to avoid silently doing something surprising due to mis-typed names. As described in the previous section, you can disable the warning for custom marks by registering them in your pytest.ini file or using a custom pytest_configure hook.

When the --strict-markers command-line flag is passed, any unknown marks applied with the @pytest.mark.name_of_the_mark decorator will trigger an error. You can enforce this validation in your project by adding --strict-markers to addopts:

[pytest]
addopts = --strict-markers
markers =
    slow: marks tests as slow (deselect with '-m "not slow"')
    serial

Monkeypatching/mocking modules and environments

Sometimes tests need to invoke functionality which depends on global settings or which invokes code which cannot be easily tested such as network access. The monkeypatch fixture helps you to safely set/delete an attribute, dictionary item or environment variable, or to modify sys.path for importing.

The monkeypatch fixture provides these helper methods for safely patching and mocking functionality in tests:

monkeypatch.setattr(obj, name, value, raising=True)
monkeypatch.delattr(obj, name, raising=True)
monkeypatch.setitem(mapping, name, value)
monkeypatch.delitem(obj, name, raising=True)
monkeypatch.setenv(name, value, prepend=False)
monkeypatch.delenv(name, raising=True)
monkeypatch.syspath_prepend(path)
monkeypatch.chdir(path)

All modifications will be undone after the requesting test function or fixture has finished. The raising parameter determines if a KeyError or AttributeError will be raised if the target of the set/deletion operation does not exist.

Consider the following scenarios:

1. Modifying the behavior of a function or the property of a class for a test e.g. there is an API call or database connection you will not make for a test but you know what the expected output should be. Use monkeypatch.setattr() to patch the function or property with your desired testing behavior. This can include your own functions. Use monkeypatch.delattr() to remove the function or property for the test.

2. Modifying the values of dictionaries e.g. you have a global configuration that you want to modify for certain test cases. Use monkeypatch.setitem() to patch the dictionary for the test. monkeypatch.delitem() can be used to remove items.

3. Modifying environment variables for a test e.g. to test program behavior if an environment variable is missing, or to set multiple values to a known variable. monkeypatch.setenv() and monkeypatch.delenv() can be used for these patches.

4. Use monkeypatch.setenv("PATH", value, prepend=os.pathsep) to modify $PATH, and monkeypatch.chdir() to change the context of the current working directory during a test.

5. Use monkeypatch.syspath_prepend() to modify sys.path which will also call pkg_resources.fixup_namespace_packages() and importlib.invalidate_caches().

See the monkeypatch blog post for some introduction material and a discussion of its motivation.

Simple example: monkeypatching functions

Consider a scenario where you are working with user directories. In the context of testing, you do not want your test to depend on the running user. monkeypatch can be used to patch functions dependent on the user to always return a specific value.

In this example, monkeypatch.setattr() is used to patch Path.home so that the known testing path Path("/abc") is always used when the test is run. This removes any dependency on the running user for testing purposes. monkeypatch.setattr() must be called before the function which will use the patched function is called. After the test function finishes the Path.home modification will be undone.

# contents of test_module.py with source code and the test
from pathlib import Path


def getssh():
    """Simple function to return expanded homedir ssh path."""
    return Path.home() / ".ssh"


def test_getssh(monkeypatch):
    # mocked return function to replace Path.home
    # always return '/abc'
    def mockreturn():
        return Path("/abc")

    # Application of the monkeypatch to replace Path.home
    # with the behavior of mockreturn defined above.
    monkeypatch.setattr(Path, "home", mockreturn)

    # Calling getssh() will use mockreturn in place of Path.home
    # for this test with the monkeypatch.
    x = getssh()
    assert x == Path("/abc/.ssh")

Monkeypatching returned objects: building mock classes

monkeypatch.setattr() can be used in conjunction with classes to mock returned objects from functions instead of values. Imagine a simple function to take an API url and return the json response.

# contents of app.py, a simple API retrieval example
import requests


def get_json(url):
    """Takes a URL, and returns the JSON."""
    r = requests.get(url)
    return r.json()

We need to mock r, the returned response object for testing purposes. The mock of r needs a .json() method which returns a dictionary. This can be done in our test file by defining a class to represent r.

# contents of test_app.py, a simple test for our API retrieval
# import requests for the purposes of monkeypatching
import requests

# our app.py that includes the get_json() function
# this is the previous code block example
import app

# custom class to be the mock return value
# will override the requests.Response returned from requests.get
class MockResponse:

    # mock json() method always returns a specific testing dictionary
    @staticmethod
    def json():
        return {"mock_key": "mock_response"}


def test_get_json(monkeypatch):

    # Any arguments may be passed and mock_get() will always return our
    # mocked object, which only has the .json() method.
    def mock_get(*args, **kwargs):
        return MockResponse()

    # apply the monkeypatch for requests.get to mock_get
    monkeypatch.setattr(requests, "get", mock_get)

    # app.get_json, which contains requests.get, uses the monkeypatch
    result = app.get_json("https://fakeurl")
    assert result["mock_key"] == "mock_response"

monkeypatch applies the mock for requests.get with our mock_get function. The mock_get function returns an instance of the MockResponse class, which has a json() method defined to return a known testing dictionary and does not require any outside API connection.

You can build the MockResponse class with the appropriate degree of complexity for the scenario you are testing. For instance, it could include an ok property that always returns True, or return different values from the json() mocked method based on input strings.

This mock can be shared across tests using a fixture:

# contents of test_app.py, a simple test for our API retrieval
import pytest
import requests

# app.py that includes the get_json() function
import app

# custom class to be the mock return value of requests.get()
class MockResponse:
    @staticmethod
    def json():
        return {"mock_key": "mock_response"}


# monkeypatched requests.get moved to a fixture
@pytest.fixture
def mock_response(monkeypatch):
    """Requests.get() mocked to return {'mock_key':'mock_response'}."""

    def mock_get(*args, **kwargs):
        return MockResponse()

    monkeypatch.setattr(requests, "get", mock_get)


# notice our test uses the custom fixture instead of monkeypatch directly
def test_get_json(mock_response):
    result = app.get_json("https://fakeurl")
    assert result["mock_key"] == "mock_response"

Furthermore, if the mock was designed to be applied to all tests, the fixture could be moved to a conftest.py file and use the with autouse=True option.

Global patch example: preventing “requests” from remote operations

If you want to prevent the “requests” library from performing http requests in all your tests, you can do:

# contents of conftest.py
import pytest


@pytest.fixture(autouse=True)
def no_requests(monkeypatch):
    """Remove requests.sessions.Session.request for all tests."""
    monkeypatch.delattr("requests.sessions.Session.request")

This autouse fixture will be executed for each test function and it will delete the method request.session.Session.request so that any attempts within tests to create http requests will fail.

Note

Be advised that it is not recommended to patch builtin functions such as open, compile, etc., because it might break pytest’s internals. If that’s unavoidable, passing --tb=native, --assert=plain and --capture=no might help although there’s no guarantee.

Note

Mind that patching stdlib functions and some third-party libraries used by pytest might break pytest itself, therefore in those cases it is recommended to use MonkeyPatch.context() to limit the patching to the block you want tested:

import functools


def test_partial(monkeypatch):
    with monkeypatch.context() as m:
        m.setattr(functools, "partial", 3)
        assert functools.partial == 3

See issue #3290 for details.

Monkeypatching environment variables

If you are working with environment variables you often need to safely change the values or delete them from the system for testing purposes. monkeypatch provides a mechanism to do this using the setenv and delenv method. Our example code to test:

# contents of our original code file e.g. code.py
import os


def get_os_user_lower():
    """Simple retrieval function.
    Returns lowercase USER or raises EnvironmentError."""
    username = os.getenv("USER")

    if username is None:
        raise OSError("USER environment is not set.")

    return username.lower()

There are two potential paths. First, the USER environment variable is set to a value. Second, the USER environment variable does not exist. Using monkeypatch both paths can be safely tested without impacting the running environment:

# contents of our test file e.g. test_code.py
import pytest


def test_upper_to_lower(monkeypatch):
    """Set the USER env var to assert the behavior."""
    monkeypatch.setenv("USER", "TestingUser")
    assert get_os_user_lower() == "testinguser"


def test_raise_exception(monkeypatch):
    """Remove the USER env var and assert EnvironmentError is raised."""
    monkeypatch.delenv("USER", raising=False)

    with pytest.raises(OSError):
        _ = get_os_user_lower()

This behavior can be moved into fixture structures and shared across tests:

# contents of our test file e.g. test_code.py
import pytest


@pytest.fixture
def mock_env_user(monkeypatch):
    monkeypatch.setenv("USER", "TestingUser")


@pytest.fixture
def mock_env_missing(monkeypatch):
    monkeypatch.delenv("USER", raising=False)


# notice the tests reference the fixtures for mocks
def test_upper_to_lower(mock_env_user):
    assert get_os_user_lower() == "testinguser"


def test_raise_exception(mock_env_missing):
    with pytest.raises(OSError):
        _ = get_os_user_lower()

Monkeypatching dictionaries

monkeypatch.setitem() can be used to safely set the values of dictionaries to specific values during tests. Take this simplified connection string example:

# contents of app.py to generate a simple connection string
DEFAULT_CONFIG = {"user": "user1", "database": "db1"}


def create_connection_string(config=None):
    """Creates a connection string from input or defaults."""
    config = config or DEFAULT_CONFIG
    return f"User Id={config['user']}; Location={config['database']};"

For testing purposes we can patch the DEFAULT_CONFIG dictionary to specific values.

# contents of test_app.py
# app.py with the connection string function (prior code block)
import app


def test_connection(monkeypatch):

    # Patch the values of DEFAULT_CONFIG to specific
    # testing values only for this test.
    monkeypatch.setitem(app.DEFAULT_CONFIG, "user", "test_user")
    monkeypatch.setitem(app.DEFAULT_CONFIG, "database", "test_db")

    # expected result based on the mocks
    expected = "User Id=test_user; Location=test_db;"

    # the test uses the monkeypatched dictionary settings
    result = app.create_connection_string()
    assert result == expected

You can use the monkeypatch.delitem() to remove values.

# contents of test_app.py
import pytest

# app.py with the connection string function
import app


def test_missing_user(monkeypatch):

    # patch the DEFAULT_CONFIG t be missing the 'user' key
    monkeypatch.delitem(app.DEFAULT_CONFIG, "user", raising=False)

    # Key error expected because a config is not passed, and the
    # default is now missing the 'user' entry.
    with pytest.raises(KeyError):
        _ = app.create_connection_string()

The modularity of fixtures gives you the flexibility to define separate fixtures for each potential mock and reference them in the needed tests.

# contents of test_app.py
import pytest

# app.py with the connection string function
import app

# all of the mocks are moved into separated fixtures
@pytest.fixture
def mock_test_user(monkeypatch):
    """Set the DEFAULT_CONFIG user to test_user."""
    monkeypatch.setitem(app.DEFAULT_CONFIG, "user", "test_user")


@pytest.fixture
def mock_test_database(monkeypatch):
    """Set the DEFAULT_CONFIG database to test_db."""
    monkeypatch.setitem(app.DEFAULT_CONFIG, "database", "test_db")


@pytest.fixture
def mock_missing_default_user(monkeypatch):
    """Remove the user key from DEFAULT_CONFIG"""
    monkeypatch.delitem(app.DEFAULT_CONFIG, "user", raising=False)


# tests reference only the fixture mocks that are needed
def test_connection(mock_test_user, mock_test_database):

    expected = "User Id=test_user; Location=test_db;"

    result = app.create_connection_string()
    assert result == expected


def test_missing_user(mock_missing_default_user):

    with pytest.raises(KeyError):
        _ = app.create_connection_string()

API Reference

Consult the docs for the MonkeyPatch class.

Temporary directories and files

The tmp_path fixture

You can use the tmp_path fixture which will provide a temporary directory unique to the test invocation, created in the base temporary directory.

tmp_path is a pathlib/pathlib2.Path object. Here is an example test usage:

# content of test_tmp_path.py
import os

CONTENT = "content"


def test_create_file(tmp_path):
    d = tmp_path / "sub"
    d.mkdir()
    p = d / "hello.txt"
    p.write_text(CONTENT)
    assert p.read_text() == CONTENT
    assert len(list(tmp_path.iterdir())) == 1
    assert 0

Running this would result in a passed test except for the last assert 0 line which we use to look at values:

$ pytest test_tmp_path.py
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 1 item

test_tmp_path.py F                                                   [100%]

================================= FAILURES =================================
_____________________________ test_create_file _____________________________

tmp_path = PosixPath('PYTEST_TMPDIR/test_create_file0')

    def test_create_file(tmp_path):
        d = tmp_path / "sub"
        d.mkdir()
        p = d / "hello.txt"
        p.write_text(CONTENT)
        assert p.read_text() == CONTENT
        assert len(list(tmp_path.iterdir())) == 1
>       assert 0
E       assert 0

test_tmp_path.py:13: AssertionError
============================ 1 failed in 0.12s =============================

The tmp_path_factory fixture

The tmp_path_factory is a session-scoped fixture which can be used to create arbitrary temporary directories from any other fixture or test.

It is intended to replace tmpdir_factory, and returns pathlib.Path instances.

See tmp_path_factory API for details.

The ‘tmpdir’ fixture

You can use the tmpdir fixture which will provide a temporary directory unique to the test invocation, created in the base temporary directory.

tmpdir is a py.path.local object which offers os.path methods and more. Here is an example test usage:

# content of test_tmpdir.py
import os


def test_create_file(tmpdir):
    p = tmpdir.mkdir("sub").join("hello.txt")
    p.write("content")
    assert p.read() == "content"
    assert len(tmpdir.listdir()) == 1
    assert 0

Running this would result in a passed test except for the last assert 0 line which we use to look at values:

$ pytest test_tmpdir.py
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 1 item

test_tmpdir.py F                                                     [100%]

================================= FAILURES =================================
_____________________________ test_create_file _____________________________

tmpdir = local('PYTEST_TMPDIR/test_create_file0')

    def test_create_file(tmpdir):
        p = tmpdir.mkdir("sub").join("hello.txt")
        p.write("content")
        assert p.read() == "content"
        assert len(tmpdir.listdir()) == 1
>       assert 0
E       assert 0

test_tmpdir.py:9: AssertionError
============================ 1 failed in 0.12s =============================

The ‘tmpdir_factory’ fixture

The tmpdir_factory is a session-scoped fixture which can be used to create arbitrary temporary directories from any other fixture or test.

For example, suppose your test suite needs a large image on disk, which is generated procedurally. Instead of computing the same image for each test that uses it into its own tmpdir, you can generate it once per-session to save time:

# contents of conftest.py
import pytest


@pytest.fixture(scope="session")
def image_file(tmpdir_factory):
    img = compute_expensive_image()
    fn = tmpdir_factory.mktemp("data").join("img.png")
    img.save(str(fn))
    return fn


# contents of test_image.py
def test_histogram(image_file):
    img = load_image(image_file)
    # compute and test histogram

See tmpdir_factory API for details.

The default base temporary directory

Temporary directories are by default created as sub-directories of the system temporary directory. The base name will be pytest-NUM where NUM will be incremented with each test run. Moreover, entries older than 3 temporary directories will be removed.

You can override the default temporary directory setting like this:

pytest --basetemp=mydir

When distributing tests on the local machine, pytest takes care to configure a basetemp directory for the sub processes such that all temporary data lands below a single per-test run basetemp directory.

Capturing of the stdout/stderr output

Default stdout/stderr/stdin capturing behaviour

During test execution any output sent to stdout and stderr is captured. If a test or a setup method fails its according captured output will usually be shown along with the failure traceback. (this behavior can be configured by the --show-capture command-line option).

In addition, stdin is set to a “null” object which will fail on attempts to read from it because it is rarely desired to wait for interactive input when running automated tests.

By default capturing is done by intercepting writes to low level file descriptors. This allows to capture output from simple print statements as well as output from a subprocess started by a test.

Setting capturing methods or disabling capturing

There are two ways in which pytest can perform capturing:

• file descriptor (FD) level capturing (default): All writes going to the operating system file descriptors 1 and 2 will be captured.
• sys level capturing: Only writes to Python files sys.stdout and sys.stderr will be captured. No capturing of writes to filedescriptors is performed.

You can influence output capturing mechanisms from the command line:

pytest -s            # disable all capturing
pytest --capture=sys # replace sys.stdout/stderr with in-mem files
pytest --capture=fd  # also point filedescriptors 1 and 2 to temp file

Using print statements for debugging

One primary benefit of the default capturing of stdout/stderr output is that you can use print statements for debugging:

# content of test_module.py


def setup_function(function):
    print("setting up", function)


def test_func1():
    assert True


def test_func2():
    assert False

and running this module will show you precisely the output of the failing function and hide the other one:

$ pytest
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 2 items

test_module.py .F                                                    [100%]

================================= FAILURES =================================
________________________________ test_func2 ________________________________

    def test_func2():
>       assert False
E       assert False

test_module.py:12: AssertionError
-------------------------- Captured stdout setup ---------------------------
setting up <function test_func2 at 0xdeadbeef>
======================= 1 failed, 1 passed in 0.12s ========================

Accessing captured output from a test function

The capsys, capsysbinary, capfd, and capfdbinary fixtures allow access to stdout/stderr output created during test execution. Here is an example test function that performs some output related checks:

def test_myoutput(capsys):  # or use "capfd" for fd-level
    print("hello")
    sys.stderr.write("world\n")
    captured = capsys.readouterr()
    assert captured.out == "hello\n"
    assert captured.err == "world\n"
    print("next")
    captured = capsys.readouterr()
    assert captured.out == "next\n"

The readouterr() call snapshots the output so far - and capturing will be continued. After the test function finishes the original streams will be restored. Using capsys this way frees your test from having to care about setting/resetting output streams and also interacts well with pytest’s own per-test capturing.

If you want to capture on filedescriptor level you can use the capfd fixture which offers the exact same interface but allows to also capture output from libraries or subprocesses that directly write to operating system level output streams (FD1 and FD2).

The return value from readouterr changed to a namedtuple with two attributes, out and err.

If the code under test writes non-textual data, you can capture this using the capsysbinary fixture which instead returns bytes from the readouterr method. The capfsysbinary fixture is currently only available in python 3.

If the code under test writes non-textual data, you can capture this using the capfdbinary fixture which instead returns bytes from the readouterr method. The capfdbinary fixture operates on the filedescriptor level.

To temporarily disable capture within a test, both capsys and capfd have a disabled() method that can be used as a context manager, disabling capture inside the with block:

def test_disabling_capturing(capsys):
    print("this output is captured")
    with capsys.disabled():
        print("output not captured, going directly to sys.stdout")
    print("this output is also captured")

Warnings Capture

Starting from version 3.1, pytest now automatically catches warnings during test execution and displays them at the end of the session:

# content of test_show_warnings.py
import warnings


def api_v1():
    warnings.warn(UserWarning("api v1, should use functions from v2"))
    return 1


def test_one():
    assert api_v1() == 1

Running pytest now produces this output:

$ pytest test_show_warnings.py
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 1 item

test_show_warnings.py .                                              [100%]

============================= warnings summary =============================
test_show_warnings.py::test_one
  $REGENDOC_TMPDIR/test_show_warnings.py:5: UserWarning: api v1, should use functions from v2
    warnings.warn(UserWarning("api v1, should use functions from v2"))

-- Docs: https://docs.pytest.org/en/latest/warnings.html
======================= 1 passed, 1 warning in 0.12s =======================

The -W flag can be passed to control which warnings will be displayed or even turn them into errors:

$ pytest -q test_show_warnings.py -W error::UserWarning
F                                                                    [100%]
================================= FAILURES =================================
_________________________________ test_one _________________________________

    def test_one():
>       assert api_v1() == 1

test_show_warnings.py:10:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    def api_v1():
>       warnings.warn(UserWarning("api v1, should use functions from v2"))
E       UserWarning: api v1, should use functions from v2

test_show_warnings.py:5: UserWarning
1 failed in 0.12s

The same option can be set in the pytest.ini file using the filterwarnings ini option. For example, the configuration below will ignore all user warnings, but will transform all other warnings into errors.

[pytest]
filterwarnings =
    error
    ignore::UserWarning

When a warning matches more than one option in the list, the action for the last matching option is performed.

Both -W command-line option and filterwarnings ini option are based on Python’s own -W option and warnings.simplefilter, so please refer to those sections in the Python documentation for other examples and advanced usage.

@pytest.mark.filterwarnings

You can use the @pytest.mark.filterwarnings to add warning filters to specific test items, allowing you to have finer control of which warnings should be captured at test, class or even module level:

import warnings


def api_v1():
    warnings.warn(UserWarning("api v1, should use functions from v2"))
    return 1


@pytest.mark.filterwarnings("ignore:api v1")
def test_one():
    assert api_v1() == 1

Filters applied using a mark take precedence over filters passed on the command line or configured by the filterwarnings ini option.

You may apply a filter to all tests of a class by using the filterwarnings mark as a class decorator or to all tests in a module by setting the pytestmark variable:

# turns all warnings into errors for this module
pytestmark = pytest.mark.filterwarnings("error")

Credits go to Florian Schulze for the reference implementation in the pytest-warnings plugin.

Disabling warnings summary

Although not recommended, you can use the --disable-warnings command-line option to suppress the warning summary entirely from the test run output.

Disabling warning capture entirely

This plugin is enabled by default but can be disabled entirely in your pytest.ini file with:

[pytest]
addopts = -p no:warnings

Or passing -p no:warnings in the command-line. This might be useful if your test suites handles warnings using an external system.

DeprecationWarning and PendingDeprecationWarning

By default pytest will display DeprecationWarning and PendingDeprecationWarning warnings from user code and third-party libraries, as recommended by PEP-0565. This helps users keep their code modern and avoid breakages when deprecated warnings are effectively removed.

Sometimes it is useful to hide some specific deprecation warnings that happen in code that you have no control over (such as third-party libraries), in which case you might use the warning filters options (ini or marks) to ignore those warnings.

For example:

[pytest]
filterwarnings =
    ignore:.*U.*mode is deprecated:DeprecationWarning

This will ignore all warnings of type DeprecationWarning where the start of the message matches the regular expression ".*U.*mode is deprecated".

Note

If warnings are configured at the interpreter level, using the PYTHONWARNINGS environment variable or the -W command-line option, pytest will not configure any filters by default.

Also pytest doesn’t follow PEP-0506 suggestion of resetting all warning filters because it might break test suites that configure warning filters themselves by calling warnings.simplefilter (see issue #2430 for an example of that).

Ensuring code triggers a deprecation warning

You can also use pytest.deprecated_call() for checking that a certain function call triggers a DeprecationWarning or PendingDeprecationWarning:

import pytest


def test_myfunction_deprecated():
    with pytest.deprecated_call():
        myfunction(17)

This test will fail if myfunction does not issue a deprecation warning when called with a 17 argument.

By default, DeprecationWarning and PendingDeprecationWarning will not be caught when using pytest.warns() or recwarn because the default Python warnings filters hide them. If you wish to record them in your own code, use warnings.simplefilter('always'):

import warnings
import pytest


def test_deprecation(recwarn):
    warnings.simplefilter("always")
    myfunction(17)
    assert len(recwarn) == 1
    assert recwarn.pop(DeprecationWarning)

The recwarn fixture automatically ensures to reset the warnings filter at the end of the test, so no global state is leaked.

Asserting warnings with the warns function

You can check that code raises a particular warning using pytest.warns, which works in a similar manner to raises:

import warnings
import pytest


def test_warning():
    with pytest.warns(UserWarning):
        warnings.warn("my warning", UserWarning)

The test will fail if the warning in question is not raised. The keyword argument match to assert that the exception matches a text or regex:

>>> with warns(UserWarning, match='must be 0 or None'):
...     warnings.warn("value must be 0 or None", UserWarning)

>>> with warns(UserWarning, match=r'must be \d+$'):
...     warnings.warn("value must be 42", UserWarning)

>>> with warns(UserWarning, match=r'must be \d+$'):
...     warnings.warn("this is not here", UserWarning)
Traceback (most recent call last):
  ...
Failed: DID NOT WARN. No warnings of type ...UserWarning... was emitted...

You can also call pytest.warns on a function or code string:

pytest.warns(expected_warning, func, *args, **kwargs)
pytest.warns(expected_warning, "func(*args, **kwargs)")

The function also returns a list of all raised warnings (as warnings.WarningMessage objects), which you can query for additional information:

with pytest.warns(RuntimeWarning) as record:
    warnings.warn("another warning", RuntimeWarning)

# check that only one warning was raised
assert len(record) == 1
# check that the message matches
assert record[0].message.args[0] == "another warning"

Alternatively, you can examine raised warnings in detail using the recwarn fixture (see below).

Note

DeprecationWarning and PendingDeprecationWarning are treated differently; see Ensuring code triggers a deprecation warning.

Recording warnings

You can record raised warnings either using pytest.warns or with the recwarn fixture.

To record with pytest.warns without asserting anything about the warnings, pass None as the expected warning type:

with pytest.warns(None) as record:
    warnings.warn("user", UserWarning)
    warnings.warn("runtime", RuntimeWarning)

assert len(record) == 2
assert str(record[0].message) == "user"
assert str(record[1].message) == "runtime"

The recwarn fixture will record warnings for the whole function:

import warnings


def test_hello(recwarn):
    warnings.warn("hello", UserWarning)
    assert len(recwarn) == 1
    w = recwarn.pop(UserWarning)
    assert issubclass(w.category, UserWarning)
    assert str(w.message) == "hello"
    assert w.filename
    assert w.lineno

Both recwarn and pytest.warns return the same interface for recorded warnings: a WarningsRecorder instance. To view the recorded warnings, you can iterate over this instance, call len on it to get the number of recorded warnings, or index into it to get a particular recorded warning.

Full API: WarningsRecorder.

Custom failure messages

Recording warnings provides an opportunity to produce custom test failure messages for when no warnings are issued or other conditions are met.

def test():
    with pytest.warns(Warning) as record:
        f()
        if not record:
            pytest.fail("Expected a warning!")

If no warnings are issued when calling f, then not record will evaluate to True. You can then call pytest.fail with a custom error message.

Internal pytest warnings

pytest may generate its own warnings in some situations, such as improper usage or deprecated features.

For example, pytest will emit a warning if it encounters a class that matches python_classes but also defines an __init__ constructor, as this prevents the class from being instantiated:

# content of test_pytest_warnings.py
class Test:
    def __init__(self):
        pass

    def test_foo(self):
        assert 1 == 1

$ pytest test_pytest_warnings.py -q

============================= warnings summary =============================
test_pytest_warnings.py:1
  $REGENDOC_TMPDIR/test_pytest_warnings.py:1: PytestCollectionWarning: cannot collect test class 'Test' because it has a __init__ constructor (from: test_pytest_warnings.py)
    class Test:

-- Docs: https://docs.pytest.org/en/latest/warnings.html
1 warning in 0.12s

These warnings might be filtered using the same builtin mechanisms used to filter other types of warnings.

Please read our Backwards Compatibility Policy to learn how we proceed about deprecating and eventually removing features.

The following warning types are used by pytest and are part of the public API:

class PytestWarning

Bases: UserWarning.

Base class for all warnings emitted by pytest.

class PytestAssertRewriteWarning

Bases: PytestWarning.

Warning emitted by the pytest assert rewrite module.

class PytestCacheWarning

Bases: PytestWarning.

Warning emitted by the cache plugin in various situations.

class PytestCollectionWarning

Bases: PytestWarning.

Warning emitted when pytest is not able to collect a file or symbol in a module.

class PytestConfigWarning

Bases: PytestWarning.

Warning emitted for configuration issues.

class PytestDeprecationWarning

Bases: pytest.PytestWarning, DeprecationWarning.

Warning class for features that will be removed in a future version.

class PytestExperimentalApiWarning

Bases: pytest.PytestWarning, FutureWarning.

Warning category used to denote experiments in pytest. Use sparingly as the API might change or even be removed completely in future version

class PytestUnhandledCoroutineWarning

Bases: PytestWarning.

Warning emitted when pytest encounters a test function which is a coroutine, but it was not handled by any async-aware plugin. Coroutine test functions are not natively supported.

class PytestUnknownMarkWarning

Bases: PytestWarning.

Warning emitted on use of unknown markers. See https://docs.pytest.org/en/latest/mark.html for details.

Doctest integration for modules and test files

By default all files matching the test*.txt pattern will be run through the python standard doctest module. You can change the pattern by issuing:

pytest --doctest-glob='*.rst'

on the command line. --doctest-glob can be given multiple times in the command-line.

If you then have a text file like this:

# content of test_example.txt

hello this is a doctest
>>> x = 3
>>> x
3

then you can just invoke pytest directly:

$ pytest
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 1 item

test_example.txt .                                                   [100%]

============================ 1 passed in 0.12s =============================

By default, pytest will collect test*.txt files looking for doctest directives, but you can pass additional globs using the --doctest-glob option (multi-allowed).

In addition to text files, you can also execute doctests directly from docstrings of your classes and functions, including from test modules:

# content of mymodule.py
def something():
    """ a doctest in a docstring
    >>> something()
    42
    """
    return 42

$ pytest --doctest-modules
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 2 items

mymodule.py .                                                        [ 50%]
test_example.txt .                                                   [100%]

============================ 2 passed in 0.12s =============================

You can make these changes permanent in your project by putting them into a pytest.ini file like this:

# content of pytest.ini
[pytest]
addopts = --doctest-modules

Note

The builtin pytest doctest supports only doctest blocks, but if you are looking for more advanced checking over all your documentation, including doctests, .. codeblock:: python Sphinx directive support, and any other examples your documentation may include, you may wish to consider Sybil. It provides pytest integration out of the box.

Encoding

The default encoding is UTF-8, but you can specify the encoding that will be used for those doctest files using the doctest_encoding ini option:

# content of pytest.ini
[pytest]
doctest_encoding = latin1

Using ‘doctest’ options

Python’s standard doctest module provides some options to configure the strictness of doctest tests. In pytest, you can enable those flags using the configuration file.

For example, to make pytest ignore trailing whitespaces and ignore lengthy exception stack traces you can just write:

[pytest]
doctest_optionflags= NORMALIZE_WHITESPACE IGNORE_EXCEPTION_DETAIL

Alternatively, options can be enabled by an inline comment in the doc test itself:

>>> something_that_raises()  # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
ValueError: ...

pytest also introduces new options:

• ALLOW_UNICODE: when enabled, the u prefix is stripped from unicode strings in expected doctest output. This allows doctests to run in Python 2 and Python 3 unchanged.

• ALLOW_BYTES: similarly, the b prefix is stripped from byte strings in expected doctest output.

• NUMBER: when enabled, floating-point numbers only need to match as far as the precision you have written in the expected doctest output. For example, the following output would only need to match to 2 decimal places:

  >>> math.pi
  3.14
  

  If you wrote 3.1416 then the actual output would need to match to 4 decimal places; and so on.

  This avoids false positives caused by limited floating-point precision, like this:

  Expected:
      0.233
  Got:
      0.23300000000000001
  

  NUMBER also supports lists of floating-point numbers – in fact, it matches floating-point numbers appearing anywhere in the output, even inside a string! This means that it may not be appropriate to enable globally in doctest_optionflags in your configuration file.

  New in version 5.1.

Continue on failure

By default, pytest would report only the first failure for a given doctest. If you want to continue the test even when you have failures, do:

pytest --doctest-modules --doctest-continue-on-failure

Output format

You can change the diff output format on failure for your doctests by using one of standard doctest modules format in options (see doctest.REPORT_UDIFF, doctest.REPORT_CDIFF, doctest.REPORT_NDIFF, doctest.REPORT_ONLY_FIRST_FAILURE):

pytest --doctest-modules --doctest-report none
pytest --doctest-modules --doctest-report udiff
pytest --doctest-modules --doctest-report cdiff
pytest --doctest-modules --doctest-report ndiff
pytest --doctest-modules --doctest-report only_first_failure

pytest-specific features

Some features are provided to make writing doctests easier or with better integration with your existing test suite. Keep in mind however that by using those features you will make your doctests incompatible with the standard doctests module.

Using fixtures

It is possible to use fixtures using the getfixture helper:

# content of example.rst
>>> tmp = getfixture('tmpdir')
>>> ...
>>>

Also, Using fixtures from classes, modules or projects and Autouse fixtures (xUnit setup on steroids) fixtures are supported when executing text doctest files.

‘doctest_namespace’ fixture

The doctest_namespace fixture can be used to inject items into the namespace in which your doctests run. It is intended to be used within your own fixtures to provide the tests that use them with context.

doctest_namespace is a standard dict object into which you place the objects you want to appear in the doctest namespace:

# content of conftest.py
import numpy


@pytest.fixture(autouse=True)
def add_np(doctest_namespace):
    doctest_namespace["np"] = numpy

which can then be used in your doctests directly:

# content of numpy.py
def arange():
    """
    >>> a = np.arange(10)
    >>> len(a)
    10
    """
    pass

Note that like the normal conftest.py, the fixtures are discovered in the directory tree conftest is in. Meaning that if you put your doctest with your source code, the relevant conftest.py needs to be in the same directory tree. Fixtures will not be discovered in a sibling directory tree!

Skipping tests dynamically

New in version 4.4.

You can use pytest.skip to dynamically skip doctests. For example:

>>> import sys, pytest
>>> if sys.platform.startswith('win'):
...     pytest.skip('this doctest does not work on Windows')
...

Skip and xfail: dealing with tests that cannot succeed

You can mark test functions that cannot be run on certain platforms or that you expect to fail so pytest can deal with them accordingly and present a summary of the test session, while keeping the test suite green.

A skip means that you expect your test to pass only if some conditions are met, otherwise pytest should skip running the test altogether. Common examples are skipping windows-only tests on non-windows platforms, or skipping tests that depend on an external resource which is not available at the moment (for example a database).

A xfail means that you expect a test to fail for some reason. A common example is a test for a feature not yet implemented, or a bug not yet fixed. When a test passes despite being expected to fail (marked with pytest.mark.xfail), it’s an xpass and will be reported in the test summary.

pytest counts and lists skip and xfail tests separately. Detailed information about skipped/xfailed tests is not shown by default to avoid cluttering the output. You can use the -r option to see details corresponding to the “short” letters shown in the test progress:

pytest -rxXs  # show extra info on xfailed, xpassed, and skipped tests

More details on the -r option can be found by running pytest -h.

(See How to change command line options defaults)

Skipping test functions

The simplest way to skip a test function is to mark it with the skip decorator which may be passed an optional reason:

@pytest.mark.skip(reason="no way of currently testing this")
def test_the_unknown():
    ...

Alternatively, it is also possible to skip imperatively during test execution or setup by calling the pytest.skip(reason) function:

def test_function():
    if not valid_config():
        pytest.skip("unsupported configuration")

The imperative method is useful when it is not possible to evaluate the skip condition during import time.

It is also possible to skip the whole module using pytest.skip(reason, allow_module_level=True) at the module level:

import sys
import pytest

if not sys.platform.startswith("win"):
    pytest.skip("skipping windows-only tests", allow_module_level=True)

Reference: pytest.mark.skip

skipif

If you wish to skip something conditionally then you can use skipif instead. Here is an example of marking a test function to be skipped when run on an interpreter earlier than Python3.6:

import sys


@pytest.mark.skipif(sys.version_info < (3, 6), reason="requires python3.6 or higher")
def test_function():
    ...

If the condition evaluates to True during collection, the test function will be skipped, with the specified reason appearing in the summary when using -rs.

You can share skipif markers between modules. Consider this test module:

# content of test_mymodule.py
import mymodule

minversion = pytest.mark.skipif(
    mymodule.__versioninfo__ < (1, 1), reason="at least mymodule-1.1 required"
)


@minversion
def test_function():
    ...

You can import the marker and reuse it in another test module:

# test_myothermodule.py
from test_mymodule import minversion


@minversion
def test_anotherfunction():
    ...

For larger test suites it’s usually a good idea to have one file where you define the markers which you then consistently apply throughout your test suite.

Alternatively, you can use condition strings instead of booleans, but they can’t be shared between modules easily so they are supported mainly for backward compatibility reasons.

Reference: pytest.mark.skipif

Skip all test functions of a class or module

You can use the skipif marker (as any other marker) on classes:

@pytest.mark.skipif(sys.platform == "win32", reason="does not run on windows")
class TestPosixCalls:
    def test_function(self):
        "will not be setup or run under 'win32' platform"

If the condition is True, this marker will produce a skip result for each of the test methods of that class.

If you want to skip all test functions of a module, you may use the pytestmark name on the global level:

# test_module.py
pytestmark = pytest.mark.skipif(...)

If multiple skipif decorators are applied to a test function, it will be skipped if any of the skip conditions is true.

Skipping files or directories

Sometimes you may need to skip an entire file or directory, for example if the tests rely on Python version-specific features or contain code that you do not wish pytest to run. In this case, you must exclude the files and directories from collection. Refer to Customizing test collection for more information.

Skipping on a missing import dependency

You can skip tests on a missing import by using pytest.importorskip at module level, within a test, or test setup function.

docutils = pytest.importorskip("docutils")

If docutils cannot be imported here, this will lead to a skip outcome of the test. You can also skip based on the version number of a library:

docutils = pytest.importorskip("docutils", minversion="0.3")

The version will be read from the specified module’s __version__ attribute.

Summary

Here’s a quick guide on how to skip tests in a module in different situations:

1. Skip all tests in a module unconditionally:

pytestmark = pytest.mark.skip("all tests still WIP")

2. Skip all tests in a module based on some condition:

pytestmark = pytest.mark.skipif(sys.platform == "win32", reason="tests for linux only")

3. Skip all tests in a module if some import is missing:

pexpect = pytest.importorskip("pexpect")

XFail: mark test functions as expected to fail

You can use the xfail marker to indicate that you expect a test to fail:

@pytest.mark.xfail
def test_function():
    ...

This test will be run but no traceback will be reported when it fails. Instead terminal reporting will list it in the “expected to fail” (XFAIL) or “unexpectedly passing” (XPASS) sections.

Alternatively, you can also mark a test as XFAIL from within a test or setup function imperatively:

def test_function():
    if not valid_config():
        pytest.xfail("failing configuration (but should work)")

This will unconditionally make test_function XFAIL. Note that no other code is executed after pytest.xfail call, differently from the marker. That’s because it is implemented internally by raising a known exception.

Reference: pytest.mark.xfail

strict parameter

Both XFAIL and XPASS don’t fail the test suite, unless the strict keyword-only parameter is passed as True:

@pytest.mark.xfail(strict=True)
def test_function():
    ...

This will make XPASS (“unexpectedly passing”) results from this test to fail the test suite.

You can change the default value of the strict parameter using the xfail_strict ini option:

[pytest]
xfail_strict=true

reason parameter

As with skipif you can also mark your expectation of a failure on a particular platform:

@pytest.mark.xfail(sys.version_info >= (3, 6), reason="python3.6 api changes")
def test_function():
    ...

raises parameter

If you want to be more specific as to why the test is failing, you can specify a single exception, or a tuple of exceptions, in the raises argument.

@pytest.mark.xfail(raises=RuntimeError)
def test_function():
    ...

Then the test will be reported as a regular failure if it fails with an exception not mentioned in raises.

run parameter

If a test should be marked as xfail and reported as such but should not be even executed, use the run parameter as False:

@pytest.mark.xfail(run=False)
def test_function():
    ...

This is specially useful for xfailing tests that are crashing the interpreter and should be investigated later.

Ignoring xfail

By specifying on the commandline:

pytest --runxfail

you can force the running and reporting of an xfail marked test as if it weren’t marked at all. This also causes pytest.xfail to produce no effect.

Examples

Here is a simple test file with the several usages:

import pytest

xfail = pytest.mark.xfail


@xfail
def test_hello():
    assert 0


@xfail(run=False)
def test_hello2():
    assert 0


@xfail("hasattr(os, 'sep')")
def test_hello3():
    assert 0


@xfail(reason="bug 110")
def test_hello4():
    assert 0


@xfail('pytest.__version__[0] != "17"')
def test_hello5():
    assert 0


def test_hello6():
    pytest.xfail("reason")


@xfail(raises=IndexError)
def test_hello7():
    x = []
    x[1] = 1

Running it with the report-on-xfail option gives this output:

example $ pytest -rx xfail_demo.py
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR/example
collected 7 items

xfail_demo.py xxxxxxx                                                [100%]

========================= short test summary info ==========================
XFAIL xfail_demo.py::test_hello
XFAIL xfail_demo.py::test_hello2
  reason: [NOTRUN]
XFAIL xfail_demo.py::test_hello3
  condition: hasattr(os, 'sep')
XFAIL xfail_demo.py::test_hello4
  bug 110
XFAIL xfail_demo.py::test_hello5
  condition: pytest.__version__[0] != "17"
XFAIL xfail_demo.py::test_hello6
  reason: reason
XFAIL xfail_demo.py::test_hello7
============================ 7 xfailed in 0.12s ============================

Skip/xfail with parametrize

It is possible to apply markers like skip and xfail to individual test instances when using parametrize:

import pytest


@pytest.mark.parametrize(
    ("n", "expected"),
    [
        (1, 2),
        pytest.param(1, 0, marks=pytest.mark.xfail),
        pytest.param(1, 3, marks=pytest.mark.xfail(reason="some bug")),
        (2, 3),
        (3, 4),
        (4, 5),
        pytest.param(
            10, 11, marks=pytest.mark.skipif(sys.version_info >= (3, 0), reason="py2k")
        ),
    ],
)
def test_increment(n, expected):
    assert n + 1 == expected

Parametrizing fixtures and test functions

pytest enables test parametrization at several levels:

• pytest.fixture() allows one to parametrize fixture functions.

• @pytest.mark.parametrize allows one to define multiple sets of arguments and fixtures at the test function or class.
• pytest_generate_tests allows one to define custom parametrization schemes or extensions.

@pytest.mark.parametrize: parametrizing test functions

The builtin pytest.mark.parametrize decorator enables parametrization of arguments for a test function. Here is a typical example of a test function that implements checking that a certain input leads to an expected output:

# content of test_expectation.py
import pytest


@pytest.mark.parametrize("test_input,expected", [("3+5", 8), ("2+4", 6), ("6*9", 42)])
def test_eval(test_input, expected):
    assert eval(test_input) == expected

Here, the @parametrize decorator defines three different (test_input,expected) tuples so that the test_eval function will run three times using them in turn:

$ pytest
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 3 items

test_expectation.py ..F                                              [100%]

================================= FAILURES =================================
____________________________ test_eval[6*9-42] _____________________________

test_input = '6*9', expected = 42

    @pytest.mark.parametrize("test_input,expected", [("3+5", 8), ("2+4", 6), ("6*9", 42)])
    def test_eval(test_input, expected):
>       assert eval(test_input) == expected
E       AssertionError: assert 54 == 42
E        +  where 54 = eval('6*9')

test_expectation.py:6: AssertionError
======================= 1 failed, 2 passed in 0.12s ========================

Note

pytest by default escapes any non-ascii characters used in unicode strings for the parametrization because it has several downsides. If however you would like to use unicode strings in parametrization and see them in the terminal as is (non-escaped), use this option in your pytest.ini:

[pytest]
disable_test_id_escaping_and_forfeit_all_rights_to_community_support = True

Keep in mind however that this might cause unwanted side effects and even bugs depending on the OS used and plugins currently installed, so use it at your own risk.

As designed in this example, only one pair of input/output values fails the simple test function. And as usual with test function arguments, you can see the input and output values in the traceback.

Note that you could also use the parametrize marker on a class or a module (see Marking test functions with attributes) which would invoke several functions with the argument sets.

It is also possible to mark individual test instances within parametrize, for example with the builtin mark.xfail:

# content of test_expectation.py
import pytest


@pytest.mark.parametrize(
    "test_input,expected",
    [("3+5", 8), ("2+4", 6), pytest.param("6*9", 42, marks=pytest.mark.xfail)],
)
def test_eval(test_input, expected):
    assert eval(test_input) == expected

Let’s run this:

$ pytest
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 3 items

test_expectation.py ..x                                              [100%]

======================= 2 passed, 1 xfailed in 0.12s =======================

The one parameter set which caused a failure previously now shows up as an “xfailed (expected to fail)” test.

In case the values provided to parametrize result in an empty list - for example, if they’re dynamically generated by some function - the behaviour of pytest is defined by the empty_parameter_set_mark option.

To get all combinations of multiple parametrized arguments you can stack parametrize decorators:

import pytest


@pytest.mark.parametrize("x", [0, 1])
@pytest.mark.parametrize("y", [2, 3])
def test_foo(x, y):
    pass

This will run the test with the arguments set to x=0/y=2, x=1/y=2, x=0/y=3, and x=1/y=3 exhausting parameters in the order of the decorators.

Basic pytest_generate_tests example

Sometimes you may want to implement your own parametrization scheme or implement some dynamism for determining the parameters or scope of a fixture. For this, you can use the pytest_generate_tests hook which is called when collecting a test function. Through the passed in metafunc object you can inspect the requesting test context and, most importantly, you can call metafunc.parametrize() to cause parametrization.

For example, let’s say we want to run a test taking string inputs which we want to set via a new pytest command line option. Let’s first write a simple test accepting a stringinput fixture function argument:

# content of test_strings.py


def test_valid_string(stringinput):
    assert stringinput.isalpha()

Now we add a conftest.py file containing the addition of a command line option and the parametrization of our test function:

# content of conftest.py


def pytest_addoption(parser):
    parser.addoption(
        "--stringinput",
        action="append",
        default=[],
        help="list of stringinputs to pass to test functions",
    )


def pytest_generate_tests(metafunc):
    if "stringinput" in metafunc.fixturenames:
        metafunc.parametrize("stringinput", metafunc.config.getoption("stringinput"))

If we now pass two stringinput values, our test will run twice:

$ pytest -q --stringinput="hello" --stringinput="world" test_strings.py
..                                                                   [100%]
2 passed in 0.12s

Let’s also run with a stringinput that will lead to a failing test:

$ pytest -q --stringinput="!" test_strings.py
F                                                                    [100%]
================================= FAILURES =================================
___________________________ test_valid_string[!] ___________________________

stringinput = '!'

    def test_valid_string(stringinput):
>       assert stringinput.isalpha()
E       AssertionError: assert False
E        +  where False = <built-in method isalpha of str object at 0xdeadbeef>()
E        +    where <built-in method isalpha of str object at 0xdeadbeef> = '!'.isalpha

test_strings.py:4: AssertionError
1 failed in 0.12s

As expected our test function fails.

If you don’t specify a stringinput it will be skipped because metafunc.parametrize() will be called with an empty parameter list:

$ pytest -q -rs test_strings.py
s                                                                    [100%]
========================= short test summary info ==========================
SKIPPED [1] test_strings.py: got empty parameter set ['stringinput'], function test_valid_string at $REGENDOC_TMPDIR/test_strings.py:2
1 skipped in 0.12s

Note that when calling metafunc.parametrize multiple times with different parameter sets, all parameter names across those sets cannot be duplicated, otherwise an error will be raised.

More examples

For further examples, you might want to look at more parametrization examples.

Cache: working with cross-testrun state

Usage

The plugin provides two command line options to rerun failures from the last pytest invocation:

• --lf, --last-failed - to only re-run the failures.
• --ff, --failed-first - to run the failures first and then the rest of the tests.

For cleanup (usually not needed), a --cache-clear option allows to remove all cross-session cache contents ahead of a test run.

Other plugins may access the config.cache object to set/get json encodable values between pytest invocations.

Note

This plugin is enabled by default, but can be disabled if needed: see Deactivating / unregistering a plugin by name (the internal name for this plugin is cacheprovider).

Rerunning only failures or failures first

First, let’s create 50 test invocation of which only 2 fail:

# content of test_50.py
import pytest


@pytest.mark.parametrize("i", range(50))
def test_num(i):
    if i in (17, 25):
        pytest.fail("bad luck")

If you run this for the first time you will see two failures:

$ pytest -q
.................F.......F........................                   [100%]
================================= FAILURES =================================
_______________________________ test_num[17] _______________________________

i = 17

    @pytest.mark.parametrize("i", range(50))
    def test_num(i):
        if i in (17, 25):
>           pytest.fail("bad luck")
E           Failed: bad luck

test_50.py:7: Failed
_______________________________ test_num[25] _______________________________

i = 25

    @pytest.mark.parametrize("i", range(50))
    def test_num(i):
        if i in (17, 25):
>           pytest.fail("bad luck")
E           Failed: bad luck

test_50.py:7: Failed
2 failed, 48 passed in 0.12s

If you then run it with --lf:

$ pytest --lf
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 50 items / 48 deselected / 2 selected
run-last-failure: rerun previous 2 failures

test_50.py FF                                                        [100%]

================================= FAILURES =================================
_______________________________ test_num[17] _______________________________

i = 17

    @pytest.mark.parametrize("i", range(50))
    def test_num(i):
        if i in (17, 25):
>           pytest.fail("bad luck")
E           Failed: bad luck

test_50.py:7: Failed
_______________________________ test_num[25] _______________________________

i = 25

    @pytest.mark.parametrize("i", range(50))
    def test_num(i):
        if i in (17, 25):
>           pytest.fail("bad luck")
E           Failed: bad luck

test_50.py:7: Failed
===================== 2 failed, 48 deselected in 0.12s =====================

You have run only the two failing tests from the last run, while the 48 passing tests have not been run (“deselected”).

Now, if you run with the --ff option, all tests will be run but the first previous failures will be executed first (as can be seen from the series of FF and dots):

$ pytest --ff
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 50 items
run-last-failure: rerun previous 2 failures first

test_50.py FF................................................        [100%]

================================= FAILURES =================================
_______________________________ test_num[17] _______________________________

i = 17

    @pytest.mark.parametrize("i", range(50))
    def test_num(i):
        if i in (17, 25):
>           pytest.fail("bad luck")
E           Failed: bad luck

test_50.py:7: Failed
_______________________________ test_num[25] _______________________________

i = 25

    @pytest.mark.parametrize("i", range(50))
    def test_num(i):
        if i in (17, 25):
>           pytest.fail("bad luck")
E           Failed: bad luck

test_50.py:7: Failed
======================= 2 failed, 48 passed in 0.12s =======================

New --nf, --new-first options: run new tests first followed by the rest of the tests, in both cases tests are also sorted by the file modified time, with more recent files coming first.

Behavior when no tests failed in the last run

When no tests failed in the last run, or when no cached lastfailed data was found, pytest can be configured either to run all of the tests or no tests, using the --last-failed-no-failures option, which takes one of the following values:

pytest --last-failed --last-failed-no-failures all    # run all tests (default behavior)
pytest --last-failed --last-failed-no-failures none   # run no tests and exit

The new config.cache object

Plugins or conftest.py support code can get a cached value using the pytest config object. Here is a basic example plugin which implements a pytest fixtures: explicit, modular, scalable which re-uses previously created state across pytest invocations:

# content of test_caching.py
import pytest
import time


def expensive_computation():
    print("running expensive computation...")


@pytest.fixture
def mydata(request):
    val = request.config.cache.get("example/value", None)
    if val is None:
        expensive_computation()
        val = 42
        request.config.cache.set("example/value", val)
    return val


def test_function(mydata):
    assert mydata == 23

If you run this command for the first time, you can see the print statement:

$ pytest -q
F                                                                    [100%]
================================= FAILURES =================================
______________________________ test_function _______________________________

mydata = 42

    def test_function(mydata):
>       assert mydata == 23
E       assert 42 == 23

test_caching.py:20: AssertionError
-------------------------- Captured stdout setup ---------------------------
running expensive computation...
1 failed in 0.12s

If you run it a second time, the value will be retrieved from the cache and nothing will be printed:

$ pytest -q
F                                                                    [100%]
================================= FAILURES =================================
______________________________ test_function _______________________________

mydata = 42

    def test_function(mydata):
>       assert mydata == 23
E       assert 42 == 23

test_caching.py:20: AssertionError
1 failed in 0.12s

See the config.cache for more details.

Inspecting Cache content

You can always peek at the content of the cache using the --cache-show command line option:

$ pytest --cache-show
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
cachedir: $PYTHON_PREFIX/.pytest_cache
--------------------------- cache values for '*' ---------------------------
cache/lastfailed contains:
  {'test_50.py::test_num[17]': True,
   'test_50.py::test_num[25]': True,
   'test_assert1.py::test_function': True,
   'test_assert2.py::test_set_comparison': True,
   'test_caching.py::test_function': True,
   'test_foocompare.py::test_compare': True}
cache/nodeids contains:
  ['test_assert1.py::test_function',
   'test_assert2.py::test_set_comparison',
   'test_foocompare.py::test_compare',
   'test_50.py::test_num[0]',
   'test_50.py::test_num[1]',
   'test_50.py::test_num[2]',
   'test_50.py::test_num[3]',
   'test_50.py::test_num[4]',
   'test_50.py::test_num[5]',
   'test_50.py::test_num[6]',
   'test_50.py::test_num[7]',
   'test_50.py::test_num[8]',
   'test_50.py::test_num[9]',
   'test_50.py::test_num[10]',
   'test_50.py::test_num[11]',
   'test_50.py::test_num[12]',
   'test_50.py::test_num[13]',
   'test_50.py::test_num[14]',
   'test_50.py::test_num[15]',
   'test_50.py::test_num[16]',
   'test_50.py::test_num[17]',
   'test_50.py::test_num[18]',
   'test_50.py::test_num[19]',
   'test_50.py::test_num[20]',
   'test_50.py::test_num[21]',
   'test_50.py::test_num[22]',
   'test_50.py::test_num[23]',
   'test_50.py::test_num[24]',
   'test_50.py::test_num[25]',
   'test_50.py::test_num[26]',
   'test_50.py::test_num[27]',
   'test_50.py::test_num[28]',
   'test_50.py::test_num[29]',
   'test_50.py::test_num[30]',
   'test_50.py::test_num[31]',
   'test_50.py::test_num[32]',
   'test_50.py::test_num[33]',
   'test_50.py::test_num[34]',
   'test_50.py::test_num[35]',
   'test_50.py::test_num[36]',
   'test_50.py::test_num[37]',
   'test_50.py::test_num[38]',
   'test_50.py::test_num[39]',
   'test_50.py::test_num[40]',
   'test_50.py::test_num[41]',
   'test_50.py::test_num[42]',
   'test_50.py::test_num[43]',
   'test_50.py::test_num[44]',
   'test_50.py::test_num[45]',
   'test_50.py::test_num[46]',
   'test_50.py::test_num[47]',
   'test_50.py::test_num[48]',
   'test_50.py::test_num[49]',
   'test_caching.py::test_function']
cache/stepwise contains:
  []
example/value contains:
  42

========================== no tests ran in 0.12s ===========================

--cache-show takes an optional argument to specify a glob pattern for filtering:

$ pytest --cache-show example/*
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
cachedir: $PYTHON_PREFIX/.pytest_cache
----------------------- cache values for 'example/*' -----------------------
example/value contains:
  42

========================== no tests ran in 0.12s ===========================

Clearing Cache content

You can instruct pytest to clear all cache files and values by adding the --cache-clear option like this:

pytest --cache-clear

This is recommended for invocations from Continuous Integration servers where isolation and correctness is more important than speed.

Stepwise

As an alternative to --lf -x, especially for cases where you expect a large part of the test suite will fail, --sw, --stepwise allows you to fix them one at a time. The test suite will run until the first failure and then stop. At the next invocation, tests will continue from the last failing test and then run until the next failing test. You may use the --stepwise-skip option to ignore one failing test and stop the test execution on the second failing test instead. This is useful if you get stuck on a failing test and just want to ignore it until later.

unittest.TestCase Support

pytest supports running Python unittest-based tests out of the box. It’s meant for leveraging existing unittest-based test suites to use pytest as a test runner and also allow to incrementally adapt the test suite to take full advantage of pytest’s features.

To run an existing unittest-style test suite using pytest, type:

pytest tests

pytest will automatically collect unittest.TestCase subclasses and their test methods in test_*.py or *_test.py files.

Almost all unittest features are supported:

• @unittest.skip style decorators;
• setUp/tearDown;
• setUpClass/tearDownClass;
• setUpModule/tearDownModule;

Up to this point pytest does not have support for the following features:

• load_tests protocol;
• subtests;

Benefits out of the box

By running your test suite with pytest you can make use of several features, in most cases without having to modify existing code:

• Obtain more informative tracebacks;
• stdout and stderr capturing;
• Test selection options using -k and -m flags;
• Stopping after the first (or N) failures;
• –pdb command-line option for debugging on test failures (see note below);
• Distribute tests to multiple CPUs using the pytest-xdist plugin;
• Use plain assert-statements instead of self.assert* functions (unittest2pytest is immensely helpful in this);

pytest features in unittest.TestCase subclasses

The following pytest features work in unittest.TestCase subclasses:

• Marks: skip, skipif, xfail;
• Auto-use fixtures;

The following pytest features do not work, and probably never will due to different design philosophies:

• Fixtures (except for autouse fixtures, see below);
• Parametrization;
• Custom hooks;

Third party plugins may or may not work well, depending on the plugin and the test suite.

Mixing pytest fixtures into unittest.TestCase subclasses using marks

Running your unittest with pytest allows you to use its fixture mechanism with unittest.TestCase style tests. Assuming you have at least skimmed the pytest fixture features, let’s jump-start into an example that integrates a pytest db_class fixture, setting up a class-cached database object, and then reference it from a unittest-style test:

# content of conftest.py

# we define a fixture function below and it will be "used" by
# referencing its name from tests

import pytest


@pytest.fixture(scope="class")
def db_class(request):
    class DummyDB:
        pass

    # set a class attribute on the invoking test context
    request.cls.db = DummyDB()

This defines a fixture function db_class which - if used - is called once for each test class and which sets the class-level db attribute to a DummyDB instance. The fixture function achieves this by receiving a special request object which gives access to the requesting test context such as the cls attribute, denoting the class from which the fixture is used. This architecture de-couples fixture writing from actual test code and allows re-use of the fixture by a minimal reference, the fixture name. So let’s write an actual unittest.TestCase class using our fixture definition:

# content of test_unittest_db.py

import unittest
import pytest


@pytest.mark.usefixtures("db_class")
class MyTest(unittest.TestCase):
    def test_method1(self):
        assert hasattr(self, "db")
        assert 0, self.db  # fail for demo purposes

    def test_method2(self):
        assert 0, self.db  # fail for demo purposes

The @pytest.mark.usefixtures("db_class") class-decorator makes sure that the pytest fixture function db_class is called once per class. Due to the deliberately failing assert statements, we can take a look at the self.db values in the traceback:

$ pytest test_unittest_db.py
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 2 items

test_unittest_db.py FF                                               [100%]

================================= FAILURES =================================
___________________________ MyTest.test_method1 ____________________________

self = <test_unittest_db.MyTest testMethod=test_method1>

    def test_method1(self):
        assert hasattr(self, "db")
>       assert 0, self.db  # fail for demo purposes
E       AssertionError: <conftest.db_class.<locals>.DummyDB object at 0xdeadbeef>
E       assert 0

test_unittest_db.py:10: AssertionError
___________________________ MyTest.test_method2 ____________________________

self = <test_unittest_db.MyTest testMethod=test_method2>

    def test_method2(self):
>       assert 0, self.db  # fail for demo purposes
E       AssertionError: <conftest.db_class.<locals>.DummyDB object at 0xdeadbeef>
E       assert 0

test_unittest_db.py:13: AssertionError
============================ 2 failed in 0.12s =============================

This default pytest traceback shows that the two test methods share the same self.db instance which was our intention when writing the class-scoped fixture function above.

Using autouse fixtures and accessing other fixtures

Although it’s usually better to explicitly declare use of fixtures you need for a given test, you may sometimes want to have fixtures that are automatically used in a given context. After all, the traditional style of unittest-setup mandates the use of this implicit fixture writing and chances are, you are used to it or like it.

You can flag fixture functions with @pytest.fixture(autouse=True) and define the fixture function in the context where you want it used. Let’s look at an initdir fixture which makes all test methods of a TestCase class execute in a temporary directory with a pre-initialized samplefile.ini. Our initdir fixture itself uses the pytest builtin tmpdir fixture to delegate the creation of a per-test temporary directory:

# content of test_unittest_cleandir.py
import pytest
import unittest


class MyTest(unittest.TestCase):
    @pytest.fixture(autouse=True)
    def initdir(self, tmpdir):
        tmpdir.chdir()  # change to pytest-provided temporary directory
        tmpdir.join("samplefile.ini").write("# testdata")

    def test_method(self):
        with open("samplefile.ini") as f:
            s = f.read()
        assert "testdata" in s

Due to the autouse flag the initdir fixture function will be used for all methods of the class where it is defined. This is a shortcut for using a @pytest.mark.usefixtures("initdir") marker on the class like in the previous example.

Running this test module …:

$ pytest -q test_unittest_cleandir.py
.                                                                    [100%]
1 passed in 0.12s

… gives us one passed test because the initdir fixture function was executed ahead of the test_method.

Note

unittest.TestCase methods cannot directly receive fixture arguments as implementing that is likely to inflict on the ability to run general unittest.TestCase test suites.

The above usefixtures and autouse examples should help to mix in pytest fixtures into unittest suites.

You can also gradually move away from subclassing from unittest.TestCase to plain asserts and then start to benefit from the full pytest feature set step by step.

Note

Running tests from unittest.TestCase subclasses with --pdb will disable tearDown and cleanup methods for the case that an Exception occurs. This allows proper post mortem debugging for all applications which have significant logic in their tearDown machinery. However, supporting this feature has the following side effect: If people overwrite unittest.TestCase __call__ or run, they need to to overwrite debug in the same way (this is also true for standard unittest).

Note

Due to architectural differences between the two frameworks, setup and teardown for unittest-based tests is performed during the call phase of testing instead of in pytest’s standard setup and teardown stages. This can be important to understand in some situations, particularly when reasoning about errors. For example, if a unittest-based suite exhibits errors during setup, pytest will report no errors during its setup phase and will instead raise the error during call.

Running tests written for nose

pytest has basic support for running tests written for nose.

Usage

After Install pytest type:

python setup.py develop  # make sure tests can import our package
pytest  # instead of 'nosetests'

and you should be able to run your nose style tests and make use of pytest’s capabilities.

Supported nose Idioms

• setup and teardown at module/class/method level
• SkipTest exceptions and markers
• setup/teardown decorators
• yield-based tests and their setup (considered deprecated as of pytest 3.0)
• __test__ attribute on modules/classes/functions
• general usage of nose utilities

Unsupported idioms / known issues

• unittest-style setUp, tearDown, setUpClass, tearDownClass are recognized only on unittest.TestCase classes but not on plain classes. nose supports these methods also on plain classes but pytest deliberately does not. As nose and pytest already both support setup_class, teardown_class, setup_method, teardown_method it doesn’t seem useful to duplicate the unittest-API like nose does. If you however rather think pytest should support the unittest-spelling on plain classes please post to this issue.

• nose imports test modules with the same import path (e.g. tests.test_mode) but different file system paths (e.g. tests/test_mode.py and other/tests/test_mode.py) by extending sys.path/import semantics. pytest does not do that but there is discussion in #268 for adding some support. Note that nose2 choose to avoid this sys.path/import hackery.

  If you place a conftest.py file in the root directory of your project (as determined by pytest) pytest will run tests “nose style” against the code below that directory by adding it to your sys.path instead of running against your installed code.

  You may find yourself wanting to do this if you ran python setup.py install to set up your project, as opposed to python setup.py develop or any of the package manager equivalents. Installing with develop in a virtual environment like tox is recommended over this pattern.

• nose-style doctests are not collected and executed correctly, also doctest fixtures don’t work.

• no nose-configuration is recognized.

• yield-based methods don’t support setup properly because the setup method is always called in the same class instance. There are no plans to fix this currently because yield-tests are deprecated in pytest 3.0, with pytest.mark.parametrize being the recommended alternative.

classic xunit-style setup

This section describes a classic and popular way how you can implement fixtures (setup and teardown test state) on a per-module/class/function basis.

Note

While these setup/teardown methods are simple and familiar to those coming from a unittest or nose background, you may also consider using pytest’s more powerful fixture mechanism which leverages the concept of dependency injection, allowing for a more modular and more scalable approach for managing test state, especially for larger projects and for functional testing. You can mix both fixture mechanisms in the same file but test methods of unittest.TestCase subclasses cannot receive fixture arguments.

Module level setup/teardown

If you have multiple test functions and test classes in a single module you can optionally implement the following fixture methods which will usually be called once for all the functions:

def setup_module(module):
    """ setup any state specific to the execution of the given module."""


def teardown_module(module):
    """ teardown any state that was previously setup with a setup_module
    method.
    """

As of pytest-3.0, the module parameter is optional.

Class level setup/teardown

Similarly, the following methods are called at class level before and after all test methods of the class are called:

@classmethod
def setup_class(cls):
    """ setup any state specific to the execution of the given class (which
    usually contains tests).
    """


@classmethod
def teardown_class(cls):
    """ teardown any state that was previously setup with a call to
    setup_class.
    """

Method and function level setup/teardown

Similarly, the following methods are called around each method invocation:

def setup_method(self, method):
    """ setup any state tied to the execution of the given method in a
    class.  setup_method is invoked for every test method of a class.
    """


def teardown_method(self, method):
    """ teardown any state that was previously setup with a setup_method
    call.
    """

As of pytest-3.0, the method parameter is optional.

If you would rather define test functions directly at module level you can also use the following functions to implement fixtures:

def setup_function(function):
    """ setup any state tied to the execution of the given function.
    Invoked for every test function in the module.
    """


def teardown_function(function):
    """ teardown any state that was previously setup with a setup_function
    call.
    """

As of pytest-3.0, the function parameter is optional.

Remarks:

• It is possible for setup/teardown pairs to be invoked multiple times per testing process.

• teardown functions are not called if the corresponding setup function existed and failed/was skipped.

• Prior to pytest-4.2, xunit-style functions did not obey the scope rules of fixtures, so it was possible, for example, for a setup_method to be called before a session-scoped autouse fixture.

  Now the xunit-style functions are integrated with the fixture mechanism and obey the proper scope rules of fixtures involved in the call.

Installing and Using plugins

This section talks about installing and using third party plugins. For writing your own plugins, please refer to Writing plugins.

Installing a third party plugin can be easily done with pip:

pip install pytest-NAME
pip uninstall pytest-NAME

If a plugin is installed, pytest automatically finds and integrates it, there is no need to activate it.

Here is a little annotated list for some popular plugins:

• pytest-django: write tests for django apps, using pytest integration.
• pytest-twisted: write tests for twisted apps, starting a reactor and processing deferreds from test functions.
• pytest-cov: coverage reporting, compatible with distributed testing
• pytest-xdist: to distribute tests to CPUs and remote hosts, to run in boxed mode which allows to survive segmentation faults, to run in looponfailing mode, automatically re-running failing tests on file changes.
• pytest-instafail: to report failures while the test run is happening.
• pytest-bdd and pytest-konira to write tests using behaviour-driven testing.
• pytest-timeout: to timeout tests based on function marks or global definitions.
• pytest-pep8: a --pep8 option to enable PEP8 compliance checking.
• pytest-flakes: check source code with pyflakes.
• oejskit: a plugin to run javascript unittests in live browsers.

To see a complete list of all plugins with their latest testing status against different pytest and Python versions, please visit plugincompat.

You may also discover more plugins through a pytest- pypi.org search.

Requiring/Loading plugins in a test module or conftest file

You can require plugins in a test module or a conftest file like this:

pytest_plugins = ("myapp.testsupport.myplugin",)

When the test module or conftest plugin is loaded the specified plugins will be loaded as well.

Note

Requiring plugins using a pytest_plugins variable in non-root conftest.py files is deprecated. See full explanation in the Writing plugins section.

Note

The name pytest_plugins is reserved and should not be used as a name for a custom plugin module.

Finding out which plugins are active

If you want to find out which plugins are active in your environment you can type:

pytest --trace-config

and will get an extended test header which shows activated plugins and their names. It will also print local plugins aka conftest.py files when they are loaded.

Deactivating / unregistering a plugin by name

You can prevent plugins from loading or unregister them:

pytest -p no:NAME

This means that any subsequent try to activate/load the named plugin will not work.

If you want to unconditionally disable a plugin for a project, you can add this option to your pytest.ini file:

[pytest]
addopts = -p no:NAME

Alternatively to disable it only in certain environments (for example in a CI server), you can set PYTEST_ADDOPTS environment variable to -p no:name.

See Finding out which plugins are active for how to obtain the name of a plugin.

Writing plugins

It is easy to implement local conftest plugins for your own project or pip-installable plugins that can be used throughout many projects, including third party projects. Please refer to Installing and Using plugins if you only want to use but not write plugins.

A plugin contains one or multiple hook functions. Writing hooks explains the basics and details of how you can write a hook function yourself. pytest implements all aspects of configuration, collection, running and reporting by calling well specified hooks of the following plugins:

• builtin plugins: loaded from pytest’s internal _pytest directory.
• external plugins: modules discovered through setuptools entry points
• conftest.py plugins: modules auto-discovered in test directories

In principle, each hook call is a 1:N Python function call where N is the number of registered implementation functions for a given specification. All specifications and implementations follow the pytest_ prefix naming convention, making them easy to distinguish and find.

Plugin discovery order at tool startup

pytest loads plugin modules at tool startup in the following way:

• by loading all builtin plugins

• by loading all plugins registered through setuptools entry points.

• by pre-scanning the command line for the -p name option and loading the specified plugin before actual command line parsing.

• by loading all conftest.py files as inferred by the command line invocation:

  • if no test paths are specified use current dir as a test path
  • if exists, load conftest.py and test*/conftest.py relative to the directory part of the first test path.

  Note that pytest does not find conftest.py files in deeper nested sub directories at tool startup. It is usually a good idea to keep your conftest.py file in the top level test or project root directory.

• by recursively loading all plugins specified by the pytest_plugins variable in conftest.py files

conftest.py: local per-directory plugins

Local conftest.py plugins contain directory-specific hook implementations. Hook Session and test running activities will invoke all hooks defined in conftest.py files closer to the root of the filesystem. Example of implementing the pytest_runtest_setup hook so that is called for tests in the a sub directory but not for other directories:

a/conftest.py:
    def pytest_runtest_setup(item):
        # called for running each test in 'a' directory
        print("setting up", item)

a/test_sub.py:
    def test_sub():
        pass

test_flat.py:
    def test_flat():
        pass

Here is how you might run it:

pytest test_flat.py --capture=no  # will not show "setting up"
pytest a/test_sub.py --capture=no  # will show "setting up"

Note

If you have conftest.py files which do not reside in a python package directory (i.e. one containing an __init__.py) then “import conftest” can be ambiguous because there might be other conftest.py files as well on your PYTHONPATH or sys.path. It is thus good practice for projects to either put conftest.py under a package scope or to never import anything from a conftest.py file.

See also: pytest import mechanisms and sys.path/PYTHONPATH.

Writing your own plugin

If you want to write a plugin, there are many real-life examples you can copy from:

• a custom collection example plugin: A basic example for specifying tests in Yaml files
• builtin plugins which provide pytest’s own functionality
• many external plugins providing additional features

All of these plugins implement hooks and/or fixtures to extend and add functionality.

Note

Make sure to check out the excellent cookiecutter-pytest-plugin project, which is a cookiecutter template for authoring plugins.

The template provides an excellent starting point with a working plugin, tests running with tox, a comprehensive README file as well as a pre-configured entry-point.

Also consider contributing your plugin to pytest-dev once it has some happy users other than yourself.

Making your plugin installable by others

If you want to make your plugin externally available, you may define a so-called entry point for your distribution so that pytest finds your plugin module. Entry points are a feature that is provided by setuptools. pytest looks up the pytest11 entrypoint to discover its plugins and you can thus make your plugin available by defining it in your setuptools-invocation:

# sample ./setup.py file
from setuptools import setup

setup(
    name="myproject",
    packages=["myproject"],
    # the following makes a plugin available to pytest
    entry_points={"pytest11": ["name_of_plugin = myproject.pluginmodule"]},
    # custom PyPI classifier for pytest plugins
    classifiers=["Framework :: Pytest"],
)

If a package is installed this way, pytest will load myproject.pluginmodule as a plugin which can define hooks.

Note

Make sure to include Framework :: Pytest in your list of PyPI classifiers to make it easy for users to find your plugin.

Assertion Rewriting

One of the main features of pytest is the use of plain assert statements and the detailed introspection of expressions upon assertion failures. This is provided by “assertion rewriting” which modifies the parsed AST before it gets compiled to bytecode. This is done via a PEP 302 import hook which gets installed early on when pytest starts up and will perform this rewriting when modules get imported. However since we do not want to test different bytecode then you will run in production this hook only rewrites test modules themselves as well as any modules which are part of plugins. Any other imported module will not be rewritten and normal assertion behaviour will happen.

If you have assertion helpers in other modules where you would need assertion rewriting to be enabled you need to ask pytest explicitly to rewrite this module before it gets imported.

register_assert_rewrite(*names) → None

Register one or more module names to be rewritten on import.

This function will make sure that this module or all modules inside the package will get their assert statements rewritten. Thus you should make sure to call this before the module is actually imported, usually in your __init__.py if you are a plugin using a package.

Raises:TypeError – if the given module names are not strings.

This is especially important when you write a pytest plugin which is created using a package. The import hook only treats conftest.py files and any modules which are listed in the pytest11 entrypoint as plugins. As an example consider the following package:

pytest_foo/__init__.py
pytest_foo/plugin.py
pytest_foo/helper.py

With the following typical setup.py extract:

setup(..., entry_points={"pytest11": ["foo = pytest_foo.plugin"]}, ...)

In this case only pytest_foo/plugin.py will be rewritten. If the helper module also contains assert statements which need to be rewritten it needs to be marked as such, before it gets imported. This is easiest by marking it for rewriting inside the __init__.py module, which will always be imported first when a module inside a package is imported. This way plugin.py can still import helper.py normally. The contents of pytest_foo/__init__.py will then need to look like this:

import pytest

pytest.register_assert_rewrite("pytest_foo.helper")

Requiring/Loading plugins in a test module or conftest file

You can require plugins in a test module or a conftest.py file like this:

pytest_plugins = ["name1", "name2"]

When the test module or conftest plugin is loaded the specified plugins will be loaded as well. Any module can be blessed as a plugin, including internal application modules:

pytest_plugins = "myapp.testsupport.myplugin"

pytest_plugins variables are processed recursively, so note that in the example above if myapp.testsupport.myplugin also declares pytest_plugins, the contents of the variable will also be loaded as plugins, and so on.

Note

Requiring plugins using a pytest_plugins variable in non-root conftest.py files is deprecated.

This is important because conftest.py files implement per-directory hook implementations, but once a plugin is imported, it will affect the entire directory tree. In order to avoid confusion, defining pytest_plugins in any conftest.py file which is not located in the tests root directory is deprecated, and will raise a warning.

This mechanism makes it easy to share fixtures within applications or even external applications without the need to create external plugins using the setuptools’s entry point technique.

Plugins imported by pytest_plugins will also automatically be marked for assertion rewriting (see pytest.register_assert_rewrite()). However for this to have any effect the module must not be imported already; if it was already imported at the time the pytest_plugins statement is processed, a warning will result and assertions inside the plugin will not be rewritten. To fix this you can either call pytest.register_assert_rewrite() yourself before the module is imported, or you can arrange the code to delay the importing until after the plugin is registered.

Accessing another plugin by name

If a plugin wants to collaborate with code from another plugin it can obtain a reference through the plugin manager like this:

plugin = config.pluginmanager.get_plugin("name_of_plugin")

If you want to look at the names of existing plugins, use the --trace-config option.

Registering custom markers

If your plugin uses any markers, you should register them so that they appear in pytest’s help text and do not cause spurious warnings. For example, the following plugin would register cool_marker and mark_with for all users:

def pytest_configure(config):
    config.addinivalue_line("markers", "cool_marker: this one is for cool tests.")
    config.addinivalue_line(
        "markers", "mark_with(arg, arg2): this marker takes arguments."
    )

Testing plugins

pytest comes with a plugin named pytester that helps you write tests for your plugin code. The plugin is disabled by default, so you will have to enable it before you can use it.

You can do so by adding the following line to a conftest.py file in your testing directory:

# content of conftest.py

pytest_plugins = ["pytester"]

Alternatively you can invoke pytest with the -p pytester command line option.

This will allow you to use the testdir fixture for testing your plugin code.

Let’s demonstrate what you can do with the plugin with an example. Imagine we developed a plugin that provides a fixture hello which yields a function and we can invoke this function with one optional parameter. It will return a string value of Hello World! if we do not supply a value or Hello {value}! if we do supply a string value.

import pytest


def pytest_addoption(parser):
    group = parser.getgroup("helloworld")
    group.addoption(
        "--name",
        action="store",
        dest="name",
        default="World",
        help='Default "name" for hello().',
    )


@pytest.fixture
def hello(request):
    name = request.config.getoption("name")

    def _hello(name=None):
        if not name:
            name = request.config.getoption("name")
        return "Hello {name}!".format(name=name)

    return _hello

Now the testdir fixture provides a convenient API for creating temporary conftest.py files and test files. It also allows us to run the tests and return a result object, with which we can assert the tests’ outcomes.

def test_hello(testdir):
    """Make sure that our plugin works."""

    # create a temporary conftest.py file
    testdir.makeconftest(
        """
        import pytest

        @pytest.fixture(params=[
            "Brianna",
            "Andreas",
            "Floris",
        ])
        def name(request):
            return request.param
    """
    )

    # create a temporary pytest test file
    testdir.makepyfile(
        """
        def test_hello_default(hello):
            assert hello() == "Hello World!"

        def test_hello_name(hello, name):
            assert hello(name) == "Hello {0}!".format(name)
    """
    )

    # run all tests with pytest
    result = testdir.runpytest()

    # check that all 4 tests passed
    result.assert_outcomes(passed=4)

additionally it is possible to copy examples for an example folder before running pytest on it

# content of pytest.ini
[pytest]
pytester_example_dir = .

# content of test_example.py


def test_plugin(testdir):
    testdir.copy_example("test_example.py")
    testdir.runpytest("-k", "test_example")


def test_example():
    pass

$ pytest
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR, inifile: pytest.ini
collected 2 items

test_example.py ..                                                   [100%]

============================= warnings summary =============================
test_example.py::test_plugin
  $REGENDOC_TMPDIR/test_example.py:4: PytestExperimentalApiWarning: testdir.copy_example is an experimental api that may change over time
    testdir.copy_example("test_example.py")

-- Docs: https://docs.pytest.org/en/latest/warnings.html
======================= 2 passed, 1 warning in 0.12s =======================

For more information about the result object that runpytest() returns, and the methods that it provides please check out the RunResult documentation.

Writing hook functions

hook function validation and execution

pytest calls hook functions from registered plugins for any given hook specification. Let’s look at a typical hook function for the pytest_collection_modifyitems(session, config, items) hook which pytest calls after collection of all test items is completed.

When we implement a pytest_collection_modifyitems function in our plugin pytest will during registration verify that you use argument names which match the specification and bail out if not.

Let’s look at a possible implementation:

def pytest_collection_modifyitems(config, items):
    # called after collection is completed
    # you can modify the ``items`` list
    ...

Here, pytest will pass in config (the pytest config object) and items (the list of collected test items) but will not pass in the session argument because we didn’t list it in the function signature. This dynamic “pruning” of arguments allows pytest to be “future-compatible”: we can introduce new hook named parameters without breaking the signatures of existing hook implementations. It is one of the reasons for the general long-lived compatibility of pytest plugins.

Note that hook functions other than pytest_runtest_* are not allowed to raise exceptions. Doing so will break the pytest run.

firstresult: stop at first non-None result

Most calls to pytest hooks result in a list of results which contains all non-None results of the called hook functions.

Some hook specifications use the firstresult=True option so that the hook call only executes until the first of N registered functions returns a non-None result which is then taken as result of the overall hook call. The remaining hook functions will not be called in this case.

hookwrapper: executing around other hooks

pytest plugins can implement hook wrappers which wrap the execution of other hook implementations. A hook wrapper is a generator function which yields exactly once. When pytest invokes hooks it first executes hook wrappers and passes the same arguments as to the regular hooks.

At the yield point of the hook wrapper pytest will execute the next hook implementations and return their result to the yield point in the form of a Result instance which encapsulates a result or exception info. The yield point itself will thus typically not raise exceptions (unless there are bugs).

Here is an example definition of a hook wrapper:

import pytest


@pytest.hookimpl(hookwrapper=True)
def pytest_pyfunc_call(pyfuncitem):
    do_something_before_next_hook_executes()

    outcome = yield
    # outcome.excinfo may be None or a (cls, val, tb) tuple

    res = outcome.get_result()  # will raise if outcome was exception

    post_process_result(res)

    outcome.force_result(new_res)  # to override the return value to the plugin system

Note that hook wrappers don’t return results themselves, they merely perform tracing or other side effects around the actual hook implementations. If the result of the underlying hook is a mutable object, they may modify that result but it’s probably better to avoid it.

For more information, consult the pluggy documentation.

Hook function ordering / call example

For any given hook specification there may be more than one implementation and we thus generally view hook execution as a 1:N function call where N is the number of registered functions. There are ways to influence if a hook implementation comes before or after others, i.e. the position in the N-sized list of functions:

# Plugin 1
@pytest.hookimpl(tryfirst=True)
def pytest_collection_modifyitems(items):
    # will execute as early as possible
    ...


# Plugin 2
@pytest.hookimpl(trylast=True)
def pytest_collection_modifyitems(items):
    # will execute as late as possible
    ...


# Plugin 3
@pytest.hookimpl(hookwrapper=True)
def pytest_collection_modifyitems(items):
    # will execute even before the tryfirst one above!
    outcome = yield
    # will execute after all non-hookwrappers executed

Here is the order of execution:

1. Plugin3’s pytest_collection_modifyitems called until the yield point because it is a hook wrapper.
2. Plugin1’s pytest_collection_modifyitems is called because it is marked with tryfirst=True.
3. Plugin2’s pytest_collection_modifyitems is called because it is marked with trylast=True (but even without this mark it would come after Plugin1).
4. Plugin3’s pytest_collection_modifyitems then executing the code after the yield point. The yield receives a Result instance which encapsulates the result from calling the non-wrappers. Wrappers shall not modify the result.

It’s possible to use tryfirst and trylast also in conjunction with hookwrapper=True in which case it will influence the ordering of hookwrappers among each other.

Declaring new hooks

Plugins and conftest.py files may declare new hooks that can then be implemented by other plugins in order to alter behaviour or interact with the new plugin:

pytest_addhooks(pluginmanager)

called at plugin registration time to allow adding new hooks via a call to pluginmanager.add_hookspecs(module_or_class, prefix).

Parameters:pluginmanager (_pytest.config.PytestPluginManager) – pytest plugin manager

Note

This hook is incompatible with hookwrapper=True.

Hooks are usually declared as do-nothing functions that contain only documentation describing when the hook will be called and what return values are expected. The names of the functions must start with pytest_ otherwise pytest won’t recognize them.

Here’s an example. Let’s assume this code is in the hooks.py module.

def pytest_my_hook(config):
    """
    Receives the pytest config and does things with it
    """

To register the hooks with pytest they need to be structured in their own module or class. This class or module can then be passed to the pluginmanager using the pytest_addhooks function (which itself is a hook exposed by pytest).

def pytest_addhooks(pluginmanager):
    """ This example assumes the hooks are grouped in the 'hooks' module. """
    from my_app.tests import hooks

    pluginmanager.add_hookspecs(hooks)

For a real world example, see newhooks.py from xdist.

Hooks may be called both from fixtures or from other hooks. In both cases, hooks are called through the hook object, available in the config object. Most hooks receive a config object directly, while fixtures may use the pytestconfig fixture which provides the same object.

@pytest.fixture()
def my_fixture(pytestconfig):
    # call the hook called "pytest_my_hook"
    # 'result' will be a list of return values from all registered functions.
    result = pytestconfig.hook.pytest_my_hook(config=pytestconfig)

Note

Hooks receive parameters using only keyword arguments.

Now your hook is ready to be used. To register a function at the hook, other plugins or users must now simply define the function pytest_my_hook with the correct signature in their conftest.py.

Example:

def pytest_my_hook(config):
    """
    Print all active hooks to the screen.
    """
    print(config.hook)

Using hooks in pytest_addoption

Occasionally, it is necessary to change the way in which command line options are defined by one plugin based on hooks in another plugin. For example, a plugin may expose a command line option for which another plugin needs to define the default value. The pluginmanager can be used to install and use hooks to accomplish this. The plugin would define and add the hooks and use pytest_addoption as follows:

# contents of hooks.py

# Use firstresult=True because we only want one plugin to define this
# default value
@hookspec(firstresult=True)
def pytest_config_file_default_value():
    """ Return the default value for the config file command line option. """


# contents of myplugin.py


def pytest_addhooks(pluginmanager):
    """ This example assumes the hooks are grouped in the 'hooks' module. """
    from . import hook

    pluginmanager.add_hookspecs(hook)


def pytest_addoption(parser, pluginmanager):
    default_value = pluginmanager.hook.pytest_config_file_default_value()
    parser.addoption(
        "--config-file",
        help="Config file to use, defaults to %(default)s",
        default=default_value,
    )

The conftest.py that is using myplugin would simply define the hook as follows:

def pytest_config_file_default_value():
    return "config.yaml"

Optionally using hooks from 3rd party plugins

Using new hooks from plugins as explained above might be a little tricky because of the standard validation mechanism: if you depend on a plugin that is not installed, validation will fail and the error message will not make much sense to your users.

One approach is to defer the hook implementation to a new plugin instead of declaring the hook functions directly in your plugin module, for example:

# contents of myplugin.py


class DeferPlugin:
    """Simple plugin to defer pytest-xdist hook functions."""

    def pytest_testnodedown(self, node, error):
        """standard xdist hook function.
        """


def pytest_configure(config):
    if config.pluginmanager.hasplugin("xdist"):
        config.pluginmanager.register(DeferPlugin())

This has the added benefit of allowing you to conditionally install hooks depending on which plugins are installed.

Logging

pytest captures log messages of level WARNING or above automatically and displays them in their own section for each failed test in the same manner as captured stdout and stderr.

Running without options:

pytest

Shows failed tests like so:

----------------------- Captured stdlog call ----------------------
test_reporting.py    26 WARNING  text going to logger
----------------------- Captured stdout call ----------------------
text going to stdout
----------------------- Captured stderr call ----------------------
text going to stderr
==================== 2 failed in 0.02 seconds =====================

By default each captured log message shows the module, line number, log level and message.

If desired the log and date format can be specified to anything that the logging module supports by passing specific formatting options:

pytest --log-format="%(asctime)s %(levelname)s %(message)s" \
        --log-date-format="%Y-%m-%d %H:%M:%S"

Shows failed tests like so:

----------------------- Captured stdlog call ----------------------
2010-04-10 14:48:44 WARNING text going to logger
----------------------- Captured stdout call ----------------------
text going to stdout
----------------------- Captured stderr call ----------------------
text going to stderr
==================== 2 failed in 0.02 seconds =====================

These options can also be customized through pytest.ini file:

[pytest]
log_format = %(asctime)s %(levelname)s %(message)s
log_date_format = %Y-%m-%d %H:%M:%S

Further it is possible to disable reporting of captured content (stdout, stderr and logs) on failed tests completely with:

pytest --show-capture=no

caplog fixture

Inside tests it is possible to change the log level for the captured log messages. This is supported by the caplog fixture:

def test_foo(caplog):
    caplog.set_level(logging.INFO)
    pass

By default the level is set on the root logger, however as a convenience it is also possible to set the log level of any logger:

def test_foo(caplog):
    caplog.set_level(logging.CRITICAL, logger="root.baz")
    pass

The log levels set are restored automatically at the end of the test.

It is also possible to use a context manager to temporarily change the log level inside a with block:

def test_bar(caplog):
    with caplog.at_level(logging.INFO):
        pass

Again, by default the level of the root logger is affected but the level of any logger can be changed instead with:

def test_bar(caplog):
    with caplog.at_level(logging.CRITICAL, logger="root.baz"):
        pass

Lastly all the logs sent to the logger during the test run are made available on the fixture in the form of both the logging.LogRecord instances and the final log text. This is useful for when you want to assert on the contents of a message:

def test_baz(caplog):
    func_under_test()
    for record in caplog.records:
        assert record.levelname != "CRITICAL"
    assert "wally" not in caplog.text

For all the available attributes of the log records see the logging.LogRecord class.

You can also resort to record_tuples if all you want to do is to ensure, that certain messages have been logged under a given logger name with a given severity and message:

def test_foo(caplog):
    logging.getLogger().info("boo %s", "arg")

    assert caplog.record_tuples == [("root", logging.INFO, "boo arg")]

You can call caplog.clear() to reset the captured log records in a test:

def test_something_with_clearing_records(caplog):
    some_method_that_creates_log_records()
    caplog.clear()
    your_test_method()
    assert ["Foo"] == [rec.message for rec in caplog.records]

The caplog.records attribute contains records from the current stage only, so inside the setup phase it contains only setup logs, same with the call and teardown phases.

To access logs from other stages, use the caplog.get_records(when) method. As an example, if you want to make sure that tests which use a certain fixture never log any warnings, you can inspect the records for the setup and call stages during teardown like so:

@pytest.fixture
def window(caplog):
    window = create_window()
    yield window
    for when in ("setup", "call"):
        messages = [
            x.message for x in caplog.get_records(when) if x.levelno == logging.WARNING
        ]
        if messages:
            pytest.fail(
                "warning messages encountered during testing: {}".format(messages)
            )

The full API is available at _pytest.logging.LogCaptureFixture.

Live Logs

By setting the log_cli configuration option to true, pytest will output logging records as they are emitted directly into the console.

You can specify the logging level for which log records with equal or higher level are printed to the console by passing --log-cli-level. This setting accepts the logging level names as seen in python’s documentation or an integer as the logging level num.

Additionally, you can also specify --log-cli-format and --log-cli-date-format which mirror and default to --log-format and --log-date-format if not provided, but are applied only to the console logging handler.

All of the CLI log options can also be set in the configuration INI file. The option names are:

• log_cli_level
• log_cli_format
• log_cli_date_format

If you need to record the whole test suite logging calls to a file, you can pass --log-file=/path/to/log/file. This log file is opened in write mode which means that it will be overwritten at each run tests session.

You can also specify the logging level for the log file by passing --log-file-level. This setting accepts the logging level names as seen in python’s documentation(ie, uppercased level names) or an integer as the logging level num.

Additionally, you can also specify --log-file-format and --log-file-date-format which are equal to --log-format and --log-date-format but are applied to the log file logging handler.

All of the log file options can also be set in the configuration INI file. The option names are:

• log_file
• log_file_level
• log_file_format
• log_file_date_format

You can call set_log_path() to customize the log_file path dynamically. This functionality is considered experimental.

Release notes

This feature was introduced as a drop-in replacement for the pytest-catchlog plugin and they conflict with each other. The backward compatibility API with pytest-capturelog has been dropped when this feature was introduced, so if for that reason you still need pytest-catchlog you can disable the internal feature by adding to your pytest.ini:

[pytest]
    addopts=-p no:logging

Incompatible changes in pytest 3.4

This feature was introduced in 3.3 and some incompatible changes have been made in 3.4 after community feedback:

• Log levels are no longer changed unless explicitly requested by the log_level configuration or --log-level command-line options. This allows users to configure logger objects themselves.
• Live Logs is now disabled by default and can be enabled setting the log_cli configuration option to true. When enabled, the verbosity is increased so logging for each test is visible.
• Live Logs are now sent to sys.stdout and no longer require the -s command-line option to work.

If you want to partially restore the logging behavior of version 3.3, you can add this options to your ini file:

[pytest]
log_cli=true
log_level=NOTSET

More details about the discussion that lead to this changes can be read in issue #3013.

API Reference

This page contains the full reference to pytest’s API.

Functions

pytest.approx

approx(expected, rel=None, abs=None, nan_ok=False)

Assert that two numbers (or two sets of numbers) are equal to each other within some tolerance.

Due to the intricacies of floating-point arithmetic, numbers that we would intuitively expect to be equal are not always so:

>>> 0.1 + 0.2 == 0.3
False

This problem is commonly encountered when writing tests, e.g. when making sure that floating-point values are what you expect them to be. One way to deal with this problem is to assert that two floating-point numbers are equal to within some appropriate tolerance:

>>> abs((0.1 + 0.2) - 0.3) < 1e-6
True

However, comparisons like this are tedious to write and difficult to understand. Furthermore, absolute comparisons like the one above are usually discouraged because there’s no tolerance that works well for all situations. 1e-6 is good for numbers around 1, but too small for very big numbers and too big for very small ones. It’s better to express the tolerance as a fraction of the expected value, but relative comparisons like that are even more difficult to write correctly and concisely.

The approx class performs floating-point comparisons using a syntax that’s as intuitive as possible:

>>> from pytest import approx
>>> 0.1 + 0.2 == approx(0.3)
True

The same syntax also works for sequences of numbers:

>>> (0.1 + 0.2, 0.2 + 0.4) == approx((0.3, 0.6))
True

Dictionary values:

>>> {'a': 0.1 + 0.2, 'b': 0.2 + 0.4} == approx({'a': 0.3, 'b': 0.6})
True

numpy arrays:

>>> import numpy as np                                                          
>>> np.array([0.1, 0.2]) + np.array([0.2, 0.4]) == approx(np.array([0.3, 0.6])) 
True

And for a numpy array against a scalar:

>>> import numpy as np                                         
>>> np.array([0.1, 0.2]) + np.array([0.2, 0.1]) == approx(0.3) 
True

By default, approx considers numbers within a relative tolerance of 1e-6 (i.e. one part in a million) of its expected value to be equal. This treatment would lead to surprising results if the expected value was 0.0, because nothing but 0.0 itself is relatively close to 0.0. To handle this case less surprisingly, approx also considers numbers within an absolute tolerance of 1e-12 of its expected value to be equal. Infinity and NaN are special cases. Infinity is only considered equal to itself, regardless of the relative tolerance. NaN is not considered equal to anything by default, but you can make it be equal to itself by setting the nan_ok argument to True. (This is meant to facilitate comparing arrays that use NaN to mean “no data”.)

Both the relative and absolute tolerances can be changed by passing arguments to the approx constructor:

>>> 1.0001 == approx(1)
False
>>> 1.0001 == approx(1, rel=1e-3)
True
>>> 1.0001 == approx(1, abs=1e-3)
True

If you specify abs but not rel, the comparison will not consider the relative tolerance at all. In other words, two numbers that are within the default relative tolerance of 1e-6 will still be considered unequal if they exceed the specified absolute tolerance. If you specify both abs and rel, the numbers will be considered equal if either tolerance is met:

>>> 1 + 1e-8 == approx(1)
True
>>> 1 + 1e-8 == approx(1, abs=1e-12)
False
>>> 1 + 1e-8 == approx(1, rel=1e-6, abs=1e-12)
True

If you’re thinking about using approx, then you might want to know how it compares to other good ways of comparing floating-point numbers. All of these algorithms are based on relative and absolute tolerances and should agree for the most part, but they do have meaningful differences:

• math.isclose(a, b, rel_tol=1e-9, abs_tol=0.0): True if the relative tolerance is met w.r.t. either a or b or if the absolute tolerance is met. Because the relative tolerance is calculated w.r.t. both a and b, this test is symmetric (i.e. neither a nor b is a “reference value”). You have to specify an absolute tolerance if you want to compare to 0.0 because there is no tolerance by default. Only available in python>=3.5. More information…
• numpy.isclose(a, b, rtol=1e-5, atol=1e-8): True if the difference between a and b is less that the sum of the relative tolerance w.r.t. b and the absolute tolerance. Because the relative tolerance is only calculated w.r.t. b, this test is asymmetric and you can think of b as the reference value. Support for comparing sequences is provided by numpy.allclose. More information…
• unittest.TestCase.assertAlmostEqual(a, b): True if a and b are within an absolute tolerance of 1e-7. No relative tolerance is considered and the absolute tolerance cannot be changed, so this function is not appropriate for very large or very small numbers. Also, it’s only available in subclasses of unittest.TestCase and it’s ugly because it doesn’t follow PEP8. More information…
• a == pytest.approx(b, rel=1e-6, abs=1e-12): True if the relative tolerance is met w.r.t. b or if the absolute tolerance is met. Because the relative tolerance is only calculated w.r.t. b, this test is asymmetric and you can think of b as the reference value. In the special case that you explicitly specify an absolute tolerance but not a relative tolerance, only the absolute tolerance is considered.

Warning

Changed in version 3.2.

In order to avoid inconsistent behavior, TypeError is raised for >, >=, < and <= comparisons. The example below illustrates the problem:

assert approx(0.1) > 0.1 + 1e-10  # calls approx(0.1).__gt__(0.1 + 1e-10)
assert 0.1 + 1e-10 > approx(0.1)  # calls approx(0.1).__lt__(0.1 + 1e-10)

In the second example one expects approx(0.1).__le__(0.1 + 1e-10) to be called. But instead, approx(0.1).__lt__(0.1 + 1e-10) is used to comparison. This is because the call hierarchy of rich comparisons follows a fixed behavior. More information…

pytest.fail

Tutorial: Skip and xfail: dealing with tests that cannot succeed

fail(msg: str = '', pytrace: bool = True) → NoReturn

Explicitly fail an executing test with the given message.

Parameters:

• msg (str) – the message to show the user as reason for the failure.
• pytrace (bool) – if false the msg represents the full failure information and no python traceback will be reported.

pytest.skip

skip(msg[, allow_module_level=False])

Skip an executing test with the given message.

This function should be called only during testing (setup, call or teardown) or during collection by using the allow_module_level flag. This function can be called in doctests as well.

Parameters:allow_module_level (bool) – allows this function to be called at module level, skipping the rest of the module. Default to False.

Note

It is better to use the pytest.mark.skipif marker when possible to declare a test to be skipped under certain conditions like mismatching platforms or dependencies. Similarly, use the # doctest: +SKIP directive (see doctest.SKIP) to skip a doctest statically.

pytest.importorskip

importorskip(modname: str, minversion: Optional[str] = None, reason: Optional[str] = None) → Any

Imports and returns the requested module modname, or skip the current test if the module cannot be imported.

Parameters:

• modname (str) – the name of the module to import
• minversion (str) – if given, the imported module’s __version__ attribute must be at least this minimal version, otherwise the test is still skipped.
• reason (str) – if given, this reason is shown as the message when the module cannot be imported.

Returns:

The imported module. This should be assigned to its canonical name.

Example:

docutils = pytest.importorskip("docutils")

pytest.xfail

xfail(reason: str = '') → NoReturn

Imperatively xfail an executing test or setup functions with the given reason.

This function should be called only during testing (setup, call or teardown).

Note

It is better to use the pytest.mark.xfail marker when possible to declare a test to be xfailed under certain conditions like known bugs or missing features.

pytest.exit

exit(msg: str, returncode: Optional[int] = None) → NoReturn

Exit testing process.

Parameters:

• msg (str) – message to display upon exit.
• returncode (int) – return code to be used when exiting pytest.

pytest.main

main(args=None, plugins=None) → Union[int, _pytest.main.ExitCode]

return exit code, after performing an in-process test run.

Parameters:

• args – list of command line arguments.
• plugins – list of plugin objects to be auto-registered during initialization.

pytest.param

param(*values[, id][, marks])

Specify a parameter in pytest.mark.parametrize calls or parametrized fixtures.

@pytest.mark.parametrize("test_input,expected", [
    ("3+5", 8),
    pytest.param("6*9", 42, marks=pytest.mark.xfail),
])
def test_eval(test_input, expected):
    assert eval(test_input) == expected

Parameters:

• values – variable args of the values of the parameter set, in order.
• marks – a single mark or a list of marks to be applied to this parameter set.
• id (str) – the id to attribute to this parameter set.

pytest.raises

Tutorial: Assertions about expected exceptions.

with raises(expected_exception: Exception[, *, match]) as excinfo

Assert that a code block/function call raises expected_exception or raise a failure exception otherwise.

Parameters:match –

if specified, a string containing a regular expression, or a regular expression object, that is tested against the string representation of the exception using re.search. To match a literal string that may contain special characters, the pattern can first be escaped with re.escape.

Use pytest.raises as a context manager, which will capture the exception of the given type:

>>> with raises(ZeroDivisionError):
...    1/0

If the code block does not raise the expected exception (ZeroDivisionError in the example above), or no exception at all, the check will fail instead.

You can also use the keyword argument match to assert that the exception matches a text or regex:

>>> with raises(ValueError, match='must be 0 or None'):
...     raise ValueError("value must be 0 or None")

>>> with raises(ValueError, match=r'must be \d+$'):
...     raise ValueError("value must be 42")

The context manager produces an ExceptionInfo object which can be used to inspect the details of the captured exception:

>>> with raises(ValueError) as exc_info:
...     raise ValueError("value must be 42")
>>> assert exc_info.type is ValueError
>>> assert exc_info.value.args[0] == "value must be 42"

Deprecated since version 4.1: In the context manager form you may use the keyword argument message to specify a custom failure message that will be displayed in case the pytest.raises check fails. This has been deprecated as it is considered error prone as users often mean to use match instead. See the deprecation docs for a workaround.

Note

When using pytest.raises as a context manager, it’s worthwhile to note that normal context manager rules apply and that the exception raised must be the final line in the scope of the context manager. Lines of code after that, within the scope of the context manager will not be executed. For example:

>>> value = 15
>>> with raises(ValueError) as exc_info:
...     if value > 10:
...         raise ValueError("value must be <= 10")
...     assert exc_info.type is ValueError  # this will not execute

Instead, the following approach must be taken (note the difference in scope):

>>> with raises(ValueError) as exc_info:
...     if value > 10:
...         raise ValueError("value must be <= 10")
...
>>> assert exc_info.type is ValueError

Using with pytest.mark.parametrize

When using pytest.mark.parametrize it is possible to parametrize tests such that some runs raise an exception and others do not.

See Parametrizing conditional raising for an example.

Legacy form

It is possible to specify a callable by passing a to-be-called lambda:

>>> raises(ZeroDivisionError, lambda: 1/0)
<ExceptionInfo ...>

or you can specify an arbitrary callable with arguments:

>>> def f(x): return 1/x
...
>>> raises(ZeroDivisionError, f, 0)
<ExceptionInfo ...>
>>> raises(ZeroDivisionError, f, x=0)
<ExceptionInfo ...>

The form above is fully supported but discouraged for new code because the context manager form is regarded as more readable and less error-prone.

Note

Similar to caught exception objects in Python, explicitly clearing local references to returned ExceptionInfo objects can help the Python interpreter speed up its garbage collection.

Clearing those references breaks a reference cycle (ExceptionInfo –> caught exception –> frame stack raising the exception –> current frame stack –> local variables –> ExceptionInfo) which makes Python keep all objects referenced from that cycle (including all local variables in the current frame) alive until the next cyclic garbage collection run. See the official Python try statement documentation for more detailed information.

pytest.deprecated_call

Tutorial: Ensuring code triggers a deprecation warning.

with deprecated_call()

context manager that can be used to ensure a block of code triggers a DeprecationWarning or PendingDeprecationWarning:

>>> import warnings
>>> def api_call_v2():
...     warnings.warn('use v3 of this api', DeprecationWarning)
...     return 200

>>> with deprecated_call():
...    assert api_call_v2() == 200

deprecated_call can also be used by passing a function and *args and *kwargs, in which case it will ensure calling func(*args, **kwargs) produces one of the warnings types above.

pytest.register_assert_rewrite

Tutorial: Assertion Rewriting.

register_assert_rewrite(*names) → None

Register one or more module names to be rewritten on import.

This function will make sure that this module or all modules inside the package will get their assert statements rewritten. Thus you should make sure to call this before the module is actually imported, usually in your __init__.py if you are a plugin using a package.

Raises:TypeError – if the given module names are not strings.

pytest.warns

Tutorial: Asserting warnings with the warns function

with warns(expected_warning: Exception[, match])

Assert that code raises a particular class of warning.

Specifically, the parameter expected_warning can be a warning class or sequence of warning classes, and the inside the with block must issue a warning of that class or classes.

This helper produces a list of warnings.WarningMessage objects, one for each warning raised.

This function can be used as a context manager, or any of the other ways pytest.raises can be used:

>>> with warns(RuntimeWarning):
...    warnings.warn("my warning", RuntimeWarning)

In the context manager form you may use the keyword argument match to assert that the exception matches a text or regex:

>>> with warns(UserWarning, match='must be 0 or None'):
...     warnings.warn("value must be 0 or None", UserWarning)

>>> with warns(UserWarning, match=r'must be \d+$'):
...     warnings.warn("value must be 42", UserWarning)

>>> with warns(UserWarning, match=r'must be \d+$'):
...     warnings.warn("this is not here", UserWarning)
Traceback (most recent call last):
  ...
Failed: DID NOT WARN. No warnings of type ...UserWarning... was emitted...

pytest.freeze_includes

Tutorial: Freezing pytest.

freeze_includes()

Returns a list of module names used by pytest that should be included by cx_freeze.

Marks

Marks can be used apply meta data to test functions (but not fixtures), which can then be accessed by fixtures or plugins.

pytest.mark.filterwarnings

Tutorial: @pytest.mark.filterwarnings.

Add warning filters to marked test items.

pytest.mark.filterwarnings(filter)

Parameters:filter (str) –

A warning specification string, which is composed of contents of the tuple (action, message, category, module, lineno) as specified in The Warnings filter section of the Python documentation, separated by ":". Optional fields can be omitted. Module names passed for filtering are not regex-escaped.

For example:

@pytest.mark.warnings("ignore:.*usage will be deprecated.*:DeprecationWarning")
def test_foo():
    ...

pytest.mark.parametrize

Tutorial: Parametrizing fixtures and test functions.

Metafunc.parametrize(argnames, argvalues, indirect=False, ids=None, scope=None)

Add new invocations to the underlying test function using the list of argvalues for the given argnames. Parametrization is performed during the collection phase. If you need to setup expensive resources see about setting indirect to do it rather at test setup time.

Parameters:

• argnames – a comma-separated string denoting one or more argument names, or a list/tuple of argument strings.
• argvalues – The list of argvalues determines how often a test is invoked with different argument values. If only one argname was specified argvalues is a list of values. If N argnames were specified, argvalues must be a list of N-tuples, where each tuple-element specifies a value for its respective argname.
• indirect – The list of argnames or boolean. A list of arguments’ names (subset of argnames). If True the list contains all names from the argnames. Each argvalue corresponding to an argname in this list will be passed as request.param to its respective argname fixture function so that it can perform more expensive setups during the setup phase of a test rather than at collection time.
• ids – list of string ids, or a callable. If strings, each is corresponding to the argvalues so that they are part of the test id. If None is given as id of specific test, the automatically generated id for that argument will be used. If callable, it should take one argument (a single argvalue) and return a string or return None. If None, the automatically generated id for that argument will be used. If no ids are provided they will be generated automatically from the argvalues.
• scope – if specified it denotes the scope of the parameters. The scope is used for grouping tests by parameter instances. It will also override any fixture-function defined scope, allowing to set a dynamic scope using test context or configuration.

pytest.mark.skip

Tutorial: Skipping test functions.

Unconditionally skip a test function.

pytest.mark.skip(*, reason=None)

Parameters:reason (str) – Reason why the test function is being skipped.

pytest.mark.skipif

Tutorial: Skipping test functions.

Skip a test function if a condition is True.

pytest.mark.skipif(condition, *, reason=None)

Parameters:

• condition (bool or str) – True/False if the condition should be skipped or a condition string.
• reason (str) – Reason why the test function is being skipped.

pytest.mark.usefixtures

Tutorial: Using fixtures from classes, modules or projects.

Mark a test function as using the given fixture names.

Warning

This mark has no effect when applied to a fixture function.

pytest.mark.usefixtures(*names)

Parameters:args – the names of the fixture to use, as strings

pytest.mark.xfail

Tutorial: XFail: mark test functions as expected to fail.

Marks a test function as expected to fail.

pytest.mark.xfail(condition=None, *, reason=None, raises=None, run=True, strict=False)

Parameters:

• condition (bool or str) – Condition for marking the test function as xfail (True/False or a condition string).
• reason (str) – Reason why the test function is marked as xfail.
• raises (Exception) – Exception subclass expected to be raised by the test function; other exceptions will fail the test.
• run (bool) – If the test function should actually be executed. If False, the function will always xfail and will not be executed (useful if a function is segfaulting).
• strict (bool) –
  • If False (the default) the function will be shown in the terminal output as xfailed if it fails and as xpass if it passes. In both cases this will not cause the test suite to fail as a whole. This is particularly useful to mark flaky tests (tests that fail at random) to be tackled later.
  • If True, the function will be shown in the terminal output as xfailed if it fails, but if it unexpectedly passes then it will fail the test suite. This is particularly useful to mark functions that are always failing and there should be a clear indication if they unexpectedly start to pass (for example a new release of a library fixes a known bug).

custom marks

Marks are created dynamically using the factory object pytest.mark and applied as a decorator.

For example:

@pytest.mark.timeout(10, "slow", method="thread")
def test_function():
    ...

Will create and attach a Mark object to the collected Item, which can then be accessed by fixtures or hooks with Node.iter_markers. The mark object will have the following attributes:

mark.args == (10, "slow")
mark.kwargs == {"method": "thread"}

Fixtures

Tutorial: pytest fixtures: explicit, modular, scalable.

Fixtures are requested by test functions or other fixtures by declaring them as argument names.

Example of a test requiring a fixture:

def test_output(capsys):
    print("hello")
    out, err = capsys.readouterr()
    assert out == "hello\n"

Example of a fixture requiring another fixture:

@pytest.fixture
def db_session(tmpdir):
    fn = tmpdir / "db.file"
    return connect(str(fn))

For more details, consult the full fixtures docs.

@pytest.fixture

@fixture(callable_or_scope=None, *args, scope='function', params=None, autouse=False, ids=None, name=None)

Decorator to mark a fixture factory function.

This decorator can be used, with or without parameters, to define a fixture function.

The name of the fixture function can later be referenced to cause its invocation ahead of running tests: test modules or classes can use the pytest.mark.usefixtures(fixturename) marker.

Test functions can directly use fixture names as input arguments in which case the fixture instance returned from the fixture function will be injected.

Fixtures can provide their values to test functions using return or yield statements. When using yield the code block after the yield statement is executed as teardown code regardless of the test outcome, and must yield exactly once.

Parameters:

• scope –

  the scope for which this fixture is shared, one of "function" (default), "class", "module", "package" or "session" ("package" is considered experimental at this time).

  This parameter may also be a callable which receives (fixture_name, config) as parameters, and must return a str with one of the values mentioned above.

  See Dynamic scope in the docs for more information.

• params – an optional list of parameters which will cause multiple invocations of the fixture function and all of the tests using it. The current parameter is available in request.param.
• autouse – if True, the fixture func is activated for all tests that can see it. If False (the default) then an explicit reference is needed to activate the fixture.
• ids – list of string ids each corresponding to the params so that they are part of the test id. If no ids are provided they will be generated automatically from the params.
• name – the name of the fixture. This defaults to the name of the decorated function. If a fixture is used in the same module in which it is defined, the function name of the fixture will be shadowed by the function arg that requests the fixture; one way to resolve this is to name the decorated function fixture_<fixturename> and then use @pytest.fixture(name='<fixturename>').

config.cache

Tutorial: Cache: working with cross-testrun state.

The config.cache object allows other plugins and fixtures to store and retrieve values across test runs. To access it from fixtures request pytestconfig into your fixture and get it with pytestconfig.cache.

Under the hood, the cache plugin uses the simple dumps/loads API of the json stdlib module.

Cache.get(key, default)

return cached value for the given key. If no value was yet cached or the value cannot be read, the specified default is returned.

Parameters:

• key – must be a / separated value. Usually the first name is the name of your plugin or your application.
• default – must be provided in case of a cache-miss or invalid cache values.

Cache.set(key, value)

save value for the given key.

Parameters:

• key – must be a / separated value. Usually the first name is the name of your plugin or your application.
• value – must be of any combination of basic python types, including nested types like e. g. lists of dictionaries.

Cache.makedir(name)

return a directory path object with the given name. If the directory does not yet exist, it will be created. You can use it to manage files likes e. g. store/retrieve database dumps across test sessions.

Parameters:name – must be a string not containing a / separator. Make sure the name contains your plugin or application identifiers to prevent clashes with other cache users.

capsys

Tutorial: Capturing of the stdout/stderr output.

capsys()

Enable text capturing of writes to sys.stdout and sys.stderr.

The captured output is made available via capsys.readouterr() method calls, which return a (out, err) namedtuple. out and err will be text objects.

Returns an instance of CaptureFixture.

Example:

def test_output(capsys):
    print("hello")
    captured = capsys.readouterr()
    assert captured.out == "hello\n"

class CaptureFixture

Object returned by capsys(), capsysbinary(), capfd() and capfdbinary() fixtures.

readouterr()

Read and return the captured output so far, resetting the internal buffer.

Returns:captured content as a namedtuple with out and err string attributes

with disabled()

Temporarily disables capture while inside the ‘with’ block.

capsysbinary

Tutorial: Capturing of the stdout/stderr output.

capsysbinary()

Enable bytes capturing of writes to sys.stdout and sys.stderr.

The captured output is made available via capsysbinary.readouterr() method calls, which return a (out, err) namedtuple. out and err will be bytes objects.

Returns an instance of CaptureFixture.

Example:

def test_output(capsysbinary):
    print("hello")
    captured = capsysbinary.readouterr()
    assert captured.out == b"hello\n"

capfd

Tutorial: Capturing of the stdout/stderr output.

capfd()

Enable text capturing of writes to file descriptors 1 and 2.

The captured output is made available via capfd.readouterr() method calls, which return a (out, err) namedtuple. out and err will be text objects.

Returns an instance of CaptureFixture.

Example:

def test_system_echo(capfd):
    os.system('echo "hello"')
    captured = capfd.readouterr()
    assert captured.out == "hello\n"

capfdbinary

Tutorial: Capturing of the stdout/stderr output.

capfdbinary()

Enable bytes capturing of writes to file descriptors 1 and 2.

The captured output is made available via capfd.readouterr() method calls, which return a (out, err) namedtuple. out and err will be byte objects.

Returns an instance of CaptureFixture.

Example:

def test_system_echo(capfdbinary):
    os.system('echo "hello"')
    captured = capfdbinary.readouterr()
    assert captured.out == b"hello\n"

doctest_namespace

Tutorial: Doctest integration for modules and test files.

doctest_namespace()

Fixture that returns a dict that will be injected into the namespace of doctests.

Usually this fixture is used in conjunction with another autouse fixture:

@pytest.fixture(autouse=True)
def add_np(doctest_namespace):
    doctest_namespace["np"] = numpy

For more details: ‘doctest_namespace’ fixture.

request

Tutorial: Pass different values to a test function, depending on command line options.

The request fixture is a special fixture providing information of the requesting test function.

class FixtureRequest

A request for a fixture from a test or fixture function.

A request object gives access to the requesting test context and has an optional param attribute in case the fixture is parametrized indirectly.

fixturename = None

fixture for which this request is being performed

scope = None

Scope string, one of “function”, “class”, “module”, “session”

fixturenames

names of all active fixtures in this request

funcargnames

alias attribute for fixturenames for pre-2.3 compatibility

node

underlying collection node (depends on current request scope)

config

the pytest config object associated with this request.

function

test function object if the request has a per-function scope.

cls

class (can be None) where the test function was collected.

instance

instance (can be None) on which test function was collected.

module

python module object where the test function was collected.

fspath

the file system path of the test module which collected this test.

keywords

keywords/markers dictionary for the underlying node.

session

pytest session object.

addfinalizer(finalizer)

add finalizer/teardown function to be called after the last test within the requesting test context finished execution.

applymarker(marker)

Apply a marker to a single test function invocation. This method is useful if you don’t want to have a keyword/marker on all function invocations.

Parameters:marker – a _pytest.mark.MarkDecorator object created by a call to pytest.mark.NAME(...).

raiseerror(msg)

raise a FixtureLookupError with the given message.

getfixturevalue(argname)

Dynamically run a named fixture function.

Declaring fixtures via function argument is recommended where possible. But if you can only decide whether to use another fixture at test setup time, you may use this function to retrieve it inside a fixture or test function body.

pytestconfig

pytestconfig()

Session-scoped fixture that returns the _pytest.config.Config object.

Example:

def test_foo(pytestconfig):
    if pytestconfig.getoption("verbose") > 0:
        ...

record_property

Tutorial: record_property.

record_property()

Add an extra properties the calling test. User properties become part of the test report and are available to the configured reporters, like JUnit XML. The fixture is callable with (name, value), with value being automatically xml-encoded.

Example:

def test_function(record_property):
    record_property("example_key", 1)

record_testsuite_property

Tutorial: record_testsuite_property.

record_testsuite_property()

Records a new <property> tag as child of the root <testsuite>. This is suitable to writing global information regarding the entire test suite, and is compatible with xunit2 JUnit family.

This is a session-scoped fixture which is called with (name, value). Example:

def test_foo(record_testsuite_property):
    record_testsuite_property("ARCH", "PPC")
    record_testsuite_property("STORAGE_TYPE", "CEPH")

name must be a string, value will be converted to a string and properly xml-escaped.

caplog

Tutorial: Logging.

caplog()

Access and control log capturing.

Captured logs are available through the following properties/methods:

* caplog.messages        -> list of format-interpolated log messages
* caplog.text            -> string containing formatted log output
* caplog.records         -> list of logging.LogRecord instances
* caplog.record_tuples   -> list of (logger_name, level, message) tuples
* caplog.clear()         -> clear captured records and formatted log output string

This returns a _pytest.logging.LogCaptureFixture instance.

class LogCaptureFixture(item)

Provides access and control of log capturing.

handler

Return type:LogCaptureHandler

get_records(when)

Get the logging records for one of the possible test phases.

Parameters:when (str) – Which test phase to obtain the records from. Valid values are: “setup”, “call” and “teardown”. Return type:List[logging.LogRecord] Returns:the list of captured records at the given stage

New in version 3.4.

text

Returns the formatted log text.

records

Returns the list of log records.

record_tuples

Returns a list of a stripped down version of log records intended for use in assertion comparison.

The format of the tuple is:

(logger_name, log_level, message)

messages

Returns a list of format-interpolated log messages.

Unlike ‘records’, which contains the format string and parameters for interpolation, log messages in this list are all interpolated. Unlike ‘text’, which contains the output from the handler, log messages in this list are unadorned with levels, timestamps, etc, making exact comparisons more reliable.

Note that traceback or stack info (from logging.exception() or the exc_info or stack_info arguments to the logging functions) is not included, as this is added by the formatter in the handler.

New in version 3.7.

clear()

Reset the list of log records and the captured log text.

set_level(level, logger=None)

Sets the level for capturing of logs. The level will be restored to its previous value at the end of the test.

Parameters:

• level (int) – the logger to level.
• logger (str) – the logger to update the level. If not given, the root logger level is updated.

Changed in version 3.4: The levels of the loggers changed by this function will be restored to their initial values at the end of the test.

with at_level(level, logger=None)

Context manager that sets the level for capturing of logs. After the end of the ‘with’ statement the level is restored to its original value.

Parameters:

• level (int) – the logger to level.
• logger (str) – the logger to update the level. If not given, the root logger level is updated.

monkeypatch

Tutorial: Monkeypatching/mocking modules and environments.

monkeypatch()

The returned monkeypatch fixture provides these helper methods to modify objects, dictionaries or os.environ:

monkeypatch.setattr(obj, name, value, raising=True)
monkeypatch.delattr(obj, name, raising=True)
monkeypatch.setitem(mapping, name, value)
monkeypatch.delitem(obj, name, raising=True)
monkeypatch.setenv(name, value, prepend=False)
monkeypatch.delenv(name, raising=True)
monkeypatch.syspath_prepend(path)
monkeypatch.chdir(path)

All modifications will be undone after the requesting test function or fixture has finished. The raising parameter determines if a KeyError or AttributeError will be raised if the set/deletion operation has no target.

This returns a MonkeyPatch instance.

class MonkeyPatch

Object returned by the monkeypatch fixture keeping a record of setattr/item/env/syspath changes.

with context()

Context manager that returns a new MonkeyPatch object which undoes any patching done inside the with block upon exit:

import functools
def test_partial(monkeypatch):
    with monkeypatch.context() as m:
        m.setattr(functools, "partial", 3)

Useful in situations where it is desired to undo some patches before the test ends, such as mocking stdlib functions that might break pytest itself if mocked (for examples of this see #3290.

setattr(target, name, value=<notset>, raising=True)

Set attribute value on target, memorizing the old value. By default raise AttributeError if the attribute did not exist.

For convenience you can specify a string as target which will be interpreted as a dotted import path, with the last part being the attribute name. Example: monkeypatch.setattr("os.getcwd", lambda: "/") would set the getcwd function of the os module.

The raising value determines if the setattr should fail if the attribute is not already present (defaults to True which means it will raise).

delattr(target, name=<notset>, raising=True)

Delete attribute name from target, by default raise AttributeError it the attribute did not previously exist.

If no name is specified and target is a string it will be interpreted as a dotted import path with the last part being the attribute name.

If raising is set to False, no exception will be raised if the attribute is missing.

setitem(dic, name, value)

Set dictionary entry name to value.

delitem(dic, name, raising=True)

Delete name from dict. Raise KeyError if it doesn’t exist.

If raising is set to False, no exception will be raised if the key is missing.

setenv(name, value, prepend=None)

Set environment variable name to value. If prepend is a character, read the current environment variable value and prepend the value adjoined with the prepend character.

delenv(name, raising=True)

Delete name from the environment. Raise KeyError if it does not exist.

If raising is set to False, no exception will be raised if the environment variable is missing.

syspath_prepend(path)

Prepend path to sys.path list of import locations.

chdir(path)

Change the current working directory to the specified path. Path can be a string or a py.path.local object.

undo()

Undo previous changes. This call consumes the undo stack. Calling it a second time has no effect unless you do more monkeypatching after the undo call.

There is generally no need to call undo(), since it is called automatically during tear-down.

Note that the same monkeypatch fixture is used across a single test function invocation. If monkeypatch is used both by the test function itself and one of the test fixtures, calling undo() will undo all of the changes made in both functions.

testdir

This fixture provides a Testdir instance useful for black-box testing of test files, making it ideal to test plugins.

To use it, include in your top-most conftest.py file:

pytest_plugins = "pytester"

class Testdir

Temporary test directory with tools to test/run pytest itself.

This is based on the tmpdir fixture but provides a number of methods which aid with testing pytest itself. Unless chdir() is used all methods will use tmpdir as their current working directory.

Attributes:

Variables:

• tmpdir – The py.path.local instance of the temporary directory.
• plugins – A list of plugins to use with parseconfig() and runpytest(). Initially this is an empty list but plugins can be added to the list. The type of items to add to the list depends on the method using them so refer to them for details.

CLOSE_STDIN

alias of builtins.object

exception TimeoutExpired

finalize()

Clean up global state artifacts.

Some methods modify the global interpreter state and this tries to clean this up. It does not remove the temporary directory however so it can be looked at after the test run has finished.

make_hook_recorder(pluginmanager)

Create a new HookRecorder for a PluginManager.

chdir()

Cd into the temporary directory.

This is done automatically upon instantiation.

makefile(ext, *args, **kwargs)

Create new file(s) in the testdir.

Parameters:

• ext (str) – The extension the file(s) should use, including the dot, e.g. .py.
• args (list[str]) – All args will be treated as strings and joined using newlines. The result will be written as contents to the file. The name of the file will be based on the test function requesting this fixture.
• kwargs – Each keyword is the name of a file, while the value of it will be written as contents of the file.

Examples:

testdir.makefile(".txt", "line1", "line2")

testdir.makefile(".ini", pytest="[pytest]\naddopts=-rs\n")

makeconftest(source)

Write a contest.py file with ‘source’ as contents.

makeini(source)

Write a tox.ini file with ‘source’ as contents.

getinicfg(source)

Return the pytest section from the tox.ini config file.

makepyfile(*args, **kwargs)

Shortcut for .makefile() with a .py extension.

maketxtfile(*args, **kwargs)

Shortcut for .makefile() with a .txt extension.

syspathinsert(path=None)

Prepend a directory to sys.path, defaults to tmpdir.

This is undone automatically when this object dies at the end of each test.

mkdir(name)

Create a new (sub)directory.

mkpydir(name)

Create a new python package.

This creates a (sub)directory with an empty __init__.py file so it gets recognised as a python package.

copy_example(name=None)

Copy file from project’s directory into the testdir.

Parameters:name (str) – The name of the file to copy. Returns:path to the copied directory (inside self.tmpdir).

class Session(config)

exception Failed

signals a stop as failed test run.

exception Interrupted

signals an interrupted test run.

for ... in collect()

returns a list of children (items and collectors) for this collection node.

getnode(config, arg)

Return the collection node of a file.

Parameters:

• config – _pytest.config.Config instance, see parseconfig() and parseconfigure() to create the configuration
• arg – a py.path.local instance of the file

getpathnode(path)

Return the collection node of a file.

This is like getnode() but uses parseconfigure() to create the (configured) pytest Config instance.

Parameters:path – a py.path.local instance of the file

genitems(colitems)

Generate all test items from a collection node.

This recurses into the collection node and returns a list of all the test items contained within.

runitem(source)

Run the “test_func” Item.

The calling test instance (class containing the test method) must provide a .getrunner() method which should return a runner which can run the test protocol for a single item, e.g. _pytest.runner.runtestprotocol().

inline_runsource(source, *cmdlineargs)

Run a test module in process using pytest.main().

This run writes “source” into a temporary file and runs pytest.main() on it, returning a HookRecorder instance for the result.

Parameters:

• source – the source code of the test module
• cmdlineargs – any extra command line arguments to use

Returns:

HookRecorder instance of the result

inline_genitems(*args)

Run pytest.main(['--collectonly']) in-process.

Runs the pytest.main() function to run all of pytest inside the test process itself like inline_run(), but returns a tuple of the collected items and a HookRecorder instance.

inline_run(*args, plugins=(), no_reraise_ctrlc: bool = False)

Run pytest.main() in-process, returning a HookRecorder.

Runs the pytest.main() function to run all of pytest inside the test process itself. This means it can return a HookRecorder instance which gives more detailed results from that run than can be done by matching stdout/stderr from runpytest().

Parameters:

• args – command line arguments to pass to pytest.main()
• plugins – extra plugin instances the pytest.main() instance should use.
• no_reraise_ctrlc – typically we reraise keyboard interrupts from the child run. If True, the KeyboardInterrupt exception is captured.

Returns:

a HookRecorder instance

runpytest_inprocess(*args, **kwargs) → _pytest.pytester.RunResult

Return result of running pytest in-process, providing a similar interface to what self.runpytest() provides.

runpytest(*args, **kwargs) → _pytest.pytester.RunResult

Run pytest inline or in a subprocess, depending on the command line option “–runpytest” and return a RunResult.

parseconfig(*args)

Return a new pytest Config instance from given commandline args.

This invokes the pytest bootstrapping code in _pytest.config to create a new _pytest.core.PluginManager and call the pytest_cmdline_parse hook to create a new _pytest.config.Config instance.

If plugins has been populated they should be plugin modules to be registered with the PluginManager.

parseconfigure(*args)

Return a new pytest configured Config instance.

This returns a new _pytest.config.Config instance like parseconfig(), but also calls the pytest_configure hook.

getitem(source, funcname='test_func')

Return the test item for a test function.

This writes the source to a python file and runs pytest’s collection on the resulting module, returning the test item for the requested function name.

Parameters:

• source – the module source
• funcname – the name of the test function for which to return a test item

getitems(source)

Return all test items collected from the module.

This writes the source to a python file and runs pytest’s collection on the resulting module, returning all test items contained within.

getmodulecol(source, configargs=(), withinit=False)

Return the module collection node for source.

This writes source to a file using makepyfile() and then runs the pytest collection on it, returning the collection node for the test module.

Parameters:

• source – the source code of the module to collect
• configargs – any extra arguments to pass to parseconfigure()
• withinit – whether to also write an __init__.py file to the same directory to ensure it is a package

collect_by_name(modcol, name)

Return the collection node for name from the module collection.

This will search a module collection node for a collection node matching the given name.

Parameters:

• modcol – a module collection node; see getmodulecol()
• name – the name of the node to return

popen(cmdargs, stdout=-1, stderr=-1, stdin=<class 'object'>, **kw)

Invoke subprocess.Popen.

This calls subprocess.Popen making sure the current working directory is in the PYTHONPATH.

You probably want to use run() instead.

run(*cmdargs, timeout=None, stdin=<class 'object'>) → _pytest.pytester.RunResult

Run a command with arguments.

Run a process using subprocess.Popen saving the stdout and stderr.

Parameters:

• args – the sequence of arguments to pass to subprocess.Popen()
• timeout – the period in seconds after which to timeout and raise Testdir.TimeoutExpired
• stdin – optional standard input. Bytes are being send, closing the pipe, otherwise it is passed through to popen. Defaults to CLOSE_STDIN, which translates to using a pipe (subprocess.PIPE) that gets closed.

Returns a RunResult.

runpython(script) → _pytest.pytester.RunResult

Run a python script using sys.executable as interpreter.

Returns a RunResult.

runpython_c(command)

Run python -c “command”, return a RunResult.

runpytest_subprocess(*args, timeout=None) → _pytest.pytester.RunResult

Run pytest as a subprocess with given arguments.

Any plugins added to the plugins list will be added using the -p command line option. Additionally --basetemp is used to put any temporary files and directories in a numbered directory prefixed with “runpytest-” to not conflict with the normal numbered pytest location for temporary files and directories.

Parameters:

• args – the sequence of arguments to pass to the pytest subprocess
• timeout – the period in seconds after which to timeout and raise Testdir.TimeoutExpired

Returns a RunResult.

spawn_pytest(string: str, expect_timeout: float = 10.0) → pexpect.spawn

Run pytest using pexpect.

This makes sure to use the right pytest and sets up the temporary directory locations.

The pexpect child is returned.

spawn(cmd: str, expect_timeout: float = 10.0) → pexpect.spawn

Run a command using pexpect.

The pexpect child is returned.

class RunResult

The result of running a command.

Attributes:

Variables:

• ret – the return value
• outlines – list of lines captured from stdout
• errlines – list of lines captured from stderr
• stdout – LineMatcher of stdout, use stdout.str() to reconstruct stdout or the commonly used stdout.fnmatch_lines() method
• stderr – LineMatcher of stderr
• duration – duration in seconds

parseoutcomes() → Dict[str, int]

Return a dictionary of outcomestring->num from parsing the terminal output that the test process produced.

assert_outcomes(passed: int = 0, skipped: int = 0, failed: int = 0, error: int = 0, xpassed: int = 0, xfailed: int = 0) → None

Assert that the specified outcomes appear with the respective numbers (0 means it didn’t occur) in the text output from a test run.

class LineMatcher

Flexible matching of text.

This is a convenience class to test large texts like the output of commands.

The constructor takes a list of lines without their trailing newlines, i.e. text.splitlines().

str()

Return the entire original text.

fnmatch_lines_random(lines2)

Check lines exist in the output using in any order.

Lines are checked using fnmatch.fnmatch. The argument is a list of lines which have to occur in the output, in any order.

re_match_lines_random(lines2)

Check lines exist in the output using re.match, in any order.

The argument is a list of lines which have to occur in the output, in any order.

get_lines_after(fnline)

Return all lines following the given line in the text.

The given line can contain glob wildcards.

fnmatch_lines(lines2)

Search captured text for matching lines using fnmatch.fnmatch.

The argument is a list of lines which have to match and can use glob wildcards. If they do not match a pytest.fail() is called. The matches and non-matches are also shown as part of the error message.

re_match_lines(lines2)

Search captured text for matching lines using re.match.

The argument is a list of lines which have to match using re.match. If they do not match a pytest.fail() is called.

The matches and non-matches are also shown as part of the error message.

no_fnmatch_line(pat)

Ensure captured lines do not match the given pattern, using fnmatch.fnmatch.

Parameters:pat (str) – the pattern to match lines.

no_re_match_line(pat)

Ensure captured lines do not match the given pattern, using re.match.

Parameters:pat (str) – the regular expression to match lines.

recwarn

Tutorial: Asserting warnings with the warns function

recwarn()

Return a WarningsRecorder instance that records all warnings emitted by test functions.

See http://docs.python.org/library/warnings.html for information on warning categories.

class WarningsRecorder

A context manager to record raised warnings.

Adapted from warnings.catch_warnings.

list

The list of recorded warnings.

pop(cls: Type[Warning] = <class 'Warning'>) → warnings._Record

Pop the first recorded warning, raise exception if not exists.

clear() → None

Clear the list of recorded warnings.

Each recorded warning is an instance of warnings.WarningMessage.

Note

RecordedWarning was changed from a plain class to a namedtuple in pytest 3.1

Note

DeprecationWarning and PendingDeprecationWarning are treated differently; see Ensuring code triggers a deprecation warning.

tmp_path

Tutorial: Temporary directories and files

tmp_path()

Return a temporary directory path object which is unique to each test function invocation, created as a sub directory of the base temporary directory. The returned object is a pathlib.Path object.

Note

in python < 3.6 this is a pathlib2.Path

tmp_path_factory

Tutorial: The tmp_path_factory fixture

tmp_path_factory instances have the following methods:

TempPathFactory.mktemp(basename: str, numbered: bool = True) → pathlib.Path

makes a temporary directory managed by the factory

TempPathFactory.getbasetemp() → pathlib.Path

return base temporary directory.

tmpdir

Tutorial: Temporary directories and files

tmpdir()

Return a temporary directory path object which is unique to each test function invocation, created as a sub directory of the base temporary directory. The returned object is a py.path.local path object.

tmpdir_factory

Tutorial: The ‘tmpdir_factory’ fixture

tmpdir_factory instances have the following methods:

TempdirFactory.mktemp(basename: str, numbered: bool = True)

Create a subdirectory of the base temporary directory and return it. If numbered, ensure the directory is unique by adding a number prefix greater than any existing one.

TempdirFactory.getbasetemp()

backward compat wrapper for _tmppath_factory.getbasetemp

Hooks

Tutorial: Writing plugins.

Reference to all hooks which can be implemented by conftest.py files and plugins.

Bootstrapping hooks

Bootstrapping hooks called for plugins registered early enough (internal and setuptools plugins).

pytest_load_initial_conftests(early_config, parser, args)

implements the loading of initial conftest files ahead of command line option parsing.

Note

This hook will not be called for conftest.py files, only for setuptools plugins.

Parameters:

• early_config (_pytest.config.Config) – pytest config object
• args (list[str]) – list of arguments passed on the command line
• parser (_pytest.config.argparsing.Parser) – to add command line options

pytest_cmdline_preparse(config, args)

(Deprecated) modify command line arguments before option parsing.

This hook is considered deprecated and will be removed in a future pytest version. Consider using pytest_load_initial_conftests() instead.

Note

This hook will not be called for conftest.py files, only for setuptools plugins.

Parameters:

• config (_pytest.config.Config) – pytest config object
• args (list[str]) – list of arguments passed on the command line

pytest_cmdline_parse(pluginmanager, args)

return initialized config object, parsing the specified args.

Stops at first non-None result, see firstresult: stop at first non-None result

Note

This hook will only be called for plugin classes passed to the plugins arg when using pytest.main to perform an in-process test run.

Parameters:

• pluginmanager (_pytest.config.PytestPluginManager) – pytest plugin manager
• args (list[str]) – list of arguments passed on the command line

pytest_cmdline_main(config)

called for performing the main command line action. The default implementation will invoke the configure hooks and runtest_mainloop.

Note

This hook will not be called for conftest.py files, only for setuptools plugins.

Stops at first non-None result, see firstresult: stop at first non-None result

Parameters:config (_pytest.config.Config) – pytest config object

Initialization hooks

Initialization hooks called for plugins and conftest.py files.

pytest_addoption(parser, pluginmanager)

register argparse-style options and ini-style config values, called once at the beginning of a test run.

Note

This function should be implemented only in plugins or conftest.py files situated at the tests root directory due to how pytest discovers plugins during startup.

Parameters:

• parser (_pytest.config.argparsing.Parser) – To add command line options, call parser.addoption(...). To add ini-file values call parser.addini(...).
• pluginmanager (_pytest.config.PytestPluginManager) – pytest plugin manager, which can be used to install hookspec()’s or hookimpl()’s and allow one plugin to call another plugin’s hooks to change how command line options are added.

Options can later be accessed through the config object, respectively:

• config.getoption(name) to retrieve the value of a command line option.
• config.getini(name) to retrieve a value read from an ini-style file.

The config object is passed around on many internal objects via the .config attribute or can be retrieved as the pytestconfig fixture.

Note

This hook is incompatible with hookwrapper=True.

pytest_addhooks(pluginmanager)

called at plugin registration time to allow adding new hooks via a call to pluginmanager.add_hookspecs(module_or_class, prefix).

Parameters:pluginmanager (_pytest.config.PytestPluginManager) – pytest plugin manager

Note

This hook is incompatible with hookwrapper=True.

pytest_configure(config)

Allows plugins and conftest files to perform initial configuration.

This hook is called for every plugin and initial conftest file after command line options have been parsed.

After that, the hook is called for other conftest files as they are imported.

Note

This hook is incompatible with hookwrapper=True.

Parameters:config (_pytest.config.Config) – pytest config object

pytest_unconfigure(config)

called before test process is exited.

Parameters:config (_pytest.config.Config) – pytest config object

pytest_sessionstart(session)

called after the Session object has been created and before performing collection and entering the run test loop.

Parameters:session (_pytest.main.Session) – the pytest session object

pytest_sessionfinish(session, exitstatus)

called after whole test run finished, right before returning the exit status to the system.

Parameters:

• session (_pytest.main.Session) – the pytest session object
• exitstatus (int) – the status which pytest will return to the system

pytest_plugin_registered(plugin, manager)

a new pytest plugin got registered.

Parameters:

• plugin – the plugin module or instance
• manager (_pytest.config.PytestPluginManager) – pytest plugin manager

Note

This hook is incompatible with hookwrapper=True.

Test running hooks

All runtest related hooks receive a pytest.Item object.

pytest_runtestloop(session)

called for performing the main runtest loop (after collection finished).

Stops at first non-None result, see firstresult: stop at first non-None result

Parameters:session (_pytest.main.Session) – the pytest session object

pytest_runtest_protocol(item, nextitem)

implements the runtest_setup/call/teardown protocol for the given test item, including capturing exceptions and calling reporting hooks.

Parameters:

• item – test item for which the runtest protocol is performed.
• nextitem – the scheduled-to-be-next test item (or None if this is the end my friend). This argument is passed on to pytest_runtest_teardown().

Return boolean:

True if no further hook implementations should be invoked.

Stops at first non-None result, see firstresult: stop at first non-None result

pytest_runtest_logstart(nodeid, location)

signal the start of running a single test item.

This hook will be called before pytest_runtest_setup(), pytest_runtest_call() and pytest_runtest_teardown() hooks.

Parameters:

• nodeid (str) – full id of the item
• location – a triple of (filename, linenum, testname)

pytest_runtest_logfinish(nodeid, location)

signal the complete finish of running a single test item.

This hook will be called after pytest_runtest_setup(), pytest_runtest_call() and pytest_runtest_teardown() hooks.

Parameters:

• nodeid (str) – full id of the item
• location – a triple of (filename, linenum, testname)

pytest_runtest_setup(item)

called before pytest_runtest_call(item).

pytest_runtest_call(item)

called to execute the test item.

pytest_runtest_teardown(item, nextitem)

called after pytest_runtest_call.

Parameters:nextitem – the scheduled-to-be-next test item (None if no further test item is scheduled). This argument can be used to perform exact teardowns, i.e. calling just enough finalizers so that nextitem only needs to call setup-functions.

pytest_runtest_makereport(item, call)

return a _pytest.runner.TestReport object for the given pytest.Item and _pytest.runner.CallInfo.

Stops at first non-None result, see firstresult: stop at first non-None result

For deeper understanding you may look at the default implementation of these hooks in _pytest.runner and maybe also in _pytest.pdb which interacts with _pytest.capture and its input/output capturing in order to immediately drop into interactive debugging when a test failure occurs.

The _pytest.terminal reported specifically uses the reporting hook to print information about a test run.

pytest_pyfunc_call(pyfuncitem)

call underlying test function.

Stops at first non-None result, see firstresult: stop at first non-None result

Collection hooks

pytest calls the following hooks for collecting files and directories:

pytest_collection(session)

Perform the collection protocol for the given session.

Stops at first non-None result, see firstresult: stop at first non-None result.

Parameters:session (_pytest.main.Session) – the pytest session object

pytest_ignore_collect(path, config)

return True to prevent considering this path for collection. This hook is consulted for all files and directories prior to calling more specific hooks.

Stops at first non-None result, see firstresult: stop at first non-None result

Parameters:

• path – a py.path.local - the path to analyze
• config (_pytest.config.Config) – pytest config object

pytest_collect_directory(path, parent)

called before traversing a directory for collection files.

Stops at first non-None result, see firstresult: stop at first non-None result

Parameters:path – a py.path.local - the path to analyze

pytest_collect_file(path, parent)

return collection Node or None for the given path. Any new node needs to have the specified parent as a parent.

Parameters:path – a py.path.local - the path to collect

pytest_pycollect_makemodule(path, parent)

return a Module collector or None for the given path. This hook will be called for each matching test module path. The pytest_collect_file hook needs to be used if you want to create test modules for files that do not match as a test module.

Stops at first non-None result, see firstresult: stop at first non-None result

Parameters:path – a py.path.local - the path of module to collect

For influencing the collection of objects in Python modules you can use the following hook:

pytest_pycollect_makeitem(collector, name, obj)

return custom item/collector for a python object in a module, or None.

Stops at first non-None result, see firstresult: stop at first non-None result

pytest_generate_tests(metafunc)

generate (multiple) parametrized calls to a test function.

pytest_make_parametrize_id(config, val, argname)

Return a user-friendly string representation of the given val that will be used by @pytest.mark.parametrize calls. Return None if the hook doesn’t know about val. The parameter name is available as argname, if required.

Stops at first non-None result, see firstresult: stop at first non-None result

Parameters:

• config (_pytest.config.Config) – pytest config object
• val – the parametrized value
• argname (str) – the automatic parameter name produced by pytest

After collection is complete, you can modify the order of items, delete or otherwise amend the test items:

pytest_collection_modifyitems(session, config, items)

called after collection has been performed, may filter or re-order the items in-place.

Parameters:

• session (_pytest.main.Session) – the pytest session object
• config (_pytest.config.Config) – pytest config object
• items (List[_pytest.nodes.Item]) – list of item objects

pytest_collection_finish(session)

called after collection has been performed and modified.

Parameters:session (_pytest.main.Session) – the pytest session object

Reporting hooks

Session related reporting hooks:

pytest_collectstart(collector)

collector starts collecting.

pytest_make_collect_report(collector)

perform collector.collect() and return a CollectReport.

Stops at first non-None result, see firstresult: stop at first non-None result

pytest_itemcollected(item)

we just collected a test item.

pytest_collectreport(report)

collector finished collecting.

pytest_deselected(items)

called for test items deselected, e.g. by keyword.

pytest_report_header(config, startdir)

return a string or list of strings to be displayed as header info for terminal reporting.

Parameters:

• config (_pytest.config.Config) – pytest config object
• startdir – py.path object with the starting dir

Note

This function should be implemented only in plugins or conftest.py files situated at the tests root directory due to how pytest discovers plugins during startup.

pytest_report_collectionfinish(config, startdir, items)

New in version 3.2.

return a string or list of strings to be displayed after collection has finished successfully.

This strings will be displayed after the standard “collected X items” message.

Parameters:

• config (_pytest.config.Config) – pytest config object
• startdir – py.path object with the starting dir
• items – list of pytest items that are going to be executed; this list should not be modified.

pytest_report_teststatus(report, config)

return result-category, shortletter and verbose word for reporting.

Parameters:config (_pytest.config.Config) – pytest config object

Stops at first non-None result, see firstresult: stop at first non-None result

pytest_terminal_summary(terminalreporter, exitstatus, config)

Add a section to terminal summary reporting.

Parameters:

• terminalreporter (_pytest.terminal.TerminalReporter) – the internal terminal reporter object
• exitstatus (int) – the exit status that will be reported back to the OS
• config (_pytest.config.Config) – pytest config object

New in version 4.2: The config parameter.

pytest_fixture_setup(fixturedef, request)

performs fixture setup execution.

Returns:The return value of the call to the fixture function

Stops at first non-None result, see firstresult: stop at first non-None result

Note

If the fixture function returns None, other implementations of this hook function will continue to be called, according to the behavior of the firstresult: stop at first non-None result option.

pytest_fixture_post_finalizer(fixturedef, request)

called after fixture teardown, but before the cache is cleared so the fixture result cache fixturedef.cached_result can still be accessed.

pytest_warning_captured(warning_message, when, item)

Process a warning captured by the internal pytest warnings plugin.

Parameters:

• warning_message (warnings.WarningMessage) – The captured warning. This is the same object produced by warnings.catch_warnings(), and contains the same attributes as the parameters of warnings.showwarning().
• when (str) –

  Indicates when the warning was captured. Possible values:

  • "config": during pytest configuration/initialization stage.
  • "collect": during test collection.
  • "runtest": during test execution.
• item (pytest.Item|None) –

  DEPRECATED: This parameter is incompatible with pytest-xdist, and will always receive None in a future release.

  The item being executed if when is "runtest", otherwise None.

Central hook for reporting about test execution:

pytest_runtest_logreport(report)

process a test setup/call/teardown report relating to the respective phase of executing a test.

Assertion related hooks:

pytest_assertrepr_compare(config, op, left, right)

return explanation for comparisons in failing assert expressions.

Return None for no custom explanation, otherwise return a list of strings. The strings will be joined by newlines but any newlines in a string will be escaped. Note that all but the first line will be indented slightly, the intention is for the first line to be a summary.

Parameters:config (_pytest.config.Config) – pytest config object

pytest_assertion_pass(item, lineno, orig, expl)

(Experimental)

New in version 5.0.

Hook called whenever an assertion passes.

Use this hook to do some processing after a passing assertion. The original assertion information is available in the orig string and the pytest introspected assertion information is available in the expl string.

This hook must be explicitly enabled by the enable_assertion_pass_hook ini-file option:

[pytest]
enable_assertion_pass_hook=true

You need to clean the .pyc files in your project directory and interpreter libraries when enabling this option, as assertions will require to be re-written.

Parameters:

• item (_pytest.nodes.Item) – pytest item object of current test
• lineno (int) – line number of the assert statement
• orig (string) – string with original assertion
• expl (string) – string with assert explanation

Note

This hook is experimental, so its parameters or even the hook itself might be changed/removed without warning in any future pytest release.

If you find this hook useful, please share your feedback opening an issue.

Debugging/Interaction hooks

There are few hooks which can be used for special reporting or interaction with exceptions:

pytest_internalerror(excrepr, excinfo)

called for internal errors.

pytest_keyboard_interrupt(excinfo)

called for keyboard interrupt.

pytest_exception_interact(node, call, report)

called when an exception was raised which can potentially be interactively handled.

This hook is only called if an exception was raised that is not an internal exception like skip.Exception.

pytest_enter_pdb(config, pdb)

called upon pdb.set_trace(), can be used by plugins to take special action just before the python debugger enters in interactive mode.

Parameters:

• config (_pytest.config.Config) – pytest config object
• pdb (pdb.Pdb) – Pdb instance

Objects

Full reference to objects accessible from fixtures or hooks.

CallInfo

class CallInfo

Result/Exception info a function invocation.

Class

class Class

Bases: _pytest.python.PyCollector

Collector for test methods.

collect()

returns a list of children (items and collectors) for this collection node.

Collector

class Collector

Bases: _pytest.nodes.Node

Collector instances create children through collect() and thus iteratively build a tree.

exception CollectError

Bases: Exception

an error during collection, contains a custom message.

collect()

returns a list of children (items and collectors) for this collection node.

repr_failure(excinfo)

represent a collection failure.

Config

class Config

Access to configuration values, pluginmanager and plugin hooks.

Variables:

• pluginmanager (PytestPluginManager) – the plugin manager handles plugin registration and hook invocation.
• option (argparse.Namespace) – access to command line option as attributes.
• invocation_params (InvocationParams) –

  Object containing the parameters regarding the pytest.main invocation.

  Contains the following read-only attributes:

  • args: tuple of command-line arguments as passed to pytest.main().
  • plugins: list of extra plugins, might be None.
  • dir: directory where pytest.main() was invoked from.

class InvocationParams(args, plugins, dir: pathlib.Path)

Holds parameters passed during pytest.main()

New in version 5.1.

Note

Note that the environment variable PYTEST_ADDOPTS and the addopts ini option are handled by pytest, not being included in the args attribute.

Plugins accessing InvocationParams must be aware of that.

invocation_dir

Backward compatibility

add_cleanup(func)

Add a function to be called when the config object gets out of use (usually coninciding with pytest_unconfigure).

classmethod fromdictargs(option_dict, args)

constructor usable for subprocesses.

addinivalue_line(name, line)

add a line to an ini-file option. The option must have been declared but might not yet be set in which case the line becomes the the first line in its value.

getini(name: str)

return configuration value from an ini file. If the specified name hasn’t been registered through a prior parser.addini call (usually from a plugin), a ValueError is raised.

getoption(name: str, default=<NOTSET>, skip: bool = False)

return command line option value.

Parameters:

• name – name of the option. You may also specify the literal --OPT option instead of the “dest” option name.
• default – default value if no option of that name exists.
• skip – if True raise pytest.skip if option does not exists or has a None value.

getvalue(name, path=None)

(deprecated, use getoption())

getvalueorskip(name, path=None)

(deprecated, use getoption(skip=True))

ExceptionInfo

class ExceptionInfo(excinfo: Optional[Tuple[Type[_E], _E, traceback]], striptext: str = '', traceback: Optional[_pytest._code.code.Traceback] = None)

wraps sys.exc_info() objects and offers help for navigating the traceback.

classmethod from_exc_info(exc_info: Tuple[Type[_E], _E, traceback], exprinfo: Optional[str] = None) → ExceptionInfo[_E]

returns an ExceptionInfo for an existing exc_info tuple.

Warning

Experimental API

Parameters:exprinfo – a text string helping to determine if we should strip AssertionError from the output, defaults to the exception message/__str__()

classmethod from_current(exprinfo: Optional[str] = None) → _pytest._code.code.ExceptionInfo[BaseException][BaseException]

returns an ExceptionInfo matching the current traceback

Warning

Experimental API

Parameters:exprinfo – a text string helping to determine if we should strip AssertionError from the output, defaults to the exception message/__str__()

classmethod for_later() → _pytest._code.code.ExceptionInfo[~_E][_E]

return an unfilled ExceptionInfo

fill_unfilled(exc_info: Tuple[Type[_E], _E, traceback]) → None

fill an unfilled ExceptionInfo created with for_later()

type

the exception class

value

the exception value

tb

the exception raw traceback

typename

the type name of the exception

traceback

the traceback

exconly(tryshort: bool = False) → str

return the exception as a string

when ‘tryshort’ resolves to True, and the exception is a _pytest._code._AssertionError, only the actual exception part of the exception representation is returned (so ‘AssertionError: ‘ is removed from the beginning)

errisinstance(exc: Union[Type[BaseException], Tuple[Type[BaseException], ...]]) → bool

return True if the exception is an instance of exc

getrepr(showlocals: bool = False, style: _TracebackStyle = 'long', abspath: bool = False, tbfilter: bool = True, funcargs: bool = False, truncate_locals: bool = True, chain: bool = True) → Union[ReprExceptionInfo, ExceptionChainRepr]

Return str()able representation of this exception info.

Parameters:

• showlocals (bool) – Show locals per traceback entry. Ignored if style=="native".
• style (str) – long|short|no|native traceback style
• abspath (bool) – If paths should be changed to absolute or left unchanged.
• tbfilter (bool) – Hide entries that contain a local variable __tracebackhide__==True. Ignored if style=="native".
• funcargs (bool) – Show fixtures (“funcargs” for legacy purposes) per traceback entry.
• truncate_locals (bool) – With showlocals==True, make sure locals can be safely represented as strings.
• chain (bool) – if chained exceptions in Python 3 should be shown.

Changed in version 3.9: Added the chain parameter.

match(regexp: Union[str, Pattern[AnyStr]]) → bool

Check whether the regular expression ‘regexp’ is found in the string representation of the exception using re.search. If it matches then True is returned (so that it is possible to write assert excinfo.match()). If it doesn’t match an AssertionError is raised.

pytest.ExitCode

class ExitCode

New in version 5.0.

Encodes the valid exit codes by pytest.

Currently users and plugins may supply other exit codes as well.

OK = 0

tests passed

TESTS_FAILED = 1

tests failed

INTERRUPTED = 2

pytest was interrupted

INTERNAL_ERROR = 3

an internal error got in the way

USAGE_ERROR = 4

pytest was misused

NO_TESTS_COLLECTED = 5

pytest couldn’t find tests

FixtureDef

class FixtureDef

Bases: object

A container for a factory definition.

FSCollector

class FSCollector

Bases: _pytest.nodes.Collector

Function

class Function

Bases: _pytest.python.FunctionMixin, _pytest.nodes.Item

a Function Item is responsible for setting up and executing a Python test function.

originalname = None

original function name, without any decorations (for example parametrization adds a "[...]" suffix to function names).

New in version 3.0.

function

underlying python ‘function’ object

funcargnames

alias attribute for fixturenames for pre-2.3 compatibility

runtest()

execute the underlying test function.

setup()

perform setup for this test function.

Item

class Item

Bases: _pytest.nodes.Node

a basic test invocation item. Note that for a single function there might be multiple test invocation items.

user_properties = None

user properties is a list of tuples (name, value) that holds user defined properties for this test.

add_report_section(when: str, key: str, content: str) → None

Adds a new report section, similar to what’s done internally to add stdout and stderr captured output:

item.add_report_section("call", "stdout", "report section contents")

Parameters:

• when (str) – One of the possible capture states, "setup", "call", "teardown".
• key (str) – Name of the section, can be customized at will. Pytest uses "stdout" and "stderr" internally.
• content (str) – The full contents as a string.

MarkDecorator

class MarkDecorator(mark)

A decorator for test functions and test classes. When applied it will create Mark objects which are often created like this:

mark1 = pytest.mark.NAME              # simple MarkDecorator
mark2 = pytest.mark.NAME(name1=value) # parametrized MarkDecorator

and can then be applied as decorators to test functions:

@mark2
def test_function():
    pass

When a MarkDecorator instance is called it does the following:

1. If called with a single class as its only positional argument and no additional keyword arguments, it attaches itself to the class so it gets applied automatically to all test cases found in that class.
2. If called with a single function as its only positional argument and no additional keyword arguments, it attaches a MarkInfo object to the function, containing all the arguments already stored internally in the MarkDecorator.
3. When called in any other case, it performs a ‘fake construction’ call, i.e. it returns a new MarkDecorator instance with the original MarkDecorator’s content updated with the arguments passed to this call.

Note: The rules above prevent MarkDecorator objects from storing only a single function or class reference as their positional argument with no additional keyword or positional arguments.

name

alias for mark.name

args

alias for mark.args

kwargs

alias for mark.kwargs

with_args(*args, **kwargs)

return a MarkDecorator with extra arguments added

unlike call this can be used even if the sole argument is a callable/class

Returns:MarkDecorator

MarkGenerator

class MarkGenerator

Factory for MarkDecorator objects - exposed as a pytest.mark singleton instance. Example:

import pytest
@pytest.mark.slowtest
def test_function():
   pass

will set a ‘slowtest’ MarkInfo object on the test_function object.

Mark

class Mark(name: str, args, kwargs)

name = None

name of the mark

args = None

positional arguments of the mark decorator

kwargs = None

keyword arguments of the mark decorator

combined_with(other)

Parameters:other (Mark) – the mark to combine with Return type:Mark

combines by appending args and merging the mappings

Metafunc

class Metafunc(definition: _pytest.python.FunctionDefinition, fixtureinfo, config, cls=None, module=None)

Metafunc objects are passed to the pytest_generate_tests hook. They help to inspect a test function and to generate tests according to test configuration or values specified in the class or module where a test function is defined.

config = None

access to the _pytest.config.Config object for the test session

module = None

the module object where the test function is defined in.

function = None

underlying python test function

fixturenames = None

set of fixture names required by the test function

cls = None

class object where the test function is defined in or None.

funcargnames

alias attribute for fixturenames for pre-2.3 compatibility

parametrize(argnames, argvalues, indirect=False, ids=None, scope=None)

Add new invocations to the underlying test function using the list of argvalues for the given argnames. Parametrization is performed during the collection phase. If you need to setup expensive resources see about setting indirect to do it rather at test setup time.

Parameters:

• argnames – a comma-separated string denoting one or more argument names, or a list/tuple of argument strings.
• argvalues – The list of argvalues determines how often a test is invoked with different argument values. If only one argname was specified argvalues is a list of values. If N argnames were specified, argvalues must be a list of N-tuples, where each tuple-element specifies a value for its respective argname.
• indirect – The list of argnames or boolean. A list of arguments’ names (subset of argnames). If True the list contains all names from the argnames. Each argvalue corresponding to an argname in this list will be passed as request.param to its respective argname fixture function so that it can perform more expensive setups during the setup phase of a test rather than at collection time.
• ids – list of string ids, or a callable. If strings, each is corresponding to the argvalues so that they are part of the test id. If None is given as id of specific test, the automatically generated id for that argument will be used. If callable, it should take one argument (a single argvalue) and return a string or return None. If None, the automatically generated id for that argument will be used. If no ids are provided they will be generated automatically from the argvalues.
• scope – if specified it denotes the scope of the parameters. The scope is used for grouping tests by parameter instances. It will also override any fixture-function defined scope, allowing to set a dynamic scope using test context or configuration.

Module

class Module

Bases: _pytest.nodes.File, _pytest.python.PyCollector

Collector for test classes and functions.

collect()

returns a list of children (items and collectors) for this collection node.

Node

class Node

base class for Collector and Item the test collection tree. Collector subclasses have children, Items are terminal nodes.

name = None

a unique name within the scope of the parent node

parent = None

the parent collector node.

fspath = None

filesystem path where this node was collected from (can be None)

keywords = None

keywords/markers collected from all scopes

own_markers = None

the marker objects belonging to this node

extra_keyword_matches = None

allow adding of extra keywords to use for matching

ihook

fspath sensitive hook proxy used to call pytest hooks

warn(warning)

Issue a warning for this item.

Warnings will be displayed after the test session, unless explicitly suppressed

Parameters:warning (Warning) – the warning instance to issue. Must be a subclass of PytestWarning. Raises:ValueError – if warning instance is not a subclass of PytestWarning.

Example usage:

node.warn(PytestWarning("some message"))

nodeid

a ::-separated string denoting its collection tree address.

listchain()

return list of all parent collectors up to self, starting from root of collection tree.

add_marker(marker: Union[str, _pytest.mark.structures.MarkDecorator], append: bool = True) → None

dynamically add a marker object to the node.

Parameters:marker (str or pytest.mark.* object) – append=True whether to append the marker, if False insert at position 0.

iter_markers(name=None)

Parameters:name – if given, filter the results by the name attribute

iterate over all markers of the node

for ... in iter_markers_with_node(name=None)

Parameters:name – if given, filter the results by the name attribute

iterate over all markers of the node returns sequence of tuples (node, mark)

get_closest_marker(name, default=None)

return the first marker matching the name, from closest (for example function) to farther level (for example module level).

Parameters:

• default – fallback return value of no marker was found
• name – name to filter by

listextrakeywords()

Return a set of all extra keywords in self and any parents.

addfinalizer(fin)

register a function to be called when this node is finalized.

This method can only be called when this node is active in a setup chain, for example during self.setup().

getparent(cls)

get the next parent node (including ourself) which is an instance of the given class

Parser

class Parser

Parser for command line arguments and ini-file values.

Variables:extra_info – dict of generic param -> value to display in case there’s an error processing the command line arguments.

getgroup(name, description='', after=None)

get (or create) a named option Group.

Name:name of the option group. Description:long description for –help output. After:name of other group, used for ordering –help output.

The returned group object has an addoption method with the same signature as parser.addoption but will be shown in the respective group in the output of pytest. --help.

addoption(*opts, **attrs)

register a command line option.

Opts:option names, can be short or long options. Attrs:same attributes which the add_option() function of the argparse library accepts.

After command line parsing options are available on the pytest config object via config.option.NAME where NAME is usually set by passing a dest attribute, for example addoption("--long", dest="NAME", ...).

parse_known_args(args, namespace=None) → argparse.Namespace

parses and returns a namespace object with known arguments at this point.

parse_known_and_unknown_args(args, namespace=None) → Tuple[argparse.Namespace, List[str]]

parses and returns a namespace object with known arguments, and the remaining arguments unknown at this point.

addini(name, help, type=None, default=None)

register an ini-file option.

Name:name of the ini-variable Type:type of the variable, can be pathlist, args, linelist or bool. Default:default value if no ini-file option exists but is queried.

The value of ini-variables can be retrieved via a call to config.getini(name).

PluginManager

class PluginManager

Core PluginManager class which manages registration of plugin objects and 1:N hook calling.

You can register new hooks by calling add_hookspecs(module_or_class). You can register plugin objects (which contain hooks) by calling register(plugin). The PluginManager is initialized with a prefix that is searched for in the names of the dict of registered plugin objects.

For debugging purposes you can call PluginManager.enable_tracing() which will subsequently send debug information to the trace helper.

register(plugin, name=None)

Register a plugin and return its canonical name or None if the name is blocked from registering. Raise a ValueError if the plugin is already registered.

unregister(plugin=None, name=None)

unregister a plugin object and all its contained hook implementations from internal data structures.

set_blocked(name)

block registrations of the given name, unregister if already registered.

is_blocked(name)

return True if the given plugin name is blocked.

add_hookspecs(module_or_class)

add new hook specifications defined in the given module_or_class. Functions are recognized if they have been decorated accordingly.

get_plugins()

return the set of registered plugins.

is_registered(plugin)

Return True if the plugin is already registered.

get_canonical_name(plugin)

Return canonical name for a plugin object. Note that a plugin may be registered under a different name which was specified by the caller of register(plugin, name). To obtain the name of an registered plugin use get_name(plugin) instead.

get_plugin(name)

Return a plugin or None for the given name.

has_plugin(name)

Return True if a plugin with the given name is registered.

get_name(plugin)

Return name for registered plugin or None if not registered.

check_pending()

Verify that all hooks which have not been verified against a hook specification are optional, otherwise raise PluginValidationError.

load_setuptools_entrypoints(group, name=None)

Load modules from querying the specified setuptools group.

Parameters:

• group (str) – entry point group to load plugins
• name (str) – if given, loads only plugins with the given name.

Return type:

int

Returns:

return the number of loaded plugins by this call.

list_plugin_distinfo()

return list of distinfo/plugin tuples for all setuptools registered plugins.

list_name_plugin()

return list of name/plugin pairs.

get_hookcallers(plugin)

get all hook callers for the specified plugin.

add_hookcall_monitoring(before, after)

add before/after tracing functions for all hooks and return an undo function which, when called, will remove the added tracers.

before(hook_name, hook_impls, kwargs) will be called ahead of all hook calls and receive a hookcaller instance, a list of HookImpl instances and the keyword arguments for the hook call.

after(outcome, hook_name, hook_impls, kwargs) receives the same arguments as before but also a pluggy.callers._Result object which represents the result of the overall hook call.

enable_tracing()

enable tracing of hook calls and return an undo function.

subset_hook_caller(name, remove_plugins)

Return a new hooks._HookCaller instance for the named method which manages calls to all registered plugins except the ones from remove_plugins.

PytestPluginManager

class PytestPluginManager

Bases: pluggy.manager.PluginManager

Overwrites pluggy.PluginManager to add pytest-specific functionality:

• loading plugins from the command line, PYTEST_PLUGINS env variable and pytest_plugins global variables found in plugins being loaded;
• conftest.py loading during start-up;

parse_hookimpl_opts(plugin, name)

parse_hookspec_opts(module_or_class, name)

register(plugin, name=None)

Register a plugin and return its canonical name or None if the name is blocked from registering. Raise a ValueError if the plugin is already registered.

getplugin(name)

hasplugin(name)

Return True if the plugin with the given name is registered.

pytest_configure(config)

consider_preparse(args)

consider_pluginarg(arg)

consider_conftest(conftestmodule)

consider_env()

consider_module(mod)

import_plugin(modname, consider_entry_points=False)

Imports a plugin with modname. If consider_entry_points is True, entry point names are also considered to find a plugin.

Session

class Session

Bases: _pytest.nodes.FSCollector

exception Interrupted

Bases: KeyboardInterrupt

signals an interrupted test run.

exception Failed

Bases: Exception

signals a stop as failed test run.

for ... in collect()

returns a list of children (items and collectors) for this collection node.

TestReport

class TestReport

Basic test report object (also used for setup and teardown calls if they fail).

nodeid = None

normalized collection node id

location = None

a (filesystempath, lineno, domaininfo) tuple indicating the actual location of a test item - it might be different from the collected one e.g. if a method is inherited from a different module.

keywords = None

a name -> value dictionary containing all keywords and markers associated with a test invocation.

outcome = None

test outcome, always one of “passed”, “failed”, “skipped”.

longrepr = None

None or a failure representation.

when = None

one of ‘setup’, ‘call’, ‘teardown’ to indicate runtest phase.

user_properties = None

user properties is a list of tuples (name, value) that holds user defined properties of the test

sections = []

list of pairs (str, str) of extra information which needs to marshallable. Used by pytest to add captured text from stdout and stderr, but may be used by other plugins to add arbitrary information to reports.

duration = None

time it took to run just the test

classmethod from_item_and_call(item, call) → _pytest.reports.TestReport

Factory method to create and fill a TestReport with standard item and call info.

caplog

Return captured log lines, if log capturing is enabled

New in version 3.5.

capstderr

Return captured text from stderr, if capturing is enabled

New in version 3.0.

capstdout

Return captured text from stdout, if capturing is enabled

New in version 3.0.

count_towards_summary

Experimental

Returns True if this report should be counted towards the totals shown at the end of the test session: “1 passed, 1 failure, etc”.

Note

This function is considered experimental, so beware that it is subject to changes even in patch releases.

head_line

Experimental

Returns the head line shown with longrepr output for this report, more commonly during traceback representation during failures:

________ Test.foo ________

In the example above, the head_line is “Test.foo”.

Note

This function is considered experimental, so beware that it is subject to changes even in patch releases.

longreprtext

Read-only property that returns the full string representation of longrepr.

New in version 3.0.

_Result

class _Result(result, excinfo)

result

Get the result(s) for this hook call (DEPRECATED in favor of get_result()).

force_result(result)

Force the result(s) to result.

If the hook was marked as a firstresult a single value should be set otherwise set a (modified) list of results. Any exceptions found during invocation will be deleted.

get_result()

Get the result(s) for this hook call.

If the hook was marked as a firstresult only a single value will be returned otherwise a list of results.

Special Variables

pytest treats some global variables in a special manner when defined in a test module.

collect_ignore

Tutorial: Customizing test collection

Can be declared in conftest.py files to exclude test directories or modules. Needs to be list[str].

collect_ignore = ["setup.py"]

collect_ignore_glob

Tutorial: Customizing test collection

Can be declared in conftest.py files to exclude test directories or modules with Unix shell-style wildcards. Needs to be list[str] where str can contain glob patterns.

collect_ignore_glob = ["*_ignore.py"]

pytest_plugins

Tutorial: Requiring/Loading plugins in a test module or conftest file

Can be declared at the global level in test modules and conftest.py files to register additional plugins. Can be either a str or Sequence[str].

pytest_plugins = "myapp.testsupport.myplugin"

pytest_plugins = ("myapp.testsupport.tools", "myapp.testsupport.regression")

pytest_mark

Tutorial: Marking whole classes or modules

Can be declared at the global level in test modules to apply one or more marks to all test functions and methods. Can be either a single mark or a list of marks.

import pytest

pytestmark = pytest.mark.webtest

import pytest

pytestmark = [pytest.mark.integration, pytest.mark.slow]

PYTEST_DONT_REWRITE (module docstring)

The text PYTEST_DONT_REWRITE can be add to any module docstring to disable assertion rewriting for that module.

Environment Variables

Environment variables that can be used to change pytest’s behavior.

PYTEST_ADDOPTS

This contains a command-line (parsed by the py:mod:shlex module) that will be prepended to the command line given by the user, see How to change command line options defaults for more information.

PYTEST_DEBUG

When set, pytest will print tracing and debug information.

PYTEST_PLUGINS

Contains comma-separated list of modules that should be loaded as plugins:

export PYTEST_PLUGINS=mymodule.plugin,xdist

PYTEST_DISABLE_PLUGIN_AUTOLOAD

When set, disables plugin auto-loading through setuptools entrypoints. Only explicitly specified plugins will be loaded.

PYTEST_CURRENT_TEST

This is not meant to be set by users, but is set by pytest internally with the name of the current test so other processes can inspect it, see PYTEST_CURRENT_TEST environment variable for more information.

Exceptions

UsageError

class UsageError

error in pytest usage or invocation

Configuration Options

Here is a list of builtin configuration options that may be written in a pytest.ini, tox.ini or setup.cfg file, usually located at the root of your repository. All options must be under a [pytest] section ([tool:pytest] for setup.cfg files).

Warning

Usage of setup.cfg is not recommended unless for very simple use cases. .cfg files use a different parser than pytest.ini and tox.ini which might cause hard to track down problems. When possible, it is recommended to use the latter files to hold your pytest configuration.

Configuration file options may be overwritten in the command-line by using -o/--override, which can also be passed multiple times. The expected format is name=value. For example:

pytest -o console_output_style=classic -o cache_dir=/tmp/mycache

addopts

Add the specified OPTS to the set of command line arguments as if they had been specified by the user. Example: if you have this ini file content:

# content of pytest.ini
[pytest]
addopts = --maxfail=2 -rf  # exit after 2 failures, report fail info

issuing pytest test_hello.py actually means:

pytest --maxfail=2 -rf test_hello.py

Default is to add no options.

cache_dir

Sets a directory where stores content of cache plugin. Default directory is .pytest_cache which is created in rootdir. Directory may be relative or absolute path. If setting relative path, then directory is created relative to rootdir. Additionally path may contain environment variables, that will be expanded. For more information about cache plugin please refer to Cache: working with cross-testrun state.

confcutdir

Sets a directory where search upwards for conftest.py files stops. By default, pytest will stop searching for conftest.py files upwards from pytest.ini/tox.ini/setup.cfg of the project if any, or up to the file-system root.

console_output_style

Sets the console output style while running tests:

• classic: classic pytest output.
• progress: like classic pytest output, but with a progress indicator.
• count: like progress, but shows progress as the number of tests completed instead of a percent.

The default is progress, but you can fallback to classic if you prefer or the new mode is causing unexpected problems:

# content of pytest.ini
[pytest]
console_output_style = classic

doctest_encoding

Default encoding to use to decode text files with docstrings. See how pytest handles doctests.

doctest_optionflags

One or more doctest flag names from the standard doctest module. See how pytest handles doctests.

empty_parameter_set_mark

Allows to pick the action for empty parametersets in parameterization

• skip skips tests with an empty parameterset (default)
• xfail marks tests with an empty parameterset as xfail(run=False)
• fail_at_collect raises an exception if parametrize collects an empty parameter set

# content of pytest.ini
[pytest]
empty_parameter_set_mark = xfail

Note

The default value of this option is planned to change to xfail in future releases as this is considered less error prone, see #3155 for more details.

faulthandler_timeout

Dumps the tracebacks of all threads if a test takes longer than X seconds to run (including fixture setup and teardown). Implemented using the faulthandler.dump_traceback_later function, so all caveats there apply.

# content of pytest.ini
[pytest]
faulthandler_timeout=5

For more information please refer to Fault Handler.

filterwarnings

Sets a list of filters and actions that should be taken for matched warnings. By default all warnings emitted during the test session will be displayed in a summary at the end of the test session.

# content of pytest.ini
[pytest]
filterwarnings =
    error
    ignore::DeprecationWarning

This tells pytest to ignore deprecation warnings and turn all other warnings into errors. For more information please refer to Warnings Capture.

junit_duration_report

New in version 4.1.

Configures how durations are recorded into the JUnit XML report:

• total (the default): duration times reported include setup, call, and teardown times.
• call: duration times reported include only call times, excluding setup and teardown.

[pytest]
junit_duration_report = call

junit_family

New in version 4.2.

Configures the format of the generated JUnit XML file. The possible options are:

• xunit1 (or legacy): produces old style output, compatible with the xunit 1.0 format. This is the default.

• xunit2: produces xunit 2.0 style output,

  which should be more compatible with latest Jenkins versions.

[pytest]
junit_family = xunit2

junit_logging

New in version 3.5.

Configures if stdout/stderr should be written to the JUnit XML file. Valid values are system-out, system-err, and no (the default).

[pytest]
junit_logging = system-out

junit_log_passing_tests

New in version 4.6.

If junit_logging != "no", configures if the captured output should be written to the JUnit XML file for passing tests. Default is True.

[pytest]
junit_log_passing_tests = False

junit_suite_name

To set the name of the root test suite xml item, you can configure the junit_suite_name option in your config file:

[pytest]
junit_suite_name = my_suite

log_auto_indent

Allow selective auto-indentation of multiline log messages.

Supports command line option --log-auto-indent [value] and config option log_auto_indent = [value] to set the auto-indentation behavior for all logging.

[value] can be:

• True or “On” - Dynamically auto-indent multiline log messages
• False or “Off” or 0 - Do not auto-indent multiline log messages (the default behavior)
• [positive integer] - auto-indent multiline log messages by [value] spaces

[pytest]
log_auto_indent = False

Supports passing kwarg extra={"auto_indent": [value]} to calls to logging.log() to specify auto-indentation behavior for a specific entry in the log. extra kwarg overrides the value specified on the command line or in the config.

log_cli

Enable log display during test run (also known as “live logging”). The default is False.

[pytest]
log_cli = True

log_cli_date_format

Sets a time.strftime()-compatible string that will be used when formatting dates for live logging.

[pytest]
log_cli_date_format = %Y-%m-%d %H:%M:%S

For more information, see Live Logs.

log_cli_format

Sets a logging-compatible string used to format live logging messages.

[pytest]
log_cli_format = %(asctime)s %(levelname)s %(message)s

For more information, see Live Logs.

log_cli_level

Sets the minimum log message level that should be captured for live logging. The integer value or the names of the levels can be used.

[pytest]
log_cli_level = INFO

For more information, see Live Logs.

log_date_format

Sets a time.strftime()-compatible string that will be used when formatting dates for logging capture.

[pytest]
log_date_format = %Y-%m-%d %H:%M:%S

For more information, see Logging.

log_file

Sets a file name relative to the pytest.ini file where log messages should be written to, in addition to the other logging facilities that are active.

[pytest]
log_file = logs/pytest-logs.txt

For more information, see Logging.

log_file_date_format

Sets a time.strftime()-compatible string that will be used when formatting dates for the logging file.

[pytest]
log_file_date_format = %Y-%m-%d %H:%M:%S

For more information, see Logging.

log_file_format

Sets a logging-compatible string used to format logging messages redirected to the logging file.

[pytest]
log_file_format = %(asctime)s %(levelname)s %(message)s

For more information, see Logging.

log_file_level

Sets the minimum log message level that should be captured for the logging file. The integer value or the names of the levels can be used.

[pytest]
log_file_level = INFO

For more information, see Logging.

log_format

Sets a logging-compatible string used to format captured logging messages.

[pytest]
log_format = %(asctime)s %(levelname)s %(message)s

For more information, see Logging.

log_level

Sets the minimum log message level that should be captured for logging capture. The integer value or the names of the levels can be used.

[pytest]
log_level = INFO

For more information, see Logging.

log_print

If set to False, will disable displaying captured logging messages for failed tests.

[pytest]
log_print = False

For more information, see Logging.

markers

When the --strict-markers or --strict command-line arguments are used, only known markers - defined in code by core pytest or some plugin - are allowed.

You can list additional markers in this setting to add them to the whitelist, in which case you probably want to add --strict-markers to addopts to avoid future regressions:

[pytest]
addopts = --strict-markers
markers =
    slow
    serial

minversion

Specifies a minimal pytest version required for running tests.

# content of pytest.ini
[pytest]
minversion = 3.0  # will fail if we run with pytest-2.8

norecursedirs

Set the directory basename patterns to avoid when recursing for test discovery. The individual (fnmatch-style) patterns are applied to the basename of a directory to decide if to recurse into it. Pattern matching characters:

*       matches everything
?       matches any single character
[seq]   matches any character in seq
[!seq]  matches any char not in seq

Default patterns are '.*', 'build', 'dist', 'CVS', '_darcs', '{arch}', '*.egg', 'venv'. Setting a norecursedirs replaces the default. Here is an example of how to avoid certain directories:

[pytest]
norecursedirs = .svn _build tmp*

This would tell pytest to not look into typical subversion or sphinx-build directories or into any tmp prefixed directory.

Additionally, pytest will attempt to intelligently identify and ignore a virtualenv by the presence of an activation script. Any directory deemed to be the root of a virtual environment will not be considered during test collection unless ‑‑collect‑in‑virtualenv is given. Note also that norecursedirs takes precedence over ‑‑collect‑in‑virtualenv; e.g. if you intend to run tests in a virtualenv with a base directory that matches '.*' you must override norecursedirs in addition to using the ‑‑collect‑in‑virtualenv flag.

python_classes

One or more name prefixes or glob-style patterns determining which classes are considered for test collection. Search for multiple glob patterns by adding a space between patterns. By default, pytest will consider any class prefixed with Test as a test collection. Here is an example of how to collect tests from classes that end in Suite:

[pytest]
python_classes = *Suite

Note that unittest.TestCase derived classes are always collected regardless of this option, as unittest’s own collection framework is used to collect those tests.

python_files

One or more Glob-style file patterns determining which python files are considered as test modules. Search for multiple glob patterns by adding a space between patterns:

[pytest]
python_files = test_*.py check_*.py example_*.py

Or one per line:

[pytest]
python_files =
    test_*.py
    check_*.py
    example_*.py

By default, files matching test_*.py and *_test.py will be considered test modules.

python_functions

One or more name prefixes or glob-patterns determining which test functions and methods are considered tests. Search for multiple glob patterns by adding a space between patterns. By default, pytest will consider any function prefixed with test as a test. Here is an example of how to collect test functions and methods that end in _test:

[pytest]
python_functions = *_test

Note that this has no effect on methods that live on a unittest .TestCase derived class, as unittest’s own collection framework is used to collect those tests.

See Changing naming conventions for more detailed examples.

testpaths

Sets list of directories that should be searched for tests when no specific directories, files or test ids are given in the command line when executing pytest from the rootdir directory. Useful when all project tests are in a known location to speed up test collection and to avoid picking up undesired tests by accident.

[pytest]
testpaths = testing doc

This tells pytest to only look for tests in testing and doc directories when executing from the root directory.

usefixtures

List of fixtures that will be applied to all test functions; this is semantically the same to apply the @pytest.mark.usefixtures marker to all test functions.

[pytest]
usefixtures =
    clean_db

xfail_strict

If set to True, tests marked with @pytest.mark.xfail that actually succeed will by default fail the test suite. For more information, see strict parameter.

[pytest]
xfail_strict = True

Good Integration Practices

Install package with pip

For development, we recommend you use venv for virtual environments and pip for installing your application and any dependencies, as well as the pytest package itself. This ensures your code and dependencies are isolated from your system Python installation.

Next, place a setup.py file in the root of your package with the following minimum content:

from setuptools import setup, find_packages

setup(name="PACKAGENAME", packages=find_packages())

Where PACKAGENAME is the name of your package. You can then install your package in “editable” mode by running from the same directory:

pip install -e .

which lets you change your source code (both tests and application) and rerun tests at will. This is similar to running python setup.py develop or conda develop in that it installs your package using a symlink to your development code.

Conventions for Python test discovery

pytest implements the following standard test discovery:

• If no arguments are specified then collection starts from testpaths (if configured) or the current directory. Alternatively, command line arguments can be used in any combination of directories, file names or node ids.
• Recurse into directories, unless they match norecursedirs.
• In those directories, search for test_*.py or *_test.py files, imported by their test package name.
• From those files, collect test items:
  • test prefixed test functions or methods outside of class
  • test prefixed test functions or methods inside Test prefixed test classes (without an __init__ method)

For examples of how to customize your test discovery Changing standard (Python) test discovery.

Within Python modules, pytest also discovers tests using the standard unittest.TestCase subclassing technique.

Choosing a test layout / import rules

pytest supports two common test layouts:

Tests outside application code

Putting tests into an extra directory outside your actual application code might be useful if you have many functional tests or for other reasons want to keep tests separate from actual application code (often a good idea):

setup.py
mypkg/
    __init__.py
    app.py
    view.py
tests/
    test_app.py
    test_view.py
    ...

This has the following benefits:

• Your tests can run against an installed version after executing pip install ..
• Your tests can run against the local copy with an editable install after executing pip install --editable ..
• If you don’t have a setup.py file and are relying on the fact that Python by default puts the current directory in sys.path to import your package, you can execute python -m pytest to execute the tests against the local copy directly, without using pip.

Note

See Invoking pytest versus python -m pytest for more information about the difference between calling pytest and python -m pytest.

Note that using this scheme your test files must have unique names, because pytest will import them as top-level modules since there are no packages to derive a full package name from. In other words, the test files in the example above will be imported as test_app and test_view top-level modules by adding tests/ to sys.path.

If you need to have test modules with the same name, you might add __init__.py files to your tests folder and subfolders, changing them to packages:

setup.py
mypkg/
    ...
tests/
    __init__.py
    foo/
        __init__.py
        test_view.py
    bar/
        __init__.py
        test_view.py

Now pytest will load the modules as tests.foo.test_view and tests.bar.test_view, allowing you to have modules with the same name. But now this introduces a subtle problem: in order to load the test modules from the tests directory, pytest prepends the root of the repository to sys.path, which adds the side-effect that now mypkg is also importable. This is problematic if you are using a tool like tox to test your package in a virtual environment, because you want to test the installed version of your package, not the local code from the repository.

In this situation, it is strongly suggested to use a src layout where application root package resides in a sub-directory of your root:

setup.py
src/
    mypkg/
        __init__.py
        app.py
        view.py
tests/
    __init__.py
    foo/
        __init__.py
        test_view.py
    bar/
        __init__.py
        test_view.py

This layout prevents a lot of common pitfalls and has many benefits, which are better explained in this excellent blog post by Ionel Cristian Mărieș.

Tests as part of application code

Inlining test directories into your application package is useful if you have direct relation between tests and application modules and want to distribute them along with your application:

setup.py
mypkg/
    __init__.py
    app.py
    view.py
    test/
        __init__.py
        test_app.py
        test_view.py
        ...

In this scheme, it is easy to run your tests using the --pyargs option:

pytest --pyargs mypkg

pytest will discover where mypkg is installed and collect tests from there.

Note that this layout also works in conjunction with the src layout mentioned in the previous section.

Note

You can use Python3 namespace packages (PEP420) for your application but pytest will still perform test package name discovery based on the presence of __init__.py files. If you use one of the two recommended file system layouts above but leave away the __init__.py files from your directories it should just work on Python3.3 and above. From “inlined tests”, however, you will need to use absolute imports for getting at your application code.

Note

If pytest finds an “a/b/test_module.py” test file while recursing into the filesystem it determines the import name as follows:

• determine basedir: this is the first “upward” (towards the root) directory not containing an __init__.py. If e.g. both a and b contain an __init__.py file then the parent directory of a will become the basedir.
• perform sys.path.insert(0, basedir) to make the test module importable under the fully qualified import name.
• import a.b.test_module where the path is determined by converting path separators / into “.” characters. This means you must follow the convention of having directory and file names map directly to the import names.

The reason for this somewhat evolved importing technique is that in larger projects multiple test modules might import from each other and thus deriving a canonical import name helps to avoid surprises such as a test module getting imported twice.

tox

Once you are done with your work and want to make sure that your actual package passes all tests you may want to look into tox, the virtualenv test automation tool and its pytest support. tox helps you to setup virtualenv environments with pre-defined dependencies and then executing a pre-configured test command with options. It will run tests against the installed package and not against your source code checkout, helping to detect packaging glitches.

Flaky tests

A “flaky” test is one that exhibits intermittent or sporadic failure, that seems to have non-deterministic behaviour. Sometimes it passes, sometimes it fails, and it’s not clear why. This page discusses pytest features that can help and other general strategies for identifying, fixing or mitigating them.

Why flaky tests are a problem

Flaky tests are particularly troublesome when a continuous integration (CI) server is being used, so that all tests must pass before a new code change can be merged. If the test result is not a reliable signal – that a test failure means the code change broke the test – developers can become mistrustful of the test results, which can lead to overlooking genuine failures. It is also a source of wasted time as developers must re-run test suites and investigate spurious failures.

Potential root causes

System state

Broadly speaking, a flaky test indicates that the test relies on some system state that is not being appropriately controlled - the test environment is not sufficiently isolated. Higher level tests are more likely to be flaky as they rely on more state.

Flaky tests sometimes appear when a test suite is run in parallel (such as use of pytest-xdist). This can indicate a test is reliant on test ordering.

• Perhaps a different test is failing to clean up after itself and leaving behind data which causes the flaky test to fail.
• The flaky test is reliant on data from a previous test that doesn’t clean up after itself, and in parallel runs that previous test is not always present
• Tests that modify global state typically cannot be run in parallel.

Overly strict assertion

Overly strict assertions can cause problems with floating point comparison as well as timing issues. pytest.approx is useful here.

Pytest features

Xfail strict

pytest.mark.xfail with strict=False can be used to mark a test so that its failure does not cause the whole build to break. This could be considered like a manual quarantine, and is rather dangerous to use permanently.

PYTEST_CURRENT_TEST

PYTEST_CURRENT_TEST environment variable may be useful for figuring out “which test got stuck”.

Plugins

Rerunning any failed tests can mitigate the negative effects of flaky tests by giving them additional chances to pass, so that the overall build does not fail. Several pytest plugins support this:

• flaky
• pytest-flakefinder - blog post
• pytest-rerunfailures
• pytest-replay: This plugin helps to reproduce locally crashes or flaky tests observed during CI runs.

Plugins to deliberately randomize tests can help expose tests with state problems:

• pytest-random-order
• pytest-randomly

Other general strategies

Split up test suites

It can be common to split a single test suite into two, such as unit vs integration, and only use the unit test suite as a CI gate. This also helps keep build times manageable as high level tests tend to be slower. However, it means it does become possible for code that breaks the build to be merged, so extra vigilance is needed for monitoring the integration test results.

Video/screenshot on failure

For UI tests these are important for understanding what the state of the UI was when the test failed. pytest-splinter can be used with plugins like pytest-bdd and can save a screenshot on test failure, which can help to isolate the cause.

Delete or rewrite the test

If the functionality is covered by other tests, perhaps the test can be removed. If not, perhaps it can be rewritten at a lower level which will remove the flakiness or make its source more apparent.

Quarantine

Mark Lapierre discusses the Pros and Cons of Quarantined Tests in a post from 2018.

CI tools that rerun on failure

Azure Pipelines (the Azure cloud CI/CD tool, formerly Visual Studio Team Services or VSTS) has a feature to identify flaky tests and rerun failed tests.

Research

This is a limited list, please submit an issue or pull request to expand it!

• Gao, Zebao, Yalan Liang, Myra B. Cohen, Atif M. Memon, and Zhen Wang. “Making system user interactive tests repeatable: When and what should we control?.” In Software Engineering (ICSE), 2015 IEEE/ACM 37th IEEE International Conference on, vol. 1, pp. 55-65. IEEE, 2015. PDF
• Palomba, Fabio, and Andy Zaidman. “Does refactoring of test smells induce fixing flaky tests?.” In Software Maintenance and Evolution (ICSME), 2017 IEEE International Conference on, pp. 1-12. IEEE, 2017. PDF in Google Drive
• Bell, Jonathan, Owolabi Legunsen, Michael Hilton, Lamyaa Eloussi, Tifany Yung, and Darko Marinov. “DeFlaker: Automatically detecting flaky tests.” In Proceedings of the 2018 International Conference on Software Engineering. 2018. PDF

Resources

• Eradicating Non-Determinism in Tests by Martin Fowler, 2011
• No more flaky tests on the Go team by Pavan Sudarshan, 2012
• The Build That Cried Broken: Building Trust in your Continuous Integration Tests talk (video) by Angie Jones at SeleniumConf Austin 2017
• Test and Code Podcast: Flaky Tests and How to Deal with Them by Brian Okken and Anthony Shaw, 2018
• Microsoft:
  • How we approach testing VSTS to enable continuous delivery by Brian Harry MS, 2017
  • Eliminating Flaky Tests blog and talk (video) by Munil Shah, 2017
• Google:
  • Flaky Tests at Google and How We Mitigate Them by John Micco, 2016
  • Where do Google’s flaky tests come from? by Jeff Listfield, 2017

pytest import mechanisms and sys.path/PYTHONPATH

Here’s a list of scenarios where pytest may need to change sys.path in order to import test modules or conftest.py files.

Test modules / conftest.py files inside packages

Consider this file and directory layout:

root/
|- foo/
   |- __init__.py
   |- conftest.py
   |- bar/
      |- __init__.py
      |- tests/
         |- __init__.py
         |- test_foo.py

When executing:

pytest root/

pytest will find foo/bar/tests/test_foo.py and realize it is part of a package given that there’s an __init__.py file in the same folder. It will then search upwards until it can find the last folder which still contains an __init__.py file in order to find the package root (in this case foo/). To load the module, it will insert root/ to the front of sys.path (if not there already) in order to load test_foo.py as the module foo.bar.tests.test_foo.

The same logic applies to the conftest.py file: it will be imported as foo.conftest module.

Preserving the full package name is important when tests live in a package to avoid problems and allow test modules to have duplicated names. This is also discussed in details in Conventions for Python test discovery.

Standalone test modules / conftest.py files

Consider this file and directory layout:

root/
|- foo/
   |- conftest.py
   |- bar/
      |- tests/
         |- test_foo.py

When executing:

pytest root/

pytest will find foo/bar/tests/test_foo.py and realize it is NOT part of a package given that there’s no __init__.py file in the same folder. It will then add root/foo/bar/tests to sys.path in order to import test_foo.py as the module test_foo. The same is done with the conftest.py file by adding root/foo to sys.path to import it as conftest.

For this reason this layout cannot have test modules with the same name, as they all will be imported in the global import namespace.

This is also discussed in details in Conventions for Python test discovery.

Invoking pytest versus python -m pytest

Running pytest with python -m pytest [...] instead of pytest [...] yields nearly equivalent behaviour, except that the former call will add the current directory to sys.path. See also Calling pytest through python -m pytest.

Configuration

Command line options and configuration file settings

You can get help on command line options and values in INI-style configurations files by using the general help option:

pytest -h   # prints options _and_ config file settings

This will display command line and configuration file settings which were registered by installed plugins.

Initialization: determining rootdir and inifile

pytest determines a rootdir for each test run which depends on the command line arguments (specified test files, paths) and on the existence of ini-files. The determined rootdir and ini-file are printed as part of the pytest header during startup.

Here’s a summary what pytest uses rootdir for:

• Construct nodeids during collection; each test is assigned a unique nodeid which is rooted at the rootdir and takes into account the full path, class name, function name and parametrization (if any).
• Is used by plugins as a stable location to store project/test run specific information; for example, the internal cache plugin creates a .pytest_cache subdirectory in rootdir to store its cross-test run state.

rootdir is NOT used to modify sys.path/PYTHONPATH or influence how modules are imported. See pytest import mechanisms and sys.path/PYTHONPATH for more details.

The --rootdir=path command-line option can be used to force a specific directory. The directory passed may contain environment variables when it is used in conjunction with addopts in a pytest.ini file.

Finding the rootdir

Here is the algorithm which finds the rootdir from args:

• determine the common ancestor directory for the specified args that are recognised as paths that exist in the file system. If no such paths are found, the common ancestor directory is set to the current working directory.
• look for pytest.ini, tox.ini and setup.cfg files in the ancestor directory and upwards. If one is matched, it becomes the ini-file and its directory becomes the rootdir.
• if no ini-file was found, look for setup.py upwards from the common ancestor directory to determine the rootdir.
• if no setup.py was found, look for pytest.ini, tox.ini and setup.cfg in each of the specified args and upwards. If one is matched, it becomes the ini-file and its directory becomes the rootdir.
• if no ini-file was found, use the already determined common ancestor as root directory. This allows the use of pytest in structures that are not part of a package and don’t have any particular ini-file configuration.

If no args are given, pytest collects test below the current working directory and also starts determining the rootdir from there.

warning:custom pytest plugin commandline arguments may include a path, as in pytest --log-output ../../test.log args. Then args is mandatory, otherwise pytest uses the folder of test.log for rootdir determination (see also issue 1435). A dot . for referencing to the current working directory is also possible.

Note that an existing pytest.ini file will always be considered a match, whereas tox.ini and setup.cfg will only match if they contain a [pytest] or [tool:pytest] section, respectively. Options from multiple ini-files candidates are never merged - the first one wins (pytest.ini always wins, even if it does not contain a [pytest] section).

The config object will subsequently carry these attributes:

• config.rootdir: the determined root directory, guaranteed to exist.
• config.inifile: the determined ini-file, may be None.

The rootdir is used as a reference directory for constructing test addresses (“nodeids”) and can be used also by plugins for storing per-testrun information.

Example:

pytest path/to/testdir path/other/

will determine the common ancestor as path and then check for ini-files as follows:

# first look for pytest.ini files
path/pytest.ini
path/tox.ini    # must also contain [pytest] section to match
path/setup.cfg  # must also contain [tool:pytest] section to match
pytest.ini
... # all the way down to the root

# now look for setup.py
path/setup.py
setup.py
... # all the way down to the root

How to change command line options defaults

It can be tedious to type the same series of command line options every time you use pytest. For example, if you always want to see detailed info on skipped and xfailed tests, as well as have terser “dot” progress output, you can write it into a configuration file:

# content of pytest.ini or tox.ini
[pytest]
addopts = -ra -q

# content of setup.cfg
[tool:pytest]
addopts = -ra -q

Alternatively, you can set a PYTEST_ADDOPTS environment variable to add command line options while the environment is in use:

export PYTEST_ADDOPTS="-v"

Here’s how the command-line is built in the presence of addopts or the environment variable:

<pytest.ini:addopts> $PYTEST_ADDOPTS <extra command-line arguments>

So if the user executes in the command-line:

pytest -m slow

The actual command line executed is:

pytest -ra -q -v -m slow

Note that as usual for other command-line applications, in case of conflicting options the last one wins, so the example above will show verbose output because -v overwrites -q.

Builtin configuration file options

For the full list of options consult the reference documentation.

Examples and customization tricks

Here is a (growing) list of examples. Contact us if you need more examples or have questions. Also take a look at the comprehensive documentation which contains many example snippets as well. Also, pytest on stackoverflow.com often comes with example answers.

For basic examples, see

• Installation and Getting Started for basic introductory examples
• Asserting with the assert statement for basic assertion examples
• pytest fixtures: explicit, modular, scalable for basic fixture/setup examples
• Parametrizing fixtures and test functions for basic test function parametrization
• unittest.TestCase Support for basic unittest integration
• Running tests written for nose for basic nosetests integration

The following examples aim at various use cases you might encounter.

Demo of Python failure reports with pytest

Here is a nice run of several failures and how pytest presents things:

assertion $ pytest failure_demo.py
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR/assertion
collected 44 items

failure_demo.py FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF         [100%]

================================= FAILURES =================================
___________________________ test_generative[3-6] ___________________________

param1 = 3, param2 = 6

    @pytest.mark.parametrize("param1, param2", [(3, 6)])
    def test_generative(param1, param2):
>       assert param1 * 2 < param2
E       assert (3 * 2) < 6

failure_demo.py:20: AssertionError
_________________________ TestFailing.test_simple __________________________

self = <failure_demo.TestFailing object at 0xdeadbeef>

    def test_simple(self):
        def f():
            return 42

        def g():
            return 43

>       assert f() == g()
E       assert 42 == 43
E        +  where 42 = <function TestFailing.test_simple.<locals>.f at 0xdeadbeef>()
E        +  and   43 = <function TestFailing.test_simple.<locals>.g at 0xdeadbeef>()

failure_demo.py:31: AssertionError
____________________ TestFailing.test_simple_multiline _____________________

self = <failure_demo.TestFailing object at 0xdeadbeef>

    def test_simple_multiline(self):
>       otherfunc_multi(42, 6 * 9)

failure_demo.py:34:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

a = 42, b = 54

    def otherfunc_multi(a, b):
>       assert a == b
E       assert 42 == 54

failure_demo.py:15: AssertionError
___________________________ TestFailing.test_not ___________________________

self = <failure_demo.TestFailing object at 0xdeadbeef>

    def test_not(self):
        def f():
            return 42

>       assert not f()
E       assert not 42
E        +  where 42 = <function TestFailing.test_not.<locals>.f at 0xdeadbeef>()

failure_demo.py:40: AssertionError
_________________ TestSpecialisedExplanations.test_eq_text _________________

self = <failure_demo.TestSpecialisedExplanations object at 0xdeadbeef>

    def test_eq_text(self):
>       assert "spam" == "eggs"
E       AssertionError: assert 'spam' == 'eggs'
E         - spam
E         + eggs

failure_demo.py:45: AssertionError
_____________ TestSpecialisedExplanations.test_eq_similar_text _____________

self = <failure_demo.TestSpecialisedExplanations object at 0xdeadbeef>

    def test_eq_similar_text(self):
>       assert "foo 1 bar" == "foo 2 bar"
E       AssertionError: assert 'foo 1 bar' == 'foo 2 bar'
E         - foo 1 bar
E         ?     ^
E         + foo 2 bar
E         ?     ^

failure_demo.py:48: AssertionError
____________ TestSpecialisedExplanations.test_eq_multiline_text ____________

self = <failure_demo.TestSpecialisedExplanations object at 0xdeadbeef>

    def test_eq_multiline_text(self):
>       assert "foo\nspam\nbar" == "foo\neggs\nbar"
E       AssertionError: assert 'foo\nspam\nbar' == 'foo\neggs\nbar'
E           foo
E         - spam
E         + eggs
E           bar

failure_demo.py:51: AssertionError
______________ TestSpecialisedExplanations.test_eq_long_text _______________

self = <failure_demo.TestSpecialisedExplanations object at 0xdeadbeef>

    def test_eq_long_text(self):
        a = "1" * 100 + "a" + "2" * 100
        b = "1" * 100 + "b" + "2" * 100
>       assert a == b
E       AssertionError: assert '111111111111...2222222222222' == '111111111111...2222222222222'
E         Skipping 90 identical leading characters in diff, use -v to show
E         Skipping 91 identical trailing characters in diff, use -v to show
E         - 1111111111a222222222
E         ?           ^
E         + 1111111111b222222222
E         ?           ^

failure_demo.py:56: AssertionError
_________ TestSpecialisedExplanations.test_eq_long_text_multiline __________

self = <failure_demo.TestSpecialisedExplanations object at 0xdeadbeef>

    def test_eq_long_text_multiline(self):
        a = "1\n" * 100 + "a" + "2\n" * 100
        b = "1\n" * 100 + "b" + "2\n" * 100
>       assert a == b
E       AssertionError: assert '1\n1\n1\n1\n...n2\n2\n2\n2\n' == '1\n1\n1\n1\n...n2\n2\n2\n2\n'
E         Skipping 190 identical leading characters in diff, use -v to show
E         Skipping 191 identical trailing characters in diff, use -v to show
E           1
E           1
E           1
E           1
E           1...
E
E         ...Full output truncated (7 lines hidden), use '-vv' to show

failure_demo.py:61: AssertionError
_________________ TestSpecialisedExplanations.test_eq_list _________________

self = <failure_demo.TestSpecialisedExplanations object at 0xdeadbeef>

    def test_eq_list(self):
>       assert [0, 1, 2] == [0, 1, 3]
E       assert [0, 1, 2] == [0, 1, 3]
E         At index 2 diff: 2 != 3
E         Use -v to get the full diff

failure_demo.py:64: AssertionError
______________ TestSpecialisedExplanations.test_eq_list_long _______________

self = <failure_demo.TestSpecialisedExplanations object at 0xdeadbeef>

    def test_eq_list_long(self):
        a = [0] * 100 + [1] + [3] * 100
        b = [0] * 100 + [2] + [3] * 100
>       assert a == b
E       assert [0, 0, 0, 0, 0, 0, ...] == [0, 0, 0, 0, 0, 0, ...]
E         At index 100 diff: 1 != 2
E         Use -v to get the full diff

failure_demo.py:69: AssertionError
_________________ TestSpecialisedExplanations.test_eq_dict _________________

self = <failure_demo.TestSpecialisedExplanations object at 0xdeadbeef>

    def test_eq_dict(self):
>       assert {"a": 0, "b": 1, "c": 0} == {"a": 0, "b": 2, "d": 0}
E       AssertionError: assert {'a': 0, 'b': 1, 'c': 0} == {'a': 0, 'b': 2, 'd': 0}
E         Omitting 1 identical items, use -vv to show
E         Differing items:
E         {'b': 1} != {'b': 2}
E         Left contains 1 more item:
E         {'c': 0}
E         Right contains 1 more item:
E         {'d': 0}...
E
E         ...Full output truncated (2 lines hidden), use '-vv' to show

failure_demo.py:72: AssertionError
_________________ TestSpecialisedExplanations.test_eq_set __________________

self = <failure_demo.TestSpecialisedExplanations object at 0xdeadbeef>

    def test_eq_set(self):
>       assert {0, 10, 11, 12} == {0, 20, 21}
E       AssertionError: assert {0, 10, 11, 12} == {0, 20, 21}
E         Extra items in the left set:
E         10
E         11
E         12
E         Extra items in the right set:
E         20
E         21...
E
E         ...Full output truncated (2 lines hidden), use '-vv' to show

failure_demo.py:75: AssertionError
_____________ TestSpecialisedExplanations.test_eq_longer_list ______________

self = <failure_demo.TestSpecialisedExplanations object at 0xdeadbeef>

    def test_eq_longer_list(self):
>       assert [1, 2] == [1, 2, 3]
E       assert [1, 2] == [1, 2, 3]
E         Right contains one more item: 3
E         Use -v to get the full diff

failure_demo.py:78: AssertionError
_________________ TestSpecialisedExplanations.test_in_list _________________

self = <failure_demo.TestSpecialisedExplanations object at 0xdeadbeef>

    def test_in_list(self):
>       assert 1 in [0, 2, 3, 4, 5]
E       assert 1 in [0, 2, 3, 4, 5]

failure_demo.py:81: AssertionError
__________ TestSpecialisedExplanations.test_not_in_text_multiline __________

self = <failure_demo.TestSpecialisedExplanations object at 0xdeadbeef>

    def test_not_in_text_multiline(self):
        text = "some multiline\ntext\nwhich\nincludes foo\nand a\ntail"
>       assert "foo" not in text
E       AssertionError: assert 'foo' not in 'some multil...nand a\ntail'
E         'foo' is contained here:
E           some multiline
E           text
E           which
E           includes foo
E         ?          +++
E           and a...
E
E         ...Full output truncated (2 lines hidden), use '-vv' to show

failure_demo.py:85: AssertionError
___________ TestSpecialisedExplanations.test_not_in_text_single ____________

self = <failure_demo.TestSpecialisedExplanations object at 0xdeadbeef>

    def test_not_in_text_single(self):
        text = "single foo line"
>       assert "foo" not in text
E       AssertionError: assert 'foo' not in 'single foo line'
E         'foo' is contained here:
E           single foo line
E         ?        +++

failure_demo.py:89: AssertionError
_________ TestSpecialisedExplanations.test_not_in_text_single_long _________

self = <failure_demo.TestSpecialisedExplanations object at 0xdeadbeef>

    def test_not_in_text_single_long(self):
        text = "head " * 50 + "foo " + "tail " * 20
>       assert "foo" not in text
E       AssertionError: assert 'foo' not in 'head head h...l tail tail '
E         'foo' is contained here:
E           head head foo tail tail tail tail tail tail tail tail tail tail tail tail tail tail tail tail tail tail tail tail
E         ?           +++

failure_demo.py:93: AssertionError
______ TestSpecialisedExplanations.test_not_in_text_single_long_term _______

self = <failure_demo.TestSpecialisedExplanations object at 0xdeadbeef>

    def test_not_in_text_single_long_term(self):
        text = "head " * 50 + "f" * 70 + "tail " * 20
>       assert "f" * 70 not in text
E       AssertionError: assert 'fffffffffff...ffffffffffff' not in 'head head h...l tail tail '
E         'ffffffffffffffffff...fffffffffffffffffff' is contained here:
E           head head fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffftail tail tail tail tail tail tail tail tail tail tail tail tail tail tail tail tail tail tail tail
E         ?           ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

failure_demo.py:97: AssertionError
______________ TestSpecialisedExplanations.test_eq_dataclass _______________

self = <failure_demo.TestSpecialisedExplanations object at 0xdeadbeef>

    def test_eq_dataclass(self):
        from dataclasses import dataclass

        @dataclass
        class Foo:
            a: int
            b: str

        left = Foo(1, "b")
        right = Foo(1, "c")
>       assert left == right
E       AssertionError: assert TestSpecialis...oo(a=1, b='b') == TestSpecialis...oo(a=1, b='c')
E         Omitting 1 identical items, use -vv to show
E         Differing attributes:
E         b: 'b' != 'c'

failure_demo.py:109: AssertionError
________________ TestSpecialisedExplanations.test_eq_attrs _________________

self = <failure_demo.TestSpecialisedExplanations object at 0xdeadbeef>

    def test_eq_attrs(self):
        import attr

        @attr.s
        class Foo:
            a = attr.ib()
            b = attr.ib()

        left = Foo(1, "b")
        right = Foo(1, "c")
>       assert left == right
E       AssertionError: assert Foo(a=1, b='b') == Foo(a=1, b='c')
E         Omitting 1 identical items, use -vv to show
E         Differing attributes:
E         b: 'b' != 'c'

failure_demo.py:121: AssertionError
______________________________ test_attribute ______________________________

    def test_attribute():
        class Foo:
            b = 1

        i = Foo()
>       assert i.b == 2
E       assert 1 == 2
E        +  where 1 = <failure_demo.test_attribute.<locals>.Foo object at 0xdeadbeef>.b

failure_demo.py:129: AssertionError
_________________________ test_attribute_instance __________________________

    def test_attribute_instance():
        class Foo:
            b = 1

>       assert Foo().b == 2
E       AssertionError: assert 1 == 2
E        +  where 1 = <failure_demo.test_attribute_instance.<locals>.Foo object at 0xdeadbeef>.b
E        +    where <failure_demo.test_attribute_instance.<locals>.Foo object at 0xdeadbeef> = <class 'failure_demo.test_attribute_instance.<locals>.Foo'>()

failure_demo.py:136: AssertionError
__________________________ test_attribute_failure __________________________

    def test_attribute_failure():
        class Foo:
            def _get_b(self):
                raise Exception("Failed to get attrib")

            b = property(_get_b)

        i = Foo()
>       assert i.b == 2

failure_demo.py:147:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <failure_demo.test_attribute_failure.<locals>.Foo object at 0xdeadbeef>

    def _get_b(self):
>       raise Exception("Failed to get attrib")
E       Exception: Failed to get attrib

failure_demo.py:142: Exception
_________________________ test_attribute_multiple __________________________

    def test_attribute_multiple():
        class Foo:
            b = 1

        class Bar:
            b = 2

>       assert Foo().b == Bar().b
E       AssertionError: assert 1 == 2
E        +  where 1 = <failure_demo.test_attribute_multiple.<locals>.Foo object at 0xdeadbeef>.b
E        +    where <failure_demo.test_attribute_multiple.<locals>.Foo object at 0xdeadbeef> = <class 'failure_demo.test_attribute_multiple.<locals>.Foo'>()
E        +  and   2 = <failure_demo.test_attribute_multiple.<locals>.Bar object at 0xdeadbeef>.b
E        +    where <failure_demo.test_attribute_multiple.<locals>.Bar object at 0xdeadbeef> = <class 'failure_demo.test_attribute_multiple.<locals>.Bar'>()

failure_demo.py:157: AssertionError
__________________________ TestRaises.test_raises __________________________

self = <failure_demo.TestRaises object at 0xdeadbeef>

    def test_raises(self):
        s = "qwe"
>       raises(TypeError, int, s)
E       ValueError: invalid literal for int() with base 10: 'qwe'

failure_demo.py:167: ValueError
______________________ TestRaises.test_raises_doesnt _______________________

self = <failure_demo.TestRaises object at 0xdeadbeef>

    def test_raises_doesnt(self):
>       raises(IOError, int, "3")
E       Failed: DID NOT RAISE <class 'OSError'>

failure_demo.py:170: Failed
__________________________ TestRaises.test_raise ___________________________

self = <failure_demo.TestRaises object at 0xdeadbeef>

    def test_raise(self):
>       raise ValueError("demo error")
E       ValueError: demo error

failure_demo.py:173: ValueError
________________________ TestRaises.test_tupleerror ________________________

self = <failure_demo.TestRaises object at 0xdeadbeef>

    def test_tupleerror(self):
>       a, b = [1]  # NOQA
E       ValueError: not enough values to unpack (expected 2, got 1)

failure_demo.py:176: ValueError
______ TestRaises.test_reinterpret_fails_with_print_for_the_fun_of_it ______

self = <failure_demo.TestRaises object at 0xdeadbeef>

    def test_reinterpret_fails_with_print_for_the_fun_of_it(self):
        items = [1, 2, 3]
        print("items is {!r}".format(items))
>       a, b = items.pop()
E       TypeError: cannot unpack non-iterable int object

failure_demo.py:181: TypeError
--------------------------- Captured stdout call ---------------------------
items is [1, 2, 3]
________________________ TestRaises.test_some_error ________________________

self = <failure_demo.TestRaises object at 0xdeadbeef>

    def test_some_error(self):
>       if namenotexi:  # NOQA
E       NameError: name 'namenotexi' is not defined

failure_demo.py:184: NameError
____________________ test_dynamic_compile_shows_nicely _____________________

    def test_dynamic_compile_shows_nicely():
        import importlib.util
        import sys

        src = "def foo():\n assert 1 == 0\n"
        name = "abc-123"
        spec = importlib.util.spec_from_loader(name, loader=None)
        module = importlib.util.module_from_spec(spec)
        code = _pytest._code.compile(src, name, "exec")
        exec(code, module.__dict__)
        sys.modules[name] = module
>       module.foo()

failure_demo.py:203:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    def foo():
>    assert 1 == 0
E    AssertionError

<0-codegen 'abc-123' $REGENDOC_TMPDIR/assertion/failure_demo.py:200>:2: AssertionError
____________________ TestMoreErrors.test_complex_error _____________________

self = <failure_demo.TestMoreErrors object at 0xdeadbeef>

    def test_complex_error(self):
        def f():
            return 44

        def g():
            return 43

>       somefunc(f(), g())

failure_demo.py:214:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
failure_demo.py:11: in somefunc
    otherfunc(x, y)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

a = 44, b = 43

    def otherfunc(a, b):
>       assert a == b
E       assert 44 == 43

failure_demo.py:7: AssertionError
___________________ TestMoreErrors.test_z1_unpack_error ____________________

self = <failure_demo.TestMoreErrors object at 0xdeadbeef>

    def test_z1_unpack_error(self):
        items = []
>       a, b = items
E       ValueError: not enough values to unpack (expected 2, got 0)

failure_demo.py:218: ValueError
____________________ TestMoreErrors.test_z2_type_error _____________________

self = <failure_demo.TestMoreErrors object at 0xdeadbeef>

    def test_z2_type_error(self):
        items = 3
>       a, b = items
E       TypeError: cannot unpack non-iterable int object

failure_demo.py:222: TypeError
______________________ TestMoreErrors.test_startswith ______________________

self = <failure_demo.TestMoreErrors object at 0xdeadbeef>

    def test_startswith(self):
        s = "123"
        g = "456"
>       assert s.startswith(g)
E       AssertionError: assert False
E        +  where False = <built-in method startswith of str object at 0xdeadbeef>('456')
E        +    where <built-in method startswith of str object at 0xdeadbeef> = '123'.startswith

failure_demo.py:227: AssertionError
__________________ TestMoreErrors.test_startswith_nested ___________________

self = <failure_demo.TestMoreErrors object at 0xdeadbeef>

    def test_startswith_nested(self):
        def f():
            return "123"

        def g():
            return "456"

>       assert f().startswith(g())
E       AssertionError: assert False
E        +  where False = <built-in method startswith of str object at 0xdeadbeef>('456')
E        +    where <built-in method startswith of str object at 0xdeadbeef> = '123'.startswith
E        +      where '123' = <function TestMoreErrors.test_startswith_nested.<locals>.f at 0xdeadbeef>()
E        +    and   '456' = <function TestMoreErrors.test_startswith_nested.<locals>.g at 0xdeadbeef>()

failure_demo.py:236: AssertionError
_____________________ TestMoreErrors.test_global_func ______________________

self = <failure_demo.TestMoreErrors object at 0xdeadbeef>

    def test_global_func(self):
>       assert isinstance(globf(42), float)
E       assert False
E        +  where False = isinstance(43, float)
E        +    where 43 = globf(42)

failure_demo.py:239: AssertionError
_______________________ TestMoreErrors.test_instance _______________________

self = <failure_demo.TestMoreErrors object at 0xdeadbeef>

    def test_instance(self):
        self.x = 6 * 7
>       assert self.x != 42
E       assert 42 != 42
E        +  where 42 = <failure_demo.TestMoreErrors object at 0xdeadbeef>.x

failure_demo.py:243: AssertionError
_______________________ TestMoreErrors.test_compare ________________________

self = <failure_demo.TestMoreErrors object at 0xdeadbeef>

    def test_compare(self):
>       assert globf(10) < 5
E       assert 11 < 5
E        +  where 11 = globf(10)

failure_demo.py:246: AssertionError
_____________________ TestMoreErrors.test_try_finally ______________________

self = <failure_demo.TestMoreErrors object at 0xdeadbeef>

    def test_try_finally(self):
        x = 1
        try:
>           assert x == 0
E           assert 1 == 0

failure_demo.py:251: AssertionError
___________________ TestCustomAssertMsg.test_single_line ___________________

self = <failure_demo.TestCustomAssertMsg object at 0xdeadbeef>

    def test_single_line(self):
        class A:
            a = 1

        b = 2
>       assert A.a == b, "A.a appears not to be b"
E       AssertionError: A.a appears not to be b
E       assert 1 == 2
E        +  where 1 = <class 'failure_demo.TestCustomAssertMsg.test_single_line.<locals>.A'>.a

failure_demo.py:262: AssertionError
____________________ TestCustomAssertMsg.test_multiline ____________________

self = <failure_demo.TestCustomAssertMsg object at 0xdeadbeef>

    def test_multiline(self):
        class A:
            a = 1

        b = 2
>       assert (
            A.a == b
        ), "A.a appears not to be b\nor does not appear to be b\none of those"
E       AssertionError: A.a appears not to be b
E         or does not appear to be b
E         one of those
E       assert 1 == 2
E        +  where 1 = <class 'failure_demo.TestCustomAssertMsg.test_multiline.<locals>.A'>.a

failure_demo.py:269: AssertionError
___________________ TestCustomAssertMsg.test_custom_repr ___________________

self = <failure_demo.TestCustomAssertMsg object at 0xdeadbeef>

    def test_custom_repr(self):
        class JSON:
            a = 1

            def __repr__(self):
                return "This is JSON\n{\n  'foo': 'bar'\n}"

        a = JSON()
        b = 2
>       assert a.a == b, a
E       AssertionError: This is JSON
E         {
E           'foo': 'bar'
E         }
E       assert 1 == 2
E        +  where 1 = This is JSON\n{\n  'foo': 'bar'\n}.a

failure_demo.py:282: AssertionError
============================ 44 failed in 0.12s ============================

Basic patterns and examples

Pass different values to a test function, depending on command line options

Suppose we want to write a test that depends on a command line option. Here is a basic pattern to achieve this:

# content of test_sample.py
def test_answer(cmdopt):
    if cmdopt == "type1":
        print("first")
    elif cmdopt == "type2":
        print("second")
    assert 0  # to see what was printed

For this to work we need to add a command line option and provide the cmdopt through a fixture function:

# content of conftest.py
import pytest


def pytest_addoption(parser):
    parser.addoption(
        "--cmdopt", action="store", default="type1", help="my option: type1 or type2"
    )


@pytest.fixture
def cmdopt(request):
    return request.config.getoption("--cmdopt")

Let’s run this without supplying our new option:

$ pytest -q test_sample.py
F                                                                    [100%]
================================= FAILURES =================================
_______________________________ test_answer ________________________________

cmdopt = 'type1'

    def test_answer(cmdopt):
        if cmdopt == "type1":
            print("first")
        elif cmdopt == "type2":
            print("second")
>       assert 0  # to see what was printed
E       assert 0

test_sample.py:6: AssertionError
--------------------------- Captured stdout call ---------------------------
first
1 failed in 0.12s

And now with supplying a command line option:

$ pytest -q --cmdopt=type2
F                                                                    [100%]
================================= FAILURES =================================
_______________________________ test_answer ________________________________

cmdopt = 'type2'

    def test_answer(cmdopt):
        if cmdopt == "type1":
            print("first")
        elif cmdopt == "type2":
            print("second")
>       assert 0  # to see what was printed
E       assert 0

test_sample.py:6: AssertionError
--------------------------- Captured stdout call ---------------------------
second
1 failed in 0.12s

You can see that the command line option arrived in our test. This completes the basic pattern. However, one often rather wants to process command line options outside of the test and rather pass in different or more complex objects.

Dynamically adding command line options

Through addopts you can statically add command line options for your project. You can also dynamically modify the command line arguments before they get processed:

# setuptools plugin
import sys


def pytest_load_initial_conftests(args):
    if "xdist" in sys.modules:  # pytest-xdist plugin
        import multiprocessing

        num = max(multiprocessing.cpu_count() / 2, 1)
        args[:] = ["-n", str(num)] + args

If you have the xdist plugin installed you will now always perform test runs using a number of subprocesses close to your CPU. Running in an empty directory with the above conftest.py:

$ pytest
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 0 items

========================== no tests ran in 0.12s ===========================

Control skipping of tests according to command line option

Here is a conftest.py file adding a --runslow command line option to control skipping of pytest.mark.slow marked tests:

# content of conftest.py

import pytest


def pytest_addoption(parser):
    parser.addoption(
        "--runslow", action="store_true", default=False, help="run slow tests"
    )


def pytest_configure(config):
    config.addinivalue_line("markers", "slow: mark test as slow to run")


def pytest_collection_modifyitems(config, items):
    if config.getoption("--runslow"):
        # --runslow given in cli: do not skip slow tests
        return
    skip_slow = pytest.mark.skip(reason="need --runslow option to run")
    for item in items:
        if "slow" in item.keywords:
            item.add_marker(skip_slow)

We can now write a test module like this:

# content of test_module.py
import pytest


def test_func_fast():
    pass


@pytest.mark.slow
def test_func_slow():
    pass

and when running it will see a skipped “slow” test:

$ pytest -rs    # "-rs" means report details on the little 's'
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 2 items

test_module.py .s                                                    [100%]

========================= short test summary info ==========================
SKIPPED [1] test_module.py:8: need --runslow option to run
======================= 1 passed, 1 skipped in 0.12s =======================

Or run it including the slow marked test:

$ pytest --runslow
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 2 items

test_module.py ..                                                    [100%]

============================ 2 passed in 0.12s =============================

Writing well integrated assertion helpers

If you have a test helper function called from a test you can use the pytest.fail marker to fail a test with a certain message. The test support function will not show up in the traceback if you set the __tracebackhide__ option somewhere in the helper function. Example:

# content of test_checkconfig.py
import pytest


def checkconfig(x):
    __tracebackhide__ = True
    if not hasattr(x, "config"):
        pytest.fail("not configured: {}".format(x))


def test_something():
    checkconfig(42)

The __tracebackhide__ setting influences pytest showing of tracebacks: the checkconfig function will not be shown unless the --full-trace command line option is specified. Let’s run our little function:

$ pytest -q test_checkconfig.py
F                                                                    [100%]
================================= FAILURES =================================
______________________________ test_something ______________________________

    def test_something():
>       checkconfig(42)
E       Failed: not configured: 42

test_checkconfig.py:11: Failed
1 failed in 0.12s

If you only want to hide certain exceptions, you can set __tracebackhide__ to a callable which gets the ExceptionInfo object. You can for example use this to make sure unexpected exception types aren’t hidden:

import operator
import pytest


class ConfigException(Exception):
    pass


def checkconfig(x):
    __tracebackhide__ = operator.methodcaller("errisinstance", ConfigException)
    if not hasattr(x, "config"):
        raise ConfigException("not configured: {}".format(x))


def test_something():
    checkconfig(42)

This will avoid hiding the exception traceback on unrelated exceptions (i.e. bugs in assertion helpers).

Detect if running from within a pytest run

Usually it is a bad idea to make application code behave differently if called from a test. But if you absolutely must find out if your application code is running from a test you can do something like this:

# content of your_module.py


_called_from_test = False

# content of conftest.py


def pytest_configure(config):
    your_module._called_from_test = True

and then check for the your_module._called_from_test flag:

if your_module._called_from_test:
    # called from within a test run
    ...
else:
    # called "normally"
    ...

accordingly in your application.

Adding info to test report header

It’s easy to present extra information in a pytest run:

# content of conftest.py


def pytest_report_header(config):
    return "project deps: mylib-1.1"

which will add the string to the test header accordingly:

$ pytest
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
project deps: mylib-1.1
rootdir: $REGENDOC_TMPDIR
collected 0 items

========================== no tests ran in 0.12s ===========================

It is also possible to return a list of strings which will be considered as several lines of information. You may consider config.getoption('verbose') in order to display more information if applicable:

# content of conftest.py


def pytest_report_header(config):
    if config.getoption("verbose") > 0:
        return ["info1: did you know that ...", "did you?"]

which will add info only when run with “–v”:

$ pytest -v
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y -- $PYTHON_PREFIX/bin/python
cachedir: $PYTHON_PREFIX/.pytest_cache
info1: did you know that ...
did you?
rootdir: $REGENDOC_TMPDIR
collecting ... collected 0 items

========================== no tests ran in 0.12s ===========================

and nothing when run plainly:

$ pytest
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 0 items

========================== no tests ran in 0.12s ===========================

profiling test duration

If you have a slow running large test suite you might want to find out which tests are the slowest. Let’s make an artificial test suite:

# content of test_some_are_slow.py
import time


def test_funcfast():
    time.sleep(0.1)


def test_funcslow1():
    time.sleep(0.2)


def test_funcslow2():
    time.sleep(0.3)

Now we can profile which test functions execute the slowest:

$ pytest --durations=3
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 3 items

test_some_are_slow.py ...                                            [100%]

========================= slowest 3 test durations =========================
0.30s call     test_some_are_slow.py::test_funcslow2
0.20s call     test_some_are_slow.py::test_funcslow1
0.10s call     test_some_are_slow.py::test_funcfast
============================ 3 passed in 0.12s =============================

incremental testing - test steps

Sometimes you may have a testing situation which consists of a series of test steps. If one step fails it makes no sense to execute further steps as they are all expected to fail anyway and their tracebacks add no insight. Here is a simple conftest.py file which introduces an incremental marker which is to be used on classes:

# content of conftest.py

import pytest


def pytest_runtest_makereport(item, call):
    if "incremental" in item.keywords:
        if call.excinfo is not None:
            parent = item.parent
            parent._previousfailed = item


def pytest_runtest_setup(item):
    if "incremental" in item.keywords:
        previousfailed = getattr(item.parent, "_previousfailed", None)
        if previousfailed is not None:
            pytest.xfail("previous test failed ({})".format(previousfailed.name))

These two hook implementations work together to abort incremental-marked tests in a class. Here is a test module example:

# content of test_step.py

import pytest


@pytest.mark.incremental
class TestUserHandling:
    def test_login(self):
        pass

    def test_modification(self):
        assert 0

    def test_deletion(self):
        pass


def test_normal():
    pass

If we run this:

$ pytest -rx
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 4 items

test_step.py .Fx.                                                    [100%]

================================= FAILURES =================================
____________________ TestUserHandling.test_modification ____________________

self = <test_step.TestUserHandling object at 0xdeadbeef>

    def test_modification(self):
>       assert 0
E       assert 0

test_step.py:11: AssertionError
========================= short test summary info ==========================
XFAIL test_step.py::TestUserHandling::test_deletion
  reason: previous test failed (test_modification)
================== 1 failed, 2 passed, 1 xfailed in 0.12s ==================

We’ll see that test_deletion was not executed because test_modification failed. It is reported as an “expected failure”.

Package/Directory-level fixtures (setups)

If you have nested test directories, you can have per-directory fixture scopes by placing fixture functions in a conftest.py file in that directory You can use all types of fixtures including autouse fixtures which are the equivalent of xUnit’s setup/teardown concept. It’s however recommended to have explicit fixture references in your tests or test classes rather than relying on implicitly executing setup/teardown functions, especially if they are far away from the actual tests.

Here is an example for making a db fixture available in a directory:

# content of a/conftest.py
import pytest


class DB:
    pass


@pytest.fixture(scope="session")
def db():
    return DB()

and then a test module in that directory:

# content of a/test_db.py
def test_a1(db):
    assert 0, db  # to show value

another test module:

# content of a/test_db2.py
def test_a2(db):
    assert 0, db  # to show value

and then a module in a sister directory which will not see the db fixture:

# content of b/test_error.py
def test_root(db):  # no db here, will error out
    pass

We can run this:

$ pytest
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 7 items

test_step.py .Fx.                                                    [ 57%]
a/test_db.py F                                                       [ 71%]
a/test_db2.py F                                                      [ 85%]
b/test_error.py E                                                    [100%]

================================== ERRORS ==================================
_______________________ ERROR at setup of test_root ________________________
file $REGENDOC_TMPDIR/b/test_error.py, line 1
  def test_root(db):  # no db here, will error out
E       fixture 'db' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

$REGENDOC_TMPDIR/b/test_error.py:1
================================= FAILURES =================================
____________________ TestUserHandling.test_modification ____________________

self = <test_step.TestUserHandling object at 0xdeadbeef>

    def test_modification(self):
>       assert 0
E       assert 0

test_step.py:11: AssertionError
_________________________________ test_a1 __________________________________

db = <conftest.DB object at 0xdeadbeef>

    def test_a1(db):
>       assert 0, db  # to show value
E       AssertionError: <conftest.DB object at 0xdeadbeef>
E       assert 0

a/test_db.py:2: AssertionError
_________________________________ test_a2 __________________________________

db = <conftest.DB object at 0xdeadbeef>

    def test_a2(db):
>       assert 0, db  # to show value
E       AssertionError: <conftest.DB object at 0xdeadbeef>
E       assert 0

a/test_db2.py:2: AssertionError
============= 3 failed, 2 passed, 1 xfailed, 1 error in 0.12s ==============

The two test modules in the a directory see the same db fixture instance while the one test in the sister-directory b doesn’t see it. We could of course also define a db fixture in that sister directory’s conftest.py file. Note that each fixture is only instantiated if there is a test actually needing it (unless you use “autouse” fixture which are always executed ahead of the first test executing).

post-process test reports / failures

If you want to postprocess test reports and need access to the executing environment you can implement a hook that gets called when the test “report” object is about to be created. Here we write out all failing test calls and also access a fixture (if it was used by the test) in case you want to query/look at it during your post processing. In our case we just write some information out to a failures file:

# content of conftest.py

import pytest
import os.path


@pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item, call):
    # execute all other hooks to obtain the report object
    outcome = yield
    rep = outcome.get_result()

    # we only look at actual failing test calls, not setup/teardown
    if rep.when == "call" and rep.failed:
        mode = "a" if os.path.exists("failures") else "w"
        with open("failures", mode) as f:
            # let's also access a fixture for the fun of it
            if "tmpdir" in item.fixturenames:
                extra = " ({})".format(item.funcargs["tmpdir"])
            else:
                extra = ""

            f.write(rep.nodeid + extra + "\n")

if you then have failing tests:

# content of test_module.py
def test_fail1(tmpdir):
    assert 0


def test_fail2():
    assert 0

and run them:

$ pytest test_module.py
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 2 items

test_module.py FF                                                    [100%]

================================= FAILURES =================================
________________________________ test_fail1 ________________________________

tmpdir = local('PYTEST_TMPDIR/test_fail10')

    def test_fail1(tmpdir):
>       assert 0
E       assert 0

test_module.py:2: AssertionError
________________________________ test_fail2 ________________________________

    def test_fail2():
>       assert 0
E       assert 0

test_module.py:6: AssertionError
============================ 2 failed in 0.12s =============================

you will have a “failures” file which contains the failing test ids:

$ cat failures
test_module.py::test_fail1 (PYTEST_TMPDIR/test_fail10)
test_module.py::test_fail2

Making test result information available in fixtures

If you want to make test result reports available in fixture finalizers here is a little example implemented via a local plugin:

# content of conftest.py

import pytest


@pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item, call):
    # execute all other hooks to obtain the report object
    outcome = yield
    rep = outcome.get_result()

    # set a report attribute for each phase of a call, which can
    # be "setup", "call", "teardown"

    setattr(item, "rep_" + rep.when, rep)


@pytest.fixture
def something(request):
    yield
    # request.node is an "item" because we use the default
    # "function" scope
    if request.node.rep_setup.failed:
        print("setting up a test failed!", request.node.nodeid)
    elif request.node.rep_setup.passed:
        if request.node.rep_call.failed:
            print("executing test failed", request.node.nodeid)

if you then have failing tests:

# content of test_module.py

import pytest


@pytest.fixture
def other():
    assert 0


def test_setup_fails(something, other):
    pass


def test_call_fails(something):
    assert 0


def test_fail2():
    assert 0

and run it:

$ pytest -s test_module.py
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 3 items

test_module.py Esetting up a test failed! test_module.py::test_setup_fails
Fexecuting test failed test_module.py::test_call_fails
F

================================== ERRORS ==================================
____________________ ERROR at setup of test_setup_fails ____________________

    @pytest.fixture
    def other():
>       assert 0
E       assert 0

test_module.py:7: AssertionError
================================= FAILURES =================================
_____________________________ test_call_fails ______________________________

something = None

    def test_call_fails(something):
>       assert 0
E       assert 0

test_module.py:15: AssertionError
________________________________ test_fail2 ________________________________

    def test_fail2():
>       assert 0
E       assert 0

test_module.py:19: AssertionError
======================== 2 failed, 1 error in 0.12s ========================

You’ll see that the fixture finalizers could use the precise reporting information.

PYTEST_CURRENT_TEST environment variable

Sometimes a test session might get stuck and there might be no easy way to figure out which test got stuck, for example if pytest was run in quiet mode (-q) or you don’t have access to the console output. This is particularly a problem if the problem helps only sporadically, the famous “flaky” kind of tests.

pytest sets a PYTEST_CURRENT_TEST environment variable when running tests, which can be inspected by process monitoring utilities or libraries like psutil to discover which test got stuck if necessary:

import psutil

for pid in psutil.pids():
    environ = psutil.Process(pid).environ()
    if "PYTEST_CURRENT_TEST" in environ:
        print(f'pytest process {pid} running: {environ["PYTEST_CURRENT_TEST"]}')

During the test session pytest will set PYTEST_CURRENT_TEST to the current test nodeid and the current stage, which can be setup, call and teardown.

For example, when running a single test function named test_foo from foo_module.py, PYTEST_CURRENT_TEST will be set to:

1. foo_module.py::test_foo (setup)
2. foo_module.py::test_foo (call)
3. foo_module.py::test_foo (teardown)

In that order.

Note

The contents of PYTEST_CURRENT_TEST is meant to be human readable and the actual format can be changed between releases (even bug fixes) so it shouldn’t be relied on for scripting or automation.

Freezing pytest

If you freeze your application using a tool like PyInstaller in order to distribute it to your end-users, it is a good idea to also package your test runner and run your tests using the frozen application. This way packaging errors such as dependencies not being included into the executable can be detected early while also allowing you to send test files to users so they can run them in their machines, which can be useful to obtain more information about a hard to reproduce bug.

Fortunately recent PyInstaller releases already have a custom hook for pytest, but if you are using another tool to freeze executables such as cx_freeze or py2exe, you can use pytest.freeze_includes() to obtain the full list of internal pytest modules. How to configure the tools to find the internal modules varies from tool to tool, however.

Instead of freezing the pytest runner as a separate executable, you can make your frozen program work as the pytest runner by some clever argument handling during program startup. This allows you to have a single executable, which is usually more convenient. Please note that the mechanism for plugin discovery used by pytest (setupttools entry points) doesn’t work with frozen executables so pytest can’t find any third party plugins automatically. To include third party plugins like pytest-timeout they must be imported explicitly and passed on to pytest.main.

# contents of app_main.py
import sys
import pytest_timeout  # Third party plugin

if len(sys.argv) > 1 and sys.argv[1] == "--pytest":
    import pytest

    sys.exit(pytest.main(sys.argv[2:], plugins=[pytest_timeout]))
else:
    # normal application execution: at this point argv can be parsed
    # by your argument-parsing library of choice as usual
    ...

This allows you to execute tests using the frozen application with standard pytest command-line options:

./app_main --pytest --verbose --tb=long --junitxml=results.xml test-suite/

Parametrizing tests

pytest allows to easily parametrize test functions. For basic docs, see Parametrizing fixtures and test functions.

In the following we provide some examples using the builtin mechanisms.

Generating parameters combinations, depending on command line

Let’s say we want to execute a test with different computation parameters and the parameter range shall be determined by a command line argument. Let’s first write a simple (do-nothing) computation test:

# content of test_compute.py


def test_compute(param1):
    assert param1 < 4

Now we add a test configuration like this:

# content of conftest.py


def pytest_addoption(parser):
    parser.addoption("--all", action="store_true", help="run all combinations")


def pytest_generate_tests(metafunc):
    if "param1" in metafunc.fixturenames:
        if metafunc.config.getoption("all"):
            end = 5
        else:
            end = 2
        metafunc.parametrize("param1", range(end))

This means that we only run 2 tests if we do not pass --all:

$ pytest -q test_compute.py
..                                                                   [100%]
2 passed in 0.12s

We run only two computations, so we see two dots. let’s run the full monty:

$ pytest -q --all
....F                                                                [100%]
================================= FAILURES =================================
_____________________________ test_compute[4] ______________________________

param1 = 4

    def test_compute(param1):
>       assert param1 < 4
E       assert 4 < 4

test_compute.py:4: AssertionError
1 failed, 4 passed in 0.12s

As expected when running the full range of param1 values we’ll get an error on the last one.

Different options for test IDs

pytest will build a string that is the test ID for each set of values in a parametrized test. These IDs can be used with -k to select specific cases to run, and they will also identify the specific case when one is failing. Running pytest with --collect-only will show the generated IDs.

Numbers, strings, booleans and None will have their usual string representation used in the test ID. For other objects, pytest will make a string based on the argument name:

# content of test_time.py

import pytest

from datetime import datetime, timedelta

testdata = [
    (datetime(2001, 12, 12), datetime(2001, 12, 11), timedelta(1)),
    (datetime(2001, 12, 11), datetime(2001, 12, 12), timedelta(-1)),
]


@pytest.mark.parametrize("a,b,expected", testdata)
def test_timedistance_v0(a, b, expected):
    diff = a - b
    assert diff == expected


@pytest.mark.parametrize("a,b,expected", testdata, ids=["forward", "backward"])
def test_timedistance_v1(a, b, expected):
    diff = a - b
    assert diff == expected


def idfn(val):
    if isinstance(val, (datetime,)):
        # note this wouldn't show any hours/minutes/seconds
        return val.strftime("%Y%m%d")


@pytest.mark.parametrize("a,b,expected", testdata, ids=idfn)
def test_timedistance_v2(a, b, expected):
    diff = a - b
    assert diff == expected


@pytest.mark.parametrize(
    "a,b,expected",
    [
        pytest.param(
            datetime(2001, 12, 12), datetime(2001, 12, 11), timedelta(1), id="forward"
        ),
        pytest.param(
            datetime(2001, 12, 11), datetime(2001, 12, 12), timedelta(-1), id="backward"
        ),
    ],
)
def test_timedistance_v3(a, b, expected):
    diff = a - b
    assert diff == expected

In test_timedistance_v0, we let pytest generate the test IDs.

In test_timedistance_v1, we specified ids as a list of strings which were used as the test IDs. These are succinct, but can be a pain to maintain.

In test_timedistance_v2, we specified ids as a function that can generate a string representation to make part of the test ID. So our datetime values use the label generated by idfn, but because we didn’t generate a label for timedelta objects, they are still using the default pytest representation:

$ pytest test_time.py --collect-only
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 8 items
<Module test_time.py>
  <Function test_timedistance_v0[a0-b0-expected0]>
  <Function test_timedistance_v0[a1-b1-expected1]>
  <Function test_timedistance_v1[forward]>
  <Function test_timedistance_v1[backward]>
  <Function test_timedistance_v2[20011212-20011211-expected0]>
  <Function test_timedistance_v2[20011211-20011212-expected1]>
  <Function test_timedistance_v3[forward]>
  <Function test_timedistance_v3[backward]>

========================== no tests ran in 0.12s ===========================

In test_timedistance_v3, we used pytest.param to specify the test IDs together with the actual data, instead of listing them separately.

A quick port of “testscenarios”

Here is a quick port to run tests configured with test scenarios, an add-on from Robert Collins for the standard unittest framework. We only have to work a bit to construct the correct arguments for pytest’s Metafunc.parametrize():

# content of test_scenarios.py


def pytest_generate_tests(metafunc):
    idlist = []
    argvalues = []
    for scenario in metafunc.cls.scenarios:
        idlist.append(scenario[0])
        items = scenario[1].items()
        argnames = [x[0] for x in items]
        argvalues.append([x[1] for x in items])
    metafunc.parametrize(argnames, argvalues, ids=idlist, scope="class")


scenario1 = ("basic", {"attribute": "value"})
scenario2 = ("advanced", {"attribute": "value2"})


class TestSampleWithScenarios:
    scenarios = [scenario1, scenario2]

    def test_demo1(self, attribute):
        assert isinstance(attribute, str)

    def test_demo2(self, attribute):
        assert isinstance(attribute, str)

this is a fully self-contained example which you can run with:

$ pytest test_scenarios.py
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 4 items

test_scenarios.py ....                                               [100%]

============================ 4 passed in 0.12s =============================

If you just collect tests you’ll also nicely see ‘advanced’ and ‘basic’ as variants for the test function:

$ pytest --collect-only test_scenarios.py
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 4 items
<Module test_scenarios.py>
  <Class TestSampleWithScenarios>
      <Function test_demo1[basic]>
      <Function test_demo2[basic]>
      <Function test_demo1[advanced]>
      <Function test_demo2[advanced]>

========================== no tests ran in 0.12s ===========================

Note that we told metafunc.parametrize() that your scenario values should be considered class-scoped. With pytest-2.3 this leads to a resource-based ordering.

Deferring the setup of parametrized resources

The parametrization of test functions happens at collection time. It is a good idea to setup expensive resources like DB connections or subprocess only when the actual test is run. Here is a simple example how you can achieve that. This test requires a db object fixture:

# content of test_backends.py

import pytest


def test_db_initialized(db):
    # a dummy test
    if db.__class__.__name__ == "DB2":
        pytest.fail("deliberately failing for demo purposes")

We can now add a test configuration that generates two invocations of the test_db_initialized function and also implements a factory that creates a database object for the actual test invocations:

# content of conftest.py
import pytest


def pytest_generate_tests(metafunc):
    if "db" in metafunc.fixturenames:
        metafunc.parametrize("db", ["d1", "d2"], indirect=True)


class DB1:
    "one database object"


class DB2:
    "alternative database object"


@pytest.fixture
def db(request):
    if request.param == "d1":
        return DB1()
    elif request.param == "d2":
        return DB2()
    else:
        raise ValueError("invalid internal test config")

Let’s first see how it looks like at collection time:

$ pytest test_backends.py --collect-only
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 2 items
<Module test_backends.py>
  <Function test_db_initialized[d1]>
  <Function test_db_initialized[d2]>

========================== no tests ran in 0.12s ===========================

And then when we run the test:

$ pytest -q test_backends.py
.F                                                                   [100%]
================================= FAILURES =================================
_________________________ test_db_initialized[d2] __________________________

db = <conftest.DB2 object at 0xdeadbeef>

    def test_db_initialized(db):
        # a dummy test
        if db.__class__.__name__ == "DB2":
>           pytest.fail("deliberately failing for demo purposes")
E           Failed: deliberately failing for demo purposes

test_backends.py:8: Failed
1 failed, 1 passed in 0.12s

The first invocation with db == "DB1" passed while the second with db == "DB2" failed. Our db fixture function has instantiated each of the DB values during the setup phase while the pytest_generate_tests generated two according calls to the test_db_initialized during the collection phase.

Apply indirect on particular arguments

Very often parametrization uses more than one argument name. There is opportunity to apply indirect parameter on particular arguments. It can be done by passing list or tuple of arguments’ names to indirect. In the example below there is a function test_indirect which uses two fixtures: x and y. Here we give to indirect the list, which contains the name of the fixture x. The indirect parameter will be applied to this argument only, and the value a will be passed to respective fixture function:

# content of test_indirect_list.py

import pytest


@pytest.fixture(scope="function")
def x(request):
    return request.param * 3


@pytest.fixture(scope="function")
def y(request):
    return request.param * 2


@pytest.mark.parametrize("x, y", [("a", "b")], indirect=["x"])
def test_indirect(x, y):
    assert x == "aaa"
    assert y == "b"

The result of this test will be successful:

$ pytest test_indirect_list.py --collect-only
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 1 item
<Module test_indirect_list.py>
  <Function test_indirect[a-b]>

========================== no tests ran in 0.12s ===========================

Parametrizing test methods through per-class configuration

Here is an example pytest_generate_tests function implementing a parametrization scheme similar to Michael Foord’s unittest parametrizer but in a lot less code:

# content of ./test_parametrize.py
import pytest


def pytest_generate_tests(metafunc):
    # called once per each test function
    funcarglist = metafunc.cls.params[metafunc.function.__name__]
    argnames = sorted(funcarglist[0])
    metafunc.parametrize(
        argnames, [[funcargs[name] for name in argnames] for funcargs in funcarglist]
    )


class TestClass:
    # a map specifying multiple argument sets for a test method
    params = {
        "test_equals": [dict(a=1, b=2), dict(a=3, b=3)],
        "test_zerodivision": [dict(a=1, b=0)],
    }

    def test_equals(self, a, b):
        assert a == b

    def test_zerodivision(self, a, b):
        with pytest.raises(ZeroDivisionError):
            a / b

Our test generator looks up a class-level definition which specifies which argument sets to use for each test function. Let’s run it:

$ pytest -q
F..                                                                  [100%]
================================= FAILURES =================================
________________________ TestClass.test_equals[1-2] ________________________

self = <test_parametrize.TestClass object at 0xdeadbeef>, a = 1, b = 2

    def test_equals(self, a, b):
>       assert a == b
E       assert 1 == 2

test_parametrize.py:21: AssertionError
1 failed, 2 passed in 0.12s

Indirect parametrization with multiple fixtures

Here is a stripped down real-life example of using parametrized testing for testing serialization of objects between different python interpreters. We define a test_basic_objects function which is to be run with different sets of arguments for its three arguments:

• python1: first python interpreter, run to pickle-dump an object to a file
• python2: second interpreter, run to pickle-load an object from a file
• obj: object to be dumped/loaded

"""
module containing a parametrized tests testing cross-python
serialization via the pickle module.
"""
import shutil
import subprocess
import textwrap

import pytest

pythonlist = ["python3.5", "python3.6", "python3.7"]


@pytest.fixture(params=pythonlist)
def python1(request, tmpdir):
    picklefile = tmpdir.join("data.pickle")
    return Python(request.param, picklefile)


@pytest.fixture(params=pythonlist)
def python2(request, python1):
    return Python(request.param, python1.picklefile)


class Python:
    def __init__(self, version, picklefile):
        self.pythonpath = shutil.which(version)
        if not self.pythonpath:
            pytest.skip("{!r} not found".format(version))
        self.picklefile = picklefile

    def dumps(self, obj):
        dumpfile = self.picklefile.dirpath("dump.py")
        dumpfile.write(
            textwrap.dedent(
                r"""
                import pickle
                f = open({!r}, 'wb')
                s = pickle.dump({!r}, f, protocol=2)
                f.close()
                """.format(
                    str(self.picklefile), obj
                )
            )
        )
        subprocess.check_call((self.pythonpath, str(dumpfile)))

    def load_and_is_true(self, expression):
        loadfile = self.picklefile.dirpath("load.py")
        loadfile.write(
            textwrap.dedent(
                r"""
                import pickle
                f = open({!r}, 'rb')
                obj = pickle.load(f)
                f.close()
                res = eval({!r})
                if not res:
                    raise SystemExit(1)
                """.format(
                    str(self.picklefile), expression
                )
            )
        )
        print(loadfile)
        subprocess.check_call((self.pythonpath, str(loadfile)))


@pytest.mark.parametrize("obj", [42, {}, {1: 3}])
def test_basic_objects(python1, python2, obj):
    python1.dumps(obj)
    python2.load_and_is_true("obj == {}".format(obj))

Running it results in some skips if we don’t have all the python interpreters installed and otherwise runs all combinations (3 interpreters times 3 interpreters times 3 objects to serialize/deserialize):

. $ pytest -rs -q multipython.py
...........................                                          [100%]
27 passed in 0.12s

Indirect parametrization of optional implementations/imports

If you want to compare the outcomes of several implementations of a given API, you can write test functions that receive the already imported implementations and get skipped in case the implementation is not importable/available. Let’s say we have a “base” implementation and the other (possibly optimized ones) need to provide similar results:

# content of conftest.py

import pytest


@pytest.fixture(scope="session")
def basemod(request):
    return pytest.importorskip("base")


@pytest.fixture(scope="session", params=["opt1", "opt2"])
def optmod(request):
    return pytest.importorskip(request.param)

And then a base implementation of a simple function:

# content of base.py
def func1():
    return 1

And an optimized version:

# content of opt1.py
def func1():
    return 1.0001

And finally a little test module:

# content of test_module.py


def test_func1(basemod, optmod):
    assert round(basemod.func1(), 3) == round(optmod.func1(), 3)

If you run this with reporting for skips enabled:

$ pytest -rs test_module.py
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 2 items

test_module.py .s                                                    [100%]

========================= short test summary info ==========================
SKIPPED [1] $REGENDOC_TMPDIR/conftest.py:12: could not import 'opt2': No module named 'opt2'
======================= 1 passed, 1 skipped in 0.12s =======================

You’ll see that we don’t have an opt2 module and thus the second test run of our test_func1 was skipped. A few notes:

• the fixture functions in the conftest.py file are “session-scoped” because we don’t need to import more than once
• if you have multiple test functions and a skipped import, you will see the [1] count increasing in the report
• you can put @pytest.mark.parametrize style parametrization on the test functions to parametrize input/output values as well.

Set marks or test ID for individual parametrized test

Use pytest.param to apply marks or set test ID to individual parametrized test. For example:

# content of test_pytest_param_example.py
import pytest


@pytest.mark.parametrize(
    "test_input,expected",
    [
        ("3+5", 8),
        pytest.param("1+7", 8, marks=pytest.mark.basic),
        pytest.param("2+4", 6, marks=pytest.mark.basic, id="basic_2+4"),
        pytest.param(
            "6*9", 42, marks=[pytest.mark.basic, pytest.mark.xfail], id="basic_6*9"
        ),
    ],
)
def test_eval(test_input, expected):
    assert eval(test_input) == expected

In this example, we have 4 parametrized tests. Except for the first test, we mark the rest three parametrized tests with the custom marker basic, and for the fourth test we also use the built-in mark xfail to indicate this test is expected to fail. For explicitness, we set test ids for some tests.

Then run pytest with verbose mode and with only the basic marker:

$ pytest -v -m basic
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y -- $PYTHON_PREFIX/bin/python
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collecting ... collected 17 items / 14 deselected / 3 selected

test_pytest_param_example.py::test_eval[1+7-8] PASSED                [ 33%]
test_pytest_param_example.py::test_eval[basic_2+4] PASSED            [ 66%]
test_pytest_param_example.py::test_eval[basic_6*9] XFAIL             [100%]

=============== 2 passed, 14 deselected, 1 xfailed in 0.12s ================

As the result:

• Four tests were collected
• One test was deselected because it doesn’t have the basic mark.
• Three tests with the basic mark was selected.
• The test test_eval[1+7-8] passed, but the name is autogenerated and confusing.
• The test test_eval[basic_2+4] passed.
• The test test_eval[basic_6*9] was expected to fail and did fail.

Parametrizing conditional raising

Use pytest.raises() with the pytest.mark.parametrize decorator to write parametrized tests in which some tests raise exceptions and others do not.

It is helpful to define a no-op context manager does_not_raise to serve as a complement to raises. For example:

from contextlib import contextmanager
import pytest


@contextmanager
def does_not_raise():
    yield


@pytest.mark.parametrize(
    "example_input,expectation",
    [
        (3, does_not_raise()),
        (2, does_not_raise()),
        (1, does_not_raise()),
        (0, pytest.raises(ZeroDivisionError)),
    ],
)
def test_division(example_input, expectation):
    """Test how much I know division."""
    with expectation:
        assert (6 / example_input) is not None

In the example above, the first three test cases should run unexceptionally, while the fourth should raise ZeroDivisionError.

If you’re only supporting Python 3.7+, you can simply use nullcontext to define does_not_raise:

from contextlib import nullcontext as does_not_raise

Or, if you’re supporting Python 3.3+ you can use:

from contextlib import ExitStack as does_not_raise

Or, if desired, you can pip install contextlib2 and use:

from contextlib2 import nullcontext as does_not_raise

Working with custom markers

Here are some examples using the Marking test functions with attributes mechanism.

Marking test functions and selecting them for a run

You can “mark” a test function with custom metadata like this:

# content of test_server.py

import pytest


@pytest.mark.webtest
def test_send_http():
    pass  # perform some webtest test for your app


def test_something_quick():
    pass


def test_another():
    pass


class TestClass:
    def test_method(self):
        pass

You can then restrict a test run to only run tests marked with webtest:

$ pytest -v -m webtest
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y -- $PYTHON_PREFIX/bin/python
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collecting ... collected 4 items / 3 deselected / 1 selected

test_server.py::test_send_http PASSED                                [100%]

===================== 1 passed, 3 deselected in 0.12s ======================

Or the inverse, running all tests except the webtest ones:

$ pytest -v -m "not webtest"
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y -- $PYTHON_PREFIX/bin/python
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collecting ... collected 4 items / 1 deselected / 3 selected

test_server.py::test_something_quick PASSED                          [ 33%]
test_server.py::test_another PASSED                                  [ 66%]
test_server.py::TestClass::test_method PASSED                        [100%]

===================== 3 passed, 1 deselected in 0.12s ======================

Selecting tests based on their node ID

You can provide one or more node IDs as positional arguments to select only specified tests. This makes it easy to select tests based on their module, class, method, or function name:

$ pytest -v test_server.py::TestClass::test_method
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y -- $PYTHON_PREFIX/bin/python
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collecting ... collected 1 item

test_server.py::TestClass::test_method PASSED                        [100%]

============================ 1 passed in 0.12s =============================

You can also select on the class:

$ pytest -v test_server.py::TestClass
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y -- $PYTHON_PREFIX/bin/python
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collecting ... collected 1 item

test_server.py::TestClass::test_method PASSED                        [100%]

============================ 1 passed in 0.12s =============================

Or select multiple nodes:

$ pytest -v test_server.py::TestClass test_server.py::test_send_http
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y -- $PYTHON_PREFIX/bin/python
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collecting ... collected 2 items

test_server.py::TestClass::test_method PASSED                        [ 50%]
test_server.py::test_send_http PASSED                                [100%]

============================ 2 passed in 0.12s =============================

Note

Node IDs are of the form module.py::class::method or module.py::function. Node IDs control which tests are collected, so module.py::class will select all test methods on the class. Nodes are also created for each parameter of a parametrized fixture or test, so selecting a parametrized test must include the parameter value, e.g. module.py::function[param].

Node IDs for failing tests are displayed in the test summary info when running pytest with the -rf option. You can also construct Node IDs from the output of pytest --collectonly.

Using -k expr to select tests based on their name

You can use the -k command line option to specify an expression which implements a substring match on the test names instead of the exact match on markers that -m provides. This makes it easy to select tests based on their names:

$ pytest -v -k http  # running with the above defined example module
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y -- $PYTHON_PREFIX/bin/python
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collecting ... collected 4 items / 3 deselected / 1 selected

test_server.py::test_send_http PASSED                                [100%]

===================== 1 passed, 3 deselected in 0.12s ======================

And you can also run all tests except the ones that match the keyword:

$ pytest -k "not send_http" -v
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y -- $PYTHON_PREFIX/bin/python
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collecting ... collected 4 items / 1 deselected / 3 selected

test_server.py::test_something_quick PASSED                          [ 33%]
test_server.py::test_another PASSED                                  [ 66%]
test_server.py::TestClass::test_method PASSED                        [100%]

===================== 3 passed, 1 deselected in 0.12s ======================

Or to select “http” and “quick” tests:

$ pytest -k "http or quick" -v
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y -- $PYTHON_PREFIX/bin/python
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collecting ... collected 4 items / 2 deselected / 2 selected

test_server.py::test_send_http PASSED                                [ 50%]
test_server.py::test_something_quick PASSED                          [100%]

===================== 2 passed, 2 deselected in 0.12s ======================

Note

If you are using expressions such as "X and Y" then both X and Y need to be simple non-keyword names. For example, "pass" or "from" will result in SyntaxErrors because "-k" evaluates the expression using Python’s eval function.

However, if the "-k" argument is a simple string, no such restrictions apply. Also "-k 'not STRING'" has no restrictions. You can also specify numbers like "-k 1.3" to match tests which are parametrized with the float "1.3".

Registering markers

Registering markers for your test suite is simple:

# content of pytest.ini
[pytest]
markers =
    webtest: mark a test as a webtest.

You can ask which markers exist for your test suite - the list includes our just defined webtest markers:

$ pytest --markers
@pytest.mark.webtest: mark a test as a webtest.

@pytest.mark.filterwarnings(warning): add a warning filter to the given test. see https://docs.pytest.org/en/latest/warnings.html#pytest-mark-filterwarnings

@pytest.mark.skip(reason=None): skip the given test function with an optional reason. Example: skip(reason="no way of currently testing this") skips the test.

@pytest.mark.skipif(condition): skip the given test function if eval(condition) results in a True value.  Evaluation happens within the module global context. Example: skipif('sys.platform == "win32"') skips the test if we are on the win32 platform. see https://docs.pytest.org/en/latest/skipping.html

@pytest.mark.xfail(condition, reason=None, run=True, raises=None, strict=False): mark the test function as an expected failure if eval(condition) has a True value. Optionally specify a reason for better reporting and run=False if you don't even want to execute the test function. If only specific exception(s) are expected, you can list them in raises, and if the test fails in other ways, it will be reported as a true failure. See https://docs.pytest.org/en/latest/skipping.html

@pytest.mark.parametrize(argnames, argvalues): call a test function multiple times passing in different arguments in turn. argvalues generally needs to be a list of values if argnames specifies only one name or a list of tuples of values if argnames specifies multiple names. Example: @parametrize('arg1', [1,2]) would lead to two calls of the decorated test function, one with arg1=1 and another with arg1=2.see https://docs.pytest.org/en/latest/parametrize.html for more info and examples.

@pytest.mark.usefixtures(fixturename1, fixturename2, ...): mark tests as needing all of the specified fixtures. see https://docs.pytest.org/en/latest/fixture.html#usefixtures

@pytest.mark.tryfirst: mark a hook implementation function such that the plugin machinery will try to call it first/as early as possible.

@pytest.mark.trylast: mark a hook implementation function such that the plugin machinery will try to call it last/as late as possible.

For an example on how to add and work with markers from a plugin, see Custom marker and command line option to control test runs.

Note

It is recommended to explicitly register markers so that:

• There is one place in your test suite defining your markers
• Asking for existing markers via pytest --markers gives good output
• Typos in function markers are treated as an error if you use the --strict-markers option.

Marking whole classes or modules

You may use pytest.mark decorators with classes to apply markers to all of its test methods:

# content of test_mark_classlevel.py
import pytest


@pytest.mark.webtest
class TestClass:
    def test_startup(self):
        pass

    def test_startup_and_more(self):
        pass

This is equivalent to directly applying the decorator to the two test functions.

Due to legacy reasons, it is possible to set the pytestmark attribute on a TestClass like this:

import pytest


class TestClass:
    pytestmark = pytest.mark.webtest

or if you need to use multiple markers you can use a list:

import pytest


class TestClass:
    pytestmark = [pytest.mark.webtest, pytest.mark.slowtest]

You can also set a module level marker:

import pytest
pytestmark = pytest.mark.webtest

or multiple markers:

pytestmark = [pytest.mark.webtest, pytest.mark.slowtest]

in which case markers will be applied (in left-to-right order) to all functions and methods defined in the module.

Marking individual tests when using parametrize

When using parametrize, applying a mark will make it apply to each individual test. However it is also possible to apply a marker to an individual test instance:

import pytest


@pytest.mark.foo
@pytest.mark.parametrize(
    ("n", "expected"), [(1, 2), pytest.param(1, 3, marks=pytest.mark.bar), (2, 3)]
)
def test_increment(n, expected):
    assert n + 1 == expected

In this example the mark “foo” will apply to each of the three tests, whereas the “bar” mark is only applied to the second test. Skip and xfail marks can also be applied in this way, see Skip/xfail with parametrize.

Custom marker and command line option to control test runs

Plugins can provide custom markers and implement specific behaviour based on it. This is a self-contained example which adds a command line option and a parametrized test function marker to run tests specifies via named environments:

# content of conftest.py

import pytest


def pytest_addoption(parser):
    parser.addoption(
        "-E",
        action="store",
        metavar="NAME",
        help="only run tests matching the environment NAME.",
    )


def pytest_configure(config):
    # register an additional marker
    config.addinivalue_line(
        "markers", "env(name): mark test to run only on named environment"
    )


def pytest_runtest_setup(item):
    envnames = [mark.args[0] for mark in item.iter_markers(name="env")]
    if envnames:
        if item.config.getoption("-E") not in envnames:
            pytest.skip("test requires env in {!r}".format(envnames))

A test file using this local plugin:

# content of test_someenv.py

import pytest


@pytest.mark.env("stage1")
def test_basic_db_operation():
    pass

and an example invocations specifying a different environment than what the test needs:

$ pytest -E stage2
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 1 item

test_someenv.py s                                                    [100%]

============================ 1 skipped in 0.12s ============================

and here is one that specifies exactly the environment needed:

$ pytest -E stage1
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 1 item

test_someenv.py .                                                    [100%]

============================ 1 passed in 0.12s =============================

The --markers option always gives you a list of available markers:

$ pytest --markers
@pytest.mark.env(name): mark test to run only on named environment

@pytest.mark.filterwarnings(warning): add a warning filter to the given test. see https://docs.pytest.org/en/latest/warnings.html#pytest-mark-filterwarnings

@pytest.mark.skip(reason=None): skip the given test function with an optional reason. Example: skip(reason="no way of currently testing this") skips the test.

@pytest.mark.skipif(condition): skip the given test function if eval(condition) results in a True value.  Evaluation happens within the module global context. Example: skipif('sys.platform == "win32"') skips the test if we are on the win32 platform. see https://docs.pytest.org/en/latest/skipping.html

@pytest.mark.xfail(condition, reason=None, run=True, raises=None, strict=False): mark the test function as an expected failure if eval(condition) has a True value. Optionally specify a reason for better reporting and run=False if you don't even want to execute the test function. If only specific exception(s) are expected, you can list them in raises, and if the test fails in other ways, it will be reported as a true failure. See https://docs.pytest.org/en/latest/skipping.html

@pytest.mark.parametrize(argnames, argvalues): call a test function multiple times passing in different arguments in turn. argvalues generally needs to be a list of values if argnames specifies only one name or a list of tuples of values if argnames specifies multiple names. Example: @parametrize('arg1', [1,2]) would lead to two calls of the decorated test function, one with arg1=1 and another with arg1=2.see https://docs.pytest.org/en/latest/parametrize.html for more info and examples.

@pytest.mark.usefixtures(fixturename1, fixturename2, ...): mark tests as needing all of the specified fixtures. see https://docs.pytest.org/en/latest/fixture.html#usefixtures

@pytest.mark.tryfirst: mark a hook implementation function such that the plugin machinery will try to call it first/as early as possible.

@pytest.mark.trylast: mark a hook implementation function such that the plugin machinery will try to call it last/as late as possible.

Passing a callable to custom markers

Below is the config file that will be used in the next examples:

# content of conftest.py
import sys


def pytest_runtest_setup(item):
    for marker in item.iter_markers(name="my_marker"):
        print(marker)
        sys.stdout.flush()

A custom marker can have its argument set, i.e. args and kwargs properties, defined by either invoking it as a callable or using pytest.mark.MARKER_NAME.with_args. These two methods achieve the same effect most of the time.

However, if there is a callable as the single positional argument with no keyword arguments, using the pytest.mark.MARKER_NAME(c) will not pass c as a positional argument but decorate c with the custom marker (see MarkDecorator). Fortunately, pytest.mark.MARKER_NAME.with_args comes to the rescue:

# content of test_custom_marker.py
import pytest


def hello_world(*args, **kwargs):
    return "Hello World"


@pytest.mark.my_marker.with_args(hello_world)
def test_with_args():
    pass

The output is as follows:

$ pytest -q -s
Mark(name='my_marker', args=(<function hello_world at 0xdeadbeef>,), kwargs={})
.
1 passed in 0.12s

We can see that the custom marker has its argument set extended with the function hello_world. This is the key difference between creating a custom marker as a callable, which invokes __call__ behind the scenes, and using with_args.

Reading markers which were set from multiple places

If you are heavily using markers in your test suite you may encounter the case where a marker is applied several times to a test function. From plugin code you can read over all such settings. Example:

# content of test_mark_three_times.py
import pytest

pytestmark = pytest.mark.glob("module", x=1)


@pytest.mark.glob("class", x=2)
class TestClass:
    @pytest.mark.glob("function", x=3)
    def test_something(self):
        pass

Here we have the marker “glob” applied three times to the same test function. From a conftest file we can read it like this:

# content of conftest.py
import sys


def pytest_runtest_setup(item):
    for mark in item.iter_markers(name="glob"):
        print("glob args={} kwargs={}".format(mark.args, mark.kwargs))
        sys.stdout.flush()

Let’s run this without capturing output and see what we get:

$ pytest -q -s
glob args=('function',) kwargs={'x': 3}
glob args=('class',) kwargs={'x': 2}
glob args=('module',) kwargs={'x': 1}
.
1 passed in 0.12s

marking platform specific tests with pytest

Consider you have a test suite which marks tests for particular platforms, namely pytest.mark.darwin, pytest.mark.win32 etc. and you also have tests that run on all platforms and have no specific marker. If you now want to have a way to only run the tests for your particular platform, you could use the following plugin:

# content of conftest.py
#
import sys
import pytest

ALL = set("darwin linux win32".split())


def pytest_runtest_setup(item):
    supported_platforms = ALL.intersection(mark.name for mark in item.iter_markers())
    plat = sys.platform
    if supported_platforms and plat not in supported_platforms:
        pytest.skip("cannot run on platform {}".format(plat))

then tests will be skipped if they were specified for a different platform. Let’s do a little test file to show how this looks like:

# content of test_plat.py

import pytest


@pytest.mark.darwin
def test_if_apple_is_evil():
    pass


@pytest.mark.linux
def test_if_linux_works():
    pass


@pytest.mark.win32
def test_if_win32_crashes():
    pass


def test_runs_everywhere():
    pass

then you will see two tests skipped and two executed tests as expected:

$ pytest -rs # this option reports skip reasons
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 4 items

test_plat.py s.s.                                                    [100%]

========================= short test summary info ==========================
SKIPPED [2] $REGENDOC_TMPDIR/conftest.py:12: cannot run on platform linux
======================= 2 passed, 2 skipped in 0.12s =======================

Note that if you specify a platform via the marker-command line option like this:

$ pytest -m linux
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 4 items / 3 deselected / 1 selected

test_plat.py .                                                       [100%]

===================== 1 passed, 3 deselected in 0.12s ======================

then the unmarked-tests will not be run. It is thus a way to restrict the run to the specific tests.

Automatically adding markers based on test names

If you a test suite where test function names indicate a certain type of test, you can implement a hook that automatically defines markers so that you can use the -m option with it. Let’s look at this test module:

# content of test_module.py


def test_interface_simple():
    assert 0


def test_interface_complex():
    assert 0


def test_event_simple():
    assert 0


def test_something_else():
    assert 0

We want to dynamically define two markers and can do it in a conftest.py plugin:

# content of conftest.py

import pytest


def pytest_collection_modifyitems(items):
    for item in items:
        if "interface" in item.nodeid:
            item.add_marker(pytest.mark.interface)
        elif "event" in item.nodeid:
            item.add_marker(pytest.mark.event)

We can now use the -m option to select one set:

$ pytest -m interface --tb=short
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 4 items / 2 deselected / 2 selected

test_module.py FF                                                    [100%]

================================= FAILURES =================================
__________________________ test_interface_simple ___________________________
test_module.py:4: in test_interface_simple
    assert 0
E   assert 0
__________________________ test_interface_complex __________________________
test_module.py:8: in test_interface_complex
    assert 0
E   assert 0
===================== 2 failed, 2 deselected in 0.12s ======================

or to select both “event” and “interface” tests:

$ pytest -m "interface or event" --tb=short
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 4 items / 1 deselected / 3 selected

test_module.py FFF                                                   [100%]

================================= FAILURES =================================
__________________________ test_interface_simple ___________________________
test_module.py:4: in test_interface_simple
    assert 0
E   assert 0
__________________________ test_interface_complex __________________________
test_module.py:8: in test_interface_complex
    assert 0
E   assert 0
____________________________ test_event_simple _____________________________
test_module.py:12: in test_event_simple
    assert 0
E   assert 0
===================== 3 failed, 1 deselected in 0.12s ======================

A session-fixture which can look at all collected tests

A session-scoped fixture effectively has access to all collected test items. Here is an example of a fixture function which walks all collected tests and looks if their test class defines a callme method and calls it:

# content of conftest.py

import pytest


@pytest.fixture(scope="session", autouse=True)
def callattr_ahead_of_alltests(request):
    print("callattr_ahead_of_alltests called")
    seen = {None}
    session = request.node
    for item in session.items:
        cls = item.getparent(pytest.Class)
        if cls not in seen:
            if hasattr(cls.obj, "callme"):
                cls.obj.callme()
            seen.add(cls)

test classes may now define a callme method which will be called ahead of running any tests:

# content of test_module.py


class TestHello:
    @classmethod
    def callme(cls):
        print("callme called!")

    def test_method1(self):
        print("test_method1 called")

    def test_method2(self):
        print("test_method1 called")


class TestOther:
    @classmethod
    def callme(cls):
        print("callme other called")

    def test_other(self):
        print("test other")


# works with unittest as well ...
import unittest


class SomeTest(unittest.TestCase):
    @classmethod
    def callme(self):
        print("SomeTest callme called")

    def test_unit1(self):
        print("test_unit1 method called")

If you run this without output capturing:

$ pytest -q -s test_module.py
callattr_ahead_of_alltests called
callme called!
callme other called
SomeTest callme called
test_method1 called
.test_method1 called
.test other
.test_unit1 method called
.
4 passed in 0.12s

Changing standard (Python) test discovery

Ignore paths during test collection

You can easily ignore certain test directories and modules during collection by passing the --ignore=path option on the cli. pytest allows multiple --ignore options. Example:

tests/
|-- example
|   |-- test_example_01.py
|   |-- test_example_02.py
|   '-- test_example_03.py
|-- foobar
|   |-- test_foobar_01.py
|   |-- test_foobar_02.py
|   '-- test_foobar_03.py
'-- hello
    '-- world
        |-- test_world_01.py
        |-- test_world_02.py
        '-- test_world_03.py

Now if you invoke pytest with --ignore=tests/foobar/test_foobar_03.py --ignore=tests/hello/, you will see that pytest only collects test-modules, which do not match the patterns specified:

=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
rootdir: $REGENDOC_TMPDIR, inifile:
collected 5 items

tests/example/test_example_01.py .                                   [ 20%]
tests/example/test_example_02.py .                                   [ 40%]
tests/example/test_example_03.py .                                   [ 60%]
tests/foobar/test_foobar_01.py .                                     [ 80%]
tests/foobar/test_foobar_02.py .                                     [100%]

========================= 5 passed in 0.02 seconds =========================

The --ignore-glob option allows to ignore test file paths based on Unix shell-style wildcards. If you want to exclude test-modules that end with _01.py, execute pytest with --ignore-glob='*_01.py'.

Deselect tests during test collection

Tests can individually be deselected during collection by passing the --deselect=item option. For example, say tests/foobar/test_foobar_01.py contains test_a and test_b. You can run all of the tests within tests/ except for tests/foobar/test_foobar_01.py::test_a by invoking pytest with --deselect tests/foobar/test_foobar_01.py::test_a. pytest allows multiple --deselect options.

Keeping duplicate paths specified from command line

Default behavior of pytest is to ignore duplicate paths specified from the command line. Example:

pytest path_a path_a

...
collected 1 item
...

Just collect tests once.

To collect duplicate tests, use the --keep-duplicates option on the cli. Example:

pytest --keep-duplicates path_a path_a

...
collected 2 items
...

As the collector just works on directories, if you specify twice a single test file, pytest will still collect it twice, no matter if the --keep-duplicates is not specified. Example:

pytest test_a.py test_a.py

...
collected 2 items
...

Changing directory recursion

You can set the norecursedirs option in an ini-file, for example your pytest.ini in the project root directory:

# content of pytest.ini
[pytest]
norecursedirs = .svn _build tmp*

This would tell pytest to not recurse into typical subversion or sphinx-build directories or into any tmp prefixed directory.

Changing naming conventions

You can configure different naming conventions by setting the python_files, python_classes and python_functions configuration options. Here is an example:

# content of pytest.ini
# Example 1: have pytest look for "check" instead of "test"
# can also be defined in tox.ini or setup.cfg file, although the section
# name in setup.cfg files should be "tool:pytest"
[pytest]
python_files = check_*.py
python_classes = Check
python_functions = *_check

This would make pytest look for tests in files that match the check_* .py glob-pattern, Check prefixes in classes, and functions and methods that match *_check. For example, if we have:

# content of check_myapp.py
class CheckMyApp:
    def simple_check(self):
        pass

    def complex_check(self):
        pass

The test collection would look like this:

$ pytest --collect-only
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR, inifile: pytest.ini
collected 2 items
<Module check_myapp.py>
  <Class CheckMyApp>
      <Function simple_check>
      <Function complex_check>

========================== no tests ran in 0.12s ===========================

You can check for multiple glob patterns by adding a space between the patterns:

# Example 2: have pytest look for files with "test" and "example"
# content of pytest.ini, tox.ini, or setup.cfg file (replace "pytest"
# with "tool:pytest" for setup.cfg)
[pytest]
python_files = test_*.py example_*.py

Note

the python_functions and python_classes options has no effect for unittest.TestCase test discovery because pytest delegates discovery of test case methods to unittest code.

Interpreting cmdline arguments as Python packages

You can use the --pyargs option to make pytest try interpreting arguments as python package names, deriving their file system path and then running the test. For example if you have unittest2 installed you can type:

pytest --pyargs unittest2.test.test_skipping -q

which would run the respective test module. Like with other options, through an ini-file and the addopts option you can make this change more permanently:

# content of pytest.ini
[pytest]
addopts = --pyargs

Now a simple invocation of pytest NAME will check if NAME exists as an importable package/module and otherwise treat it as a filesystem path.

Finding out what is collected

You can always peek at the collection tree without running tests like this:

. $ pytest --collect-only pythoncollection.py
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR, inifile: pytest.ini
collected 3 items
<Module CWD/pythoncollection.py>
  <Function test_function>
  <Class TestClass>
      <Function test_method>
      <Function test_anothermethod>

========================== no tests ran in 0.12s ===========================

Customizing test collection

You can easily instruct pytest to discover tests from every Python file:

# content of pytest.ini
[pytest]
python_files = *.py

However, many projects will have a setup.py which they don’t want to be imported. Moreover, there may files only importable by a specific python version. For such cases you can dynamically define files to be ignored by listing them in a conftest.py file:

# content of conftest.py
import sys

collect_ignore = ["setup.py"]
if sys.version_info[0] > 2:
    collect_ignore.append("pkg/module_py2.py")

and then if you have a module file like this:

# content of pkg/module_py2.py
def test_only_on_python2():
    try:
        assert 0
    except Exception, e:
        pass

and a setup.py dummy file like this:

# content of setup.py
0 / 0  # will raise exception if imported

If you run with a Python 2 interpreter then you will find the one test and will leave out the setup.py file:

#$ pytest --collect-only
====== test session starts ======
platform linux2 -- Python 2.7.10, pytest-2.9.1, py-1.4.31, pluggy-0.3.1
rootdir: $REGENDOC_TMPDIR, inifile: pytest.ini
collected 1 items
<Module 'pkg/module_py2.py'>
  <Function 'test_only_on_python2'>

====== no tests ran in 0.04 seconds ======

If you run with a Python 3 interpreter both the one test and the setup.py file will be left out:

$ pytest --collect-only
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR, inifile: pytest.ini
collected 0 items

========================== no tests ran in 0.12s ===========================

It’s also possible to ignore files based on Unix shell-style wildcards by adding patterns to collect_ignore_glob.

The following example conftest.py ignores the file setup.py and in addition all files that end with *_py2.py when executed with a Python 3 interpreter:

# content of conftest.py
import sys

collect_ignore = ["setup.py"]
if sys.version_info[0] > 2:
    collect_ignore_glob = ["*_py2.py"]

Working with non-python tests

A basic example for specifying tests in Yaml files

Here is an example conftest.py (extracted from Ali Afshnars special purpose pytest-yamlwsgi plugin). This conftest.py will collect test*.yaml files and will execute the yaml-formatted content as custom tests:

# content of conftest.py
import pytest


def pytest_collect_file(parent, path):
    if path.ext == ".yaml" and path.basename.startswith("test"):
        return YamlFile(path, parent)


class YamlFile(pytest.File):
    def collect(self):
        import yaml  # we need a yaml parser, e.g. PyYAML

        raw = yaml.safe_load(self.fspath.open())
        for name, spec in sorted(raw.items()):
            yield YamlItem(name, self, spec)


class YamlItem(pytest.Item):
    def __init__(self, name, parent, spec):
        super().__init__(name, parent)
        self.spec = spec

    def runtest(self):
        for name, value in sorted(self.spec.items()):
            # some custom test execution (dumb example follows)
            if name != value:
                raise YamlException(self, name, value)

    def repr_failure(self, excinfo):
        """ called when self.runtest() raises an exception. """
        if isinstance(excinfo.value, YamlException):
            return "\n".join(
                [
                    "usecase execution failed",
                    "   spec failed: {1!r}: {2!r}".format(*excinfo.value.args),
                    "   no further details known at this point.",
                ]
            )

    def reportinfo(self):
        return self.fspath, 0, "usecase: {}".format(self.name)


class YamlException(Exception):
    """ custom exception for error reporting. """

You can create a simple example file:

# test_simple.yaml
ok:
    sub1: sub1

hello:
    world: world
    some: other

and if you installed PyYAML or a compatible YAML-parser you can now execute the test specification:

nonpython $ pytest test_simple.yaml
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR/nonpython
collected 2 items

test_simple.yaml F.                                                  [100%]

================================= FAILURES =================================
______________________________ usecase: hello ______________________________
usecase execution failed
   spec failed: 'some': 'other'
   no further details known at this point.
======================= 1 failed, 1 passed in 0.12s ========================

You get one dot for the passing sub1: sub1 check and one failure. Obviously in the above conftest.py you’ll want to implement a more interesting interpretation of the yaml-values. You can easily write your own domain specific testing language this way.

Note

repr_failure(excinfo) is called for representing test failures. If you create custom collection nodes you can return an error representation string of your choice. It will be reported as a (red) string.

reportinfo() is used for representing the test location and is also consulted when reporting in verbose mode:

nonpython $ pytest -v
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y -- $PYTHON_PREFIX/bin/python
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR/nonpython
collecting ... collected 2 items

test_simple.yaml::hello FAILED                                       [ 50%]
test_simple.yaml::ok PASSED                                          [100%]

================================= FAILURES =================================
______________________________ usecase: hello ______________________________
usecase execution failed
   spec failed: 'some': 'other'
   no further details known at this point.
======================= 1 failed, 1 passed in 0.12s ========================

While developing your custom test collection and execution it’s also interesting to just look at the collection tree:

nonpython $ pytest --collect-only
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR/nonpython
collected 2 items
<Package $REGENDOC_TMPDIR/nonpython>
  <YamlFile test_simple.yaml>
    <YamlItem hello>
    <YamlItem ok>

========================== no tests ran in 0.12s ===========================

Setting up bash completion

When using bash as your shell, pytest can use argcomplete (https://argcomplete.readthedocs.io/) for auto-completion. For this argcomplete needs to be installed and enabled.

Install argcomplete using:

sudo pip install 'argcomplete>=0.5.7'

For global activation of all argcomplete enabled python applications run:

sudo activate-global-python-argcomplete

For permanent (but not global) pytest activation, use:

register-python-argcomplete pytest >> ~/.bashrc

For one-time activation of argcomplete for pytest only, use:

eval "$(register-python-argcomplete pytest)"

Some Issues and Questions

Note

This FAQ is here only mostly for historic reasons. Checkout pytest Q&A at Stackoverflow for many questions and answers related to pytest and/or use Contact channels to get help.

On naming, nosetests, licensing and magic

How does pytest relate to nose and unittest?

pytest and nose share basic philosophy when it comes to running and writing Python tests. In fact, you can run many tests written for nose with pytest. nose was originally created as a clone of pytest when pytest was in the 0.8 release cycle. Note that starting with pytest-2.0 support for running unittest test suites is majorly improved.

how does pytest relate to twisted’s trial?

Since some time pytest has builtin support for supporting tests written using trial. It does not itself start a reactor, however, and does not handle Deferreds returned from a test in pytest style. If you are using trial’s unittest.TestCase chances are that you can just run your tests even if you return Deferreds. In addition, there also is a dedicated pytest-twisted plugin which allows you to return deferreds from pytest-style tests, allowing the use of pytest fixtures: explicit, modular, scalable and other features.

how does pytest work with Django?

In 2012, some work is going into the pytest-django plugin. It substitutes the usage of Django’s manage.py test and allows the use of all pytest features most of which are not available from Django directly.

What’s this “magic” with pytest? (historic notes)

Around 2007 (version 0.8) some people thought that pytest was using too much “magic”. It had been part of the pylib which contains a lot of unrelated python library code. Around 2010 there was a major cleanup refactoring, which removed unused or deprecated code and resulted in the new pytest PyPI package which strictly contains only test-related code. This release also brought a complete pluginification such that the core is around 300 lines of code and everything else is implemented in plugins. Thus pytest today is a small, universally runnable and customizable testing framework for Python. Note, however, that pytest uses metaprogramming techniques and reading its source is thus likely not something for Python beginners.

A second “magic” issue was the assert statement debugging feature. Nowadays, pytest explicitly rewrites assert statements in test modules in order to provide more useful assert feedback. This completely avoids previous issues of confusing assertion-reporting. It also means, that you can use Python’s -O optimization without losing assertions in test modules.

You can also turn off all assertion interaction using the --assert=plain option.

Why can I use both pytest and py.test commands?

pytest used to be part of the py package, which provided several developer utilities, all starting with py.<TAB>, thus providing nice TAB-completion. If you install pip install pycmd you get these tools from a separate package. Once pytest became a separate package, the py.test name was retained due to avoid a naming conflict with another tool. This conflict was eventually resolved, and the pytest command was therefore introduced. In future versions of pytest, we may deprecate and later remove the py.test command to avoid perpetuating the confusion.

pytest fixtures, parametrized tests

Is using pytest fixtures versus xUnit setup a style question?

For simple applications and for people experienced with nose or unittest-style test setup using xUnit style setup probably feels natural. For larger test suites, parametrized testing or setup of complex test resources using fixtures may feel more natural. Moreover, fixtures are ideal for writing advanced test support code (like e.g. the monkeypatch, the tmpdir or capture fixtures) because the support code can register setup/teardown functions in a managed class/module/function scope.

Can I yield multiple values from a fixture function?

There are two conceptual reasons why yielding from a factory function is not possible:

• If multiple factories yielded values there would be no natural place to determine the combination policy - in real-world examples some combinations often should not run.
• Calling factories for obtaining test function arguments is part of setting up and running a test. At that point it is not possible to add new test calls to the test collection anymore.

However, with pytest-2.3 you can use the Fixtures as Function arguments decorator and specify params so that all tests depending on the factory-created resource will run multiple times with different parameters.

You can also use the pytest_generate_tests hook to implement the parametrization scheme of your choice. See also Parametrizing tests for more examples.

pytest interaction with other packages

Issues with pytest, multiprocess and setuptools?

On Windows the multiprocess package will instantiate sub processes by pickling and thus implicitly re-import a lot of local modules. Unfortunately, setuptools-0.6.11 does not if __name__=='__main__' protect its generated command line script. This leads to infinite recursion when running a test that instantiates Processes.

As of mid-2013, there shouldn’t be a problem anymore when you use the standard setuptools (note that distribute has been merged back into setuptools which is now shipped directly with virtualenv).

Backwards Compatibility Policy

Keeping backwards compatibility has a very high priority in the pytest project. Although we have deprecated functionality over the years, most of it is still supported. All deprecations in pytest were done because simpler or more efficient ways of accomplishing the same tasks have emerged, making the old way of doing things unnecessary.

With the pytest 3.0 release we introduced a clear communication scheme for when we will actually remove the old busted joint and politely ask you to use the new hotness instead, while giving you enough time to adjust your tests or raise concerns if there are valid reasons to keep deprecated functionality around.

To communicate changes we issue deprecation warnings using a custom warning hierarchy (see Internal pytest warnings). These warnings may be suppressed using the standard means: -W command-line flag or filterwarnings ini options (see Warnings Capture), but we suggest to use these sparingly and temporarily, and heed the warnings when possible.

We will only start the removal of deprecated functionality in major releases (e.g. if we deprecate something in 3.0 we will start to remove it in 4.0), and keep it around for at least two minor releases (e.g. if we deprecate something in 3.9 and 4.0 is the next release, we start to remove it in 5.0, not in 4.0).

When the deprecation expires (e.g. 4.0 is released), we won’t remove the deprecated functionality immediately, but will use the standard warning filters to turn them into errors by default. This approach makes it explicit that removal is imminent, and still gives you time to turn the deprecated feature into a warning instead of an error so it can be dealt with in your own time. In the next minor release (e.g. 4.1), the feature will be effectively removed.

Deprecation Roadmap

Features currently deprecated and removed in previous releases can be found in Deprecations and Removals.

We track future deprecation and removal of features using milestones and the deprecation and removal labels on GitHub.

Deprecations and Removals

This page lists all pytest features that are currently deprecated or have been removed in past major releases. The objective is to give users a clear rationale why a certain feature has been removed, and what alternatives should be used instead.

Deprecated Features

Below is a complete list of all pytest features which are considered deprecated. Using those features will issue _pytest.warning_types.PytestWarning or subclasses, which can be filtered using standard warning filters.

junit_family default value change to “xunit2”

Deprecated since version 5.2.

The default value of junit_family option will change to xunit2 in pytest 6.0, given that this is the version supported by default in modern tools that manipulate this type of file.

In order to smooth the transition, pytest will issue a warning in case the --junitxml option is given in the command line but junit_family is not explicitly configured in pytest.ini:

PytestDeprecationWarning: The 'junit_family' default value will change to 'xunit2' in pytest 6.0.
  Add 'junit_family=legacy' to your pytest.ini file to silence this warning and make your suite compatible.

In order to silence this warning, users just need to configure the junit_family option explicitly:

[pytest]
junit_family=legacy

funcargnames alias for fixturenames

Deprecated since version 5.0.

The FixtureRequest, Metafunc, and Function classes track the names of their associated fixtures, with the aptly-named fixturenames attribute.

Prior to pytest 2.3, this attribute was named funcargnames, and we have kept that as an alias since. It is finally due for removal, as it is often confusing in places where we or plugin authors must distinguish between fixture names and names supplied by non-fixture things such as pytest.mark.parametrize.

Result log (--result-log)

Deprecated since version 4.0.

The --result-log option produces a stream of test reports which can be analysed at runtime, but it uses a custom format which requires users to implement their own parser.

The pytest-reportlog plugin provides a --report-log option, a more standard and extensible alternative, producing one JSON object per-line, and should cover the same use cases. Please try it out and provide feedback.

The plan is remove the --result-log option in pytest 6.0 if pytest-reportlog proves satisfactory to all users and is deemed stable. The pytest-reportlog plugin might even be merged into the core at some point, depending on the plans for the plugins and number of users using it.

Removed Features

As stated in our Backwards Compatibility Policy policy, deprecated features are removed only in major releases after an appropriate period of deprecation has passed.

pytest.config global

Removed in version 5.0.

The pytest.config global object is deprecated. Instead use request.config (via the request fixture) or if you are a plugin author use the pytest_configure(config) hook. Note that many hooks can also access the config object indirectly, through session.config or item.config for example.

"message" parameter of pytest.raises

Removed in version 5.0.

It is a common mistake to think this parameter will match the exception message, while in fact it only serves to provide a custom message in case the pytest.raises check fails. To prevent users from making this mistake, and because it is believed to be little used, pytest is deprecating it without providing an alternative for the moment.

If you have a valid use case for this parameter, consider that to obtain the same results you can just call pytest.fail manually at the end of the with statement.

For example:

with pytest.raises(TimeoutError, message="Client got unexpected message"):
    wait_for(websocket.recv(), 0.5)

Becomes:

with pytest.raises(TimeoutError):
    wait_for(websocket.recv(), 0.5)
    pytest.fail("Client got unexpected message")

If you still have concerns about this deprecation and future removal, please comment on issue #3974.

raises / warns with a string as the second argument

Removed in version 5.0.

Use the context manager form of these instead. When necessary, invoke exec directly.

Example:

pytest.raises(ZeroDivisionError, "1 / 0")
pytest.raises(SyntaxError, "a $ b")

pytest.warns(DeprecationWarning, "my_function()")
pytest.warns(SyntaxWarning, "assert(1, 2)")

Becomes:

with pytest.raises(ZeroDivisionError):
    1 / 0
with pytest.raises(SyntaxError):
    exec("a $ b")  # exec is required for invalid syntax

with pytest.warns(DeprecationWarning):
    my_function()
with pytest.warns(SyntaxWarning):
    exec("assert(1, 2)")  # exec is used to avoid a top-level warning

Using Class in custom Collectors

Removed in version 4.0.

Using objects named "Class" as a way to customize the type of nodes that are collected in Collector subclasses has been deprecated. Users instead should use pytest_pycollect_makeitem to customize node types during collection.

This issue should affect only advanced plugins who create new collection types, so if you see this warning message please contact the authors so they can change the code.

marks in pytest.mark.parametrize

Removed in version 4.0.

Applying marks to values of a pytest.mark.parametrize call is now deprecated. For example:

@pytest.mark.parametrize(
    "a, b",
    [
        (3, 9),
        pytest.mark.xfail(reason="flaky")(6, 36),
        (10, 100),
        (20, 200),
        (40, 400),
        (50, 500),
    ],
)
def test_foo(a, b):
    ...

This code applies the pytest.mark.xfail(reason="flaky") mark to the (6, 36) value of the above parametrization call.

This was considered hard to read and understand, and also its implementation presented problems to the code preventing further internal improvements in the marks architecture.

To update the code, use pytest.param:

@pytest.mark.parametrize(
    "a, b",
    [
        (3, 9),
        pytest.param(6, 36, marks=pytest.mark.xfail(reason="flaky")),
        (10, 100),
        (20, 200),
        (40, 400),
        (50, 500),
    ],
)
def test_foo(a, b):
    ...

pytest_funcarg__ prefix

Removed in version 4.0.

In very early pytest versions fixtures could be defined using the pytest_funcarg__ prefix:

def pytest_funcarg__data():
    return SomeData()

Switch over to the @pytest.fixture decorator:

@pytest.fixture
def data():
    return SomeData()

[pytest] section in setup.cfg files

Removed in version 4.0.

[pytest] sections in setup.cfg files should now be named [tool:pytest] to avoid conflicts with other distutils commands.

Metafunc.addcall

Removed in version 4.0.

_pytest.python.Metafunc.addcall() was a precursor to the current parametrized mechanism. Users should use _pytest.python.Metafunc.parametrize() instead.

Example:

def pytest_generate_tests(metafunc):
    metafunc.addcall({"i": 1}, id="1")
    metafunc.addcall({"i": 2}, id="2")

Becomes:

def pytest_generate_tests(metafunc):
    metafunc.parametrize("i", [1, 2], ids=["1", "2"])

cached_setup

Removed in version 4.0.

request.cached_setup was the precursor of the setup/teardown mechanism available to fixtures.

Example:

@pytest.fixture
def db_session():
    return request.cached_setup(
        setup=Session.create, teardown=lambda session: session.close(), scope="module"
    )

This should be updated to make use of standard fixture mechanisms:

@pytest.fixture(scope="module")
def db_session():
    session = Session.create()
    yield session
    session.close()

You can consult funcarg comparison section in the docs for more information.

pytest_plugins in non-top-level conftest files

Removed in version 4.0.

Defining pytest_plugins is now deprecated in non-top-level conftest.py files because they will activate referenced plugins globally, which is surprising because for all other pytest features conftest.py files are only active for tests at or below it.

Config.warn and Node.warn

Removed in version 4.0.

Those methods were part of the internal pytest warnings system, but since 3.8 pytest is using the builtin warning system for its own warnings, so those two functions are now deprecated.

Config.warn should be replaced by calls to the standard warnings.warn, example:

config.warn("C1", "some warning")

Becomes:

warnings.warn(pytest.PytestWarning("some warning"))

Node.warn now supports two signatures:

• node.warn(PytestWarning("some message")): is now the recommended way to call this function. The warning instance must be a PytestWarning or subclass.
• node.warn("CI", "some message"): this code/message form has been removed and should be converted to the warning instance form above.

record_xml_property

Removed in version 4.0.

The record_xml_property fixture is now deprecated in favor of the more generic record_property, which can be used by other consumers (for example pytest-html) to obtain custom information about the test run.

This is just a matter of renaming the fixture as the API is the same:

def test_foo(record_xml_property):
    ...

Change to:

def test_foo(record_property):
    ...

Passing command-line string to pytest.main()

Removed in version 4.0.

Passing a command-line string to pytest.main() is deprecated:

pytest.main("-v -s")

Pass a list instead:

pytest.main(["-v", "-s"])

By passing a string, users expect that pytest will interpret that command-line using the shell rules they are working on (for example bash or Powershell), but this is very hard/impossible to do in a portable way.

Calling fixtures directly

Removed in version 4.0.

Calling a fixture function directly, as opposed to request them in a test function, is deprecated.

For example:

@pytest.fixture
def cell():
    return ...


@pytest.fixture
def full_cell():
    cell = cell()
    cell.make_full()
    return cell

This is a great source of confusion to new users, which will often call the fixture functions and request them from test functions interchangeably, which breaks the fixture resolution model.

In those cases just request the function directly in the dependent fixture:

@pytest.fixture
def cell():
    return ...


@pytest.fixture
def full_cell(cell):
    cell.make_full()
    return cell

Alternatively if the fixture function is called multiple times inside a test (making it hard to apply the above pattern) or if you would like to make minimal changes to the code, you can create a fixture which calls the original function together with the name parameter:

def cell():
    return ...


@pytest.fixture(name="cell")
def cell_fixture():
    return cell()

yield tests

Removed in version 4.0.

pytest supported yield-style tests, where a test function actually yield functions and values that are then turned into proper test methods. Example:

def check(x, y):
    assert x ** x == y


def test_squared():
    yield check, 2, 4
    yield check, 3, 9

This would result into two actual test functions being generated.

This form of test function doesn’t support fixtures properly, and users should switch to pytest.mark.parametrize:

@pytest.mark.parametrize("x, y", [(2, 4), (3, 9)])
def test_squared(x, y):
    assert x ** x == y

Internal classes accessed through Node

Removed in version 4.0.

Access of Module, Function, Class, Instance, File and Item through Node instances now issue this warning:

usage of Function.Module is deprecated, please use pytest.Module instead

Users should just import pytest and access those objects using the pytest module.

This has been documented as deprecated for years, but only now we are actually emitting deprecation warnings.

Node.get_marker

Removed in version 4.0.

As part of a large Marker revamp and iteration, _pytest.nodes.Node.get_marker() is deprecated. See the documentation on tips on how to update your code.

somefunction.markname

Removed in version 4.0.

As part of a large Marker revamp and iteration we already deprecated using MarkInfo the only correct way to get markers of an element is via node.iter_markers(name).

pytest_namespace

Removed in version 4.0.

This hook is deprecated because it greatly complicates the pytest internals regarding configuration and initialization, making some bug fixes and refactorings impossible.

Example of usage:

class MySymbol:
    ...


def pytest_namespace():
    return {"my_symbol": MySymbol()}

Plugin authors relying on this hook should instead require that users now import the plugin modules directly (with an appropriate public API).

As a stopgap measure, plugin authors may still inject their names into pytest’s namespace, usually during pytest_configure:

import pytest


def pytest_configure():
    pytest.my_symbol = MySymbol()

Reinterpretation mode (--assert=reinterp)

Removed in version 3.0.

Reinterpretation mode has now been removed and only plain and rewrite mode are available, consequently the --assert=reinterp option is no longer available. This also means files imported from plugins or conftest.py will not benefit from improved assertions by default, you should use pytest.register_assert_rewrite() to explicitly turn on assertion rewriting for those files.

Removed command-line options

Removed in version 3.0.

The following deprecated commandline options were removed:

• --genscript: no longer supported;
• --no-assert: use --assert=plain instead;
• --nomagic: use --assert=plain instead;
• --report: use -r instead;

py.test-X* entry points

Removed in version 3.0.

Removed all py.test-X* entry points. The versioned, suffixed entry points were never documented and a leftover from a pre-virtualenv era. These entry points also created broken entry points in wheels, so removing them also removes a source of confusion for users.

Python 2.7 and 3.4 support

It is demanding on the maintainers of an open source project to support many Python versions, as there’s extra cost of keeping code compatible between all versions, while holding back on features only made possible on newer Python versions.

In case of Python 2 and 3, the difference between the languages makes it even more prominent, because many new Python 3 features cannot be used in a Python 2/3 compatible code base.

Python 2.7 EOL has been reached in 2020, with the last release planned for mid-April, 2020.

Python 3.4 EOL has been reached in 2019, with the last release made in March, 2019.

For those reasons, in Jun 2019 it was decided that pytest 4.6 series will be the last to support Python 2.7 and 3.4.

What this means for general users

Thanks to the python_requires setuptools option, Python 2.7 and Python 3.4 users using a modern pip version will install the last pytest 4.6.X version automatically even if 5.0 or later versions are available on PyPI.

Users should ensure they are using the latest pip and setuptools versions for this to work.

Maintenance of 4.6.X versions

Until January 2020, the pytest core team ported many bug-fixes from the main release into the 4.6-maintenance branch, with several 4.6.X releases being made along the year.

From now on, the core team will no longer actively backport patches, but the 4.6-maintenance branch will continue to exist so the community itself can contribute patches.

The core team will be happy to accept those patches, and make new 4.6.X releases until mid-2020 (but consider that date as a ballpark, after that date the team might still decide to make new releases for critical bugs).

Technical aspects

(This section is a transcript from #5275).

In this section we describe the technical aspects of the Python 2.7 and 3.4 support plan.

What goes into 4.6.X releases

New 4.6.X releases will contain bug fixes only.

When will 4.6.X releases happen

New 4.6.X releases will happen after we have a few bugs in place to release, or if a few weeks have passed (say a single bug has been fixed a month after the latest 4.6.X release).

No hard rules here, just ballpark.

Who will handle applying bug fixes

We core maintainers expect that people still using Python 2.7/3.4 and being affected by bugs to step up and provide patches and/or port bug fixes from the active branches.

We will be happy to guide users interested in doing so, so please don’t hesitate to ask.

Backporting changes into 4.6

Please follow these instructions:

1. git fetch --all --prune

2. git checkout origin/4.6-maintenance -b backport-XXXX # use the PR number here

3. Locate the merge commit on the PR, in the merged message, for example:

   nicoddemus merged commit 0f8b462 into pytest-dev:features

4. git cherry-pick -m1 REVISION # use the revision you found above (0f8b462).

5. Open a PR targeting 4.6-maintenance:

   • Prefix the message with [4.6] so it is an obvious backport
   • Delete the PR body, it usually contains a duplicate commit message.

Providing new PRs to 4.6

Fresh pull requests to 4.6-maintenance will be accepted provided that the equivalent code in the active branches does not contain that bug (for example, a bug is specific to Python 2 only).

Bug fixes that also happen in the mainstream version should be first fixed there, and then backported as per instructions above.

Contribution getting started

Contributions are highly welcomed and appreciated. Every little help counts, so do not hesitate!

Contents

• Contribution getting started
  • Feature requests and feedback
  • Report bugs
  • Fix bugs
  • Implement features
  • Write documentation
  • Submitting Plugins to pytest-dev
  • Preparing Pull Requests
  • Writing Tests
  • Joining the Development Team

Feature requests and feedback

Do you like pytest? Share some love on Twitter or in your blog posts!

We’d also like to hear about your propositions and suggestions. Feel free to submit them as issues and:

• Explain in detail how they should work.
• Keep the scope as narrow as possible. This will make it easier to implement.

Report bugs

Report bugs for pytest in the issue tracker.

If you are reporting a bug, please include:

• Your operating system name and version.
• Any details about your local setup that might be helpful in troubleshooting, specifically the Python interpreter version, installed libraries, and pytest version.
• Detailed steps to reproduce the bug.

If you can write a demonstration test that currently fails but should pass (xfail), that is a very useful commit to make as well, even if you cannot fix the bug itself.

Fix bugs

Look through the GitHub issues for bugs.

Talk to developers to find out how you can fix specific bugs. To indicate that you are going to work on a particular issue, add a comment to that effect on the specific issue.

Don’t forget to check the issue trackers of your favourite plugins, too!

Implement features

Look through the GitHub issues for enhancements.

Talk to developers to find out how you can implement specific features.

Write documentation

Pytest could always use more documentation. What exactly is needed?

• More complementary documentation. Have you perhaps found something unclear?
• Documentation translations. We currently have only English.
• Docstrings. There can never be too many of them.
• Blog posts, articles and such – they’re all very appreciated.

You can also edit documentation files directly in the GitHub web interface, without using a local copy. This can be convenient for small fixes.

Note

Build the documentation locally with the following command:

$ tox -e docs

The built documentation should be available in the doc/en/_build/.

Where ‘en’ refers to the documentation language.

Submitting Plugins to pytest-dev

Pytest development of the core, some plugins and support code happens in repositories living under the pytest-dev organisations:

• pytest-dev on GitHub
• pytest-dev on Bitbucket

All pytest-dev Contributors team members have write access to all contained repositories. Pytest core and plugins are generally developed using pull requests to respective repositories.

The objectives of the pytest-dev organisation are:

• Having a central location for popular pytest plugins
• Sharing some of the maintenance responsibility (in case a maintainer no longer wishes to maintain a plugin)

You can submit your plugin by subscribing to the pytest-dev mail list and writing a mail pointing to your existing pytest plugin repository which must have the following:

• PyPI presence with a setup.py that contains a license, pytest- prefixed name, version number, authors, short and long description.
• a tox.ini for running tests using tox.
• a README.txt describing how to use the plugin and on which platforms it runs.
• a LICENSE.txt file or equivalent containing the licensing information, with matching info in setup.py.
• an issue tracker for bug reports and enhancement requests.
• a changelog

If no contributor strongly objects and two agree, the repository can then be transferred to the pytest-dev organisation.

Here’s a rundown of how a repository transfer usually proceeds (using a repository named joedoe/pytest-xyz as example):

• joedoe transfers repository ownership to pytest-dev administrator calvin.
• calvin creates pytest-xyz-admin and pytest-xyz-developers teams, inviting joedoe to both as maintainer.
• calvin transfers repository to pytest-dev and configures team access:
  • pytest-xyz-admin admin access;
  • pytest-xyz-developers write access;

The pytest-dev/Contributors team has write access to all projects, and every project administrator is in it. We recommend that each plugin has at least three people who have the right to release to PyPI.

Repository owners can rest assured that no pytest-dev administrator will ever make releases of your repository or take ownership in any way, except in rare cases where someone becomes unresponsive after months of contact attempts. As stated, the objective is to share maintenance and avoid “plugin-abandon”.

Preparing Pull Requests

Short version

1. Fork the repository.

2. Enable and install pre-commit to ensure style-guides and code checks are followed.

3. Target master for bugfixes and doc changes.

4. Target features for new features or functionality changes.

5. Follow PEP-8 for naming and black for formatting.

6. Tests are run using tox:

   tox -e linting,py37
   

   The test environments above are usually enough to cover most cases locally.

7. Write a changelog entry: changelog/2574.bugfix.rst, use issue id number and one of bugfix, removal, feature, vendor, doc or trivial for the issue type.

8. Unless your change is a trivial or a documentation fix (e.g., a typo or reword of a small section) please add yourself to the AUTHORS file, in alphabetical order.

Long version

What is a “pull request”? It informs the project’s core developers about the changes you want to review and merge. Pull requests are stored on GitHub servers. Once you send a pull request, we can discuss its potential modifications and even add more commits to it later on. There’s an excellent tutorial on how Pull Requests work in the GitHub Help Center.

Here is a simple overview, with pytest-specific bits:

1. Fork the pytest GitHub repository. It’s fine to use pytest as your fork repository name because it will live under your user.

2. Clone your fork locally using git and create a branch:

   $ git clone git@github.com:YOUR_GITHUB_USERNAME/pytest.git
   $ cd pytest
   # now, to fix a bug create your own branch off "master":
   
       $ git checkout -b your-bugfix-branch-name master
   
   # or to instead add a feature create your own branch off "features":
   
       $ git checkout -b your-feature-branch-name features
   

   Given we have “major.minor.micro” version numbers, bugfixes will usually be released in micro releases whereas features will be released in minor releases and incompatible changes in major releases.

   If you need some help with Git, follow this quick start guide: https://git.wiki.kernel.org/index.php/QuickStart

3. Install pre-commit and its hook on the pytest repo:

   Note: pre-commit must be installed as admin, as it will not function otherwise:

   $ pip install --user pre-commit
   $ pre-commit install
   

   Afterwards pre-commit will run whenever you commit.

   https://pre-commit.com/ is a framework for managing and maintaining multi-language pre-commit hooks to ensure code-style and code formatting is consistent.

4. Install tox

   Tox is used to run all the tests and will automatically setup virtualenvs to run the tests in. (will implicitly use http://www.virtualenv.org/en/latest/):

   $ pip install tox
   

5. Run all the tests

   You need to have Python 3.7 available in your system. Now running tests is as simple as issuing this command:

   $ tox -e linting,py37
   

   This command will run tests via the “tox” tool against Python 3.7 and also perform “lint” coding-style checks.

6. You can now edit your local working copy and run the tests again as necessary. Please follow PEP-8 for naming.

   You can pass different options to tox. For example, to run tests on Python 3.7 and pass options to pytest (e.g. enter pdb on failure) to pytest you can do:

   $ tox -e py37 -- --pdb
   

   Or to only run tests in a particular test module on Python 3.7:

   $ tox -e py37 -- testing/test_config.py
   

   When committing, pre-commit will re-format the files if necessary.

7. If instead of using tox you prefer to run the tests directly, then we suggest to create a virtual environment and use an editable install with the testing extra:

   $ python3 -m venv .venv
   $ source .venv/bin/activate  # Linux
   $ .venv/Scripts/activate.bat  # Windows
   $ pip install -e ".[testing]"
   

   Afterwards, you can edit the files and run pytest normally:

   $ pytest testing/test_config.py
   

8. Commit and push once your tests pass and you are happy with your change(s):

   $ git commit -a -m "<commit message>"
   $ git push -u
   

9. Create a new changelog entry in changelog. The file should be named <issueid>.<type>.rst, where issueid is the number of the issue related to the change and type is one of bugfix, removal, feature, vendor, doc or trivial. You may not create a changelog entry if the change doesn’t affect the documented behaviour of Pytest.

10. Add yourself to AUTHORS file if not there yet, in alphabetical order.

11. Finally, submit a pull request through the GitHub website using this data:

    head-fork: YOUR_GITHUB_USERNAME/pytest
    compare: your-branch-name
    
    base-fork: pytest-dev/pytest
    base: master          # if it's a bugfix
    base: features        # if it's a feature
    

Writing Tests

Writing tests for plugins or for pytest itself is often done using the testdir fixture, as a “black-box” test.

For example, to ensure a simple test passes you can write:

def test_true_assertion(testdir):
    testdir.makepyfile(
        """
        def test_foo():
            assert True
    """
    )
    result = testdir.runpytest()
    result.assert_outcomes(failed=0, passed=1)

Alternatively, it is possible to make checks based on the actual output of the termal using glob-like expressions:

def test_true_assertion(testdir):
    testdir.makepyfile(
        """
        def test_foo():
            assert False
    """
    )
    result = testdir.runpytest()
    result.stdout.fnmatch_lines(["*assert False*", "*1 failed*"])

When choosing a file where to write a new test, take a look at the existing files and see if there’s one file which looks like a good fit. For example, a regression test about a bug in the --lf option should go into test_cacheprovider.py, given that this option is implemented in cacheprovider.py. If in doubt, go ahead and open a PR with your best guess and we can discuss this over the code.

Joining the Development Team

Anyone who has successfully seen through a pull request which did not require any extra work from the development team to merge will themselves gain commit access if they so wish (if we forget to ask please send a friendly reminder). This does not mean your workflow to contribute changes, everyone goes through the same pull-request-and-review process and no-one merges their own pull requests unless already approved. It does however mean you can participate in the development process more fully since you can merge pull requests from other contributors yourself after having reviewed them.

Development Guide

Some general guidelines regarding development in pytest for maintainers and contributors. Nothing here is set in stone and can’t be changed, feel free to suggest improvements or changes in the workflow.

Code Style

• PEP-8
• flake8 for quality checks
• invoke to automate development tasks

Branches

We have two long term branches:

• master: contains the code for the next bugfix release.
• features: contains the code with new features for the next minor release.

The official repository usually does not contain topic branches, developers and contributors should create topic branches in their own forks.

Exceptions can be made for cases where more than one contributor is working on the same topic or where it makes sense to use some automatic capability of the main repository, such as automatic docs from readthedocs for a branch dealing with documentation refactoring.

Issues

Any question, feature, bug or proposal is welcome as an issue. Users are encouraged to use them whenever they need.

GitHub issues should use labels to categorize them. Labels should be created sporadically, to fill a niche; we should avoid creating labels just for the sake of creating them.

Each label should include a description in the GitHub’s interface stating its purpose.

Labels are managed using labels. All the labels in the repository are kept in .github/labels.toml, so any changes should be via PRs to that file. After a PR is accepted and merged, one of the maintainers must manually synchronize the labels file with the GitHub repository.

Temporary labels

To classify issues for a special event it is encouraged to create a temporary label. This helps those involved to find the relevant issues to work on. Examples of that are sprints in Python events or global hacking events.

• temporary: EP2017 sprint: candidate issues or PRs tackled during the EuroPython 2017

Issues created at those events should have other relevant labels added as well.

Those labels should be removed after they are no longer relevant.

Release Procedure

Our current policy for releasing is to aim for a bugfix every few weeks and a minor release every 2-3 months. The idea is to get fixes and new features out instead of trying to cram a ton of features into a release and by consequence taking a lot of time to make a new one.

Important

pytest releases must be prepared on Linux because the docs and examples expect to be executed in that platform.

1. Create a branch release-X.Y.Z with the version for the release.

   • maintenance releases: from 4.6-maintenance;
   • patch releases: from the latest master;
   • minor releases: from the latest features; then merge with the latest master;

   Ensure your are in a clean work tree.

2. Using tox, generate docs, changelog, announcements:

   $ tox -e release -- <VERSION>
   

   This will generate a commit with all the changes ready for pushing.

3. Open a PR for this branch targeting master (or 4.6-maintenance for maintenance releases).

4. After all tests pass and the PR has been approved, publish to PyPI by pushing the tag:

   git tag <VERSION>
   git push git@github.com:pytest-dev/pytest.git <VERSION>
   

   Wait for the deploy to complete, then make sure it is available on PyPI.

5. Merge the PR.

6. If this is a maintenance release, cherry-pick the CHANGELOG / announce files to the master branch:

   git fetch --all --prune
   git checkout origin/master -b cherry-pick-maintenance-release
   git cherry-pick --no-commit -m1 origin/4.6-maintenance
   git checkout origin/master -- changelog
   git commit  # no arguments
   

7. Send an email announcement with the contents from:

   doc/en/announce/release-<VERSION>.rst
   

   To the following mailing lists:

   • pytest-dev@python.org (all releases)
   • python-announce-list@python.org (all releases)
   • testing-in-python@lists.idyll.org (only major/minor releases)

   And announce it on Twitter with the #pytest hashtag.

Sponsor

pytest is maintained by a team of volunteers from all around the world in their free time. While we work on pytest because we love the project and use it daily at our daily jobs, monetary compensation when possible is welcome to justify time away from friends, family and personal time.

Money is also used to fund local sprints, merchandising (stickers to distribute in conferences for example) and every few years a large sprint involving all members.

OpenCollective

Open Collective is an online funding platform for open and transparent communities. It provide tools to raise money and share your finances in full transparency.

It is the platform of choice for individuals and companies that want to make one-time or monthly donations directly to the project.

See more details in the pytest collective.

pytest for enterprise

Tidelift is working with the maintainers of pytest and thousands of other open source projects to deliver commercial support and maintenance for the open source dependencies you use to build your applications. Save time, reduce risk, and improve code health, while paying the maintainers of the exact dependencies you use.

Get more details

The Tidelift Subscription is a managed open source subscription for application dependencies covering millions of open source projects across JavaScript, Python, Java, PHP, Ruby, .NET, and more.

Your subscription includes:

• Security updates
  • Tidelift’s security response team coordinates patches for new breaking security vulnerabilities and alerts immediately through a private channel, so your software supply chain is always secure.
• Licensing verification and indemnification
  • Tidelift verifies license information to enable easy policy enforcement and adds intellectual property indemnification to cover creators and users in case something goes wrong. You always have a 100% up-to-date bill of materials for your dependencies to share with your legal team, customers, or partners.
• Maintenance and code improvement
  • Tidelift ensures the software you rely on keeps working as long as you need it to work. Your managed dependencies are actively maintained and we recruit additional maintainers where required.
• Package selection and version guidance
  • Tidelift helps you choose the best open source packages from the start—and then guide you through updates to stay on the best releases as new issues arise.
• Roadmap input
  • Take a seat at the table with the creators behind the software you use. Tidelift’s participating maintainers earn more income as their software is used by more subscribers, so they’re interested in knowing what you need.
• Tooling and cloud integration
  • Tidelift works with GitHub, GitLab, BitBucket, and every cloud platform (and other deployment targets, too).

The end result? All of the capabilities you expect from commercial-grade software, for the full breadth of open source you use. That means less time grappling with esoteric open source trivia, and more time building your own applications—and your business.

Request a demo

License

Distributed under the terms of the MIT license, pytest is free and open source software.

The MIT License (MIT)

Copyright (c) 2004-2020 Holger Krekel and others

Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in
the Software without restriction, including without limitation the rights to
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
of the Software, and to permit persons to whom the Software is furnished to do
so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

Contact channels

• pytest issue tracker to report bugs or suggest features (for version 2.0 and above).
• pytest on stackoverflow.com to post questions with the tag pytest. New Questions will usually be seen by pytest users or developers and answered quickly.
• Testing In Python: a mailing list for Python testing tools and discussion.
• pytest-dev at python.org (mailing list) pytest specific announcements and discussions.
• pytest-commit at python.org (mailing list): for commits and new issues
• contribution guide for help on submitting pull requests to GitHub.
• #pylib on irc.freenode.net IRC channel for random questions.
• private mail to Holger.Krekel at gmail com if you want to communicate sensitive issues
• merlinux.eu offers pytest and tox-related professional teaching and consulting.

Historical Notes

This page lists features or behavior from previous versions of pytest which have changed over the years. They are kept here as a historical note so users looking at old code can find documentation related to them.

Marker revamp and iteration

Changed in version 3.6.

pytest’s marker implementation traditionally worked by simply updating the __dict__ attribute of functions to cumulatively add markers. As a result, markers would unintentionally be passed along class hierarchies in surprising ways. Further, the API for retrieving them was inconsistent, as markers from parameterization would be stored differently than markers applied using the @pytest.mark decorator and markers added via node.add_marker.

This state of things made it technically next to impossible to use data from markers correctly without having a deep understanding of the internals, leading to subtle and hard to understand bugs in more advanced usages.

Depending on how a marker got declared/changed one would get either a MarkerInfo which might contain markers from sibling classes, MarkDecorators when marks came from parameterization or from a node.add_marker call, discarding prior marks. Also MarkerInfo acts like a single mark, when it in fact represents a merged view on multiple marks with the same name.

On top of that markers were not accessible in the same way for modules, classes, and functions/methods. In fact, markers were only accessible in functions, even if they were declared on classes/modules.

A new API to access markers has been introduced in pytest 3.6 in order to solve the problems with the initial design, providing the _pytest.nodes.Node.iter_markers() method to iterate over markers in a consistent manner and reworking the internals, which solved a great deal of problems with the initial design.

Updating code

The old Node.get_marker(name) function is considered deprecated because it returns an internal MarkerInfo object which contains the merged name, *args and **kwargs of all the markers which apply to that node.

In general there are two scenarios on how markers should be handled:

1. Marks overwrite each other. Order matters but you only want to think of your mark as a single item. E.g. log_level('info') at a module level can be overwritten by log_level('debug') for a specific test.

In this case, use Node.get_closest_marker(name):

# replace this:
marker = item.get_marker("log_level")
if marker:
    level = marker.args[0]

# by this:
marker = item.get_closest_marker("log_level")
if marker:
    level = marker.args[0]

2. Marks compose in an additive manner. E.g. skipif(condition) marks mean you just want to evaluate all of them, order doesn’t even matter. You probably want to think of your marks as a set here.

In this case iterate over each mark and handle their *args and **kwargs individually.

# replace this
skipif = item.get_marker("skipif")
if skipif:
    for condition in skipif.args:
        # eval condition
        ...

# by this:
for skipif in item.iter_markers("skipif"):
    condition = skipif.args[0]
    # eval condition

If you are unsure or have any questions, please consider opening an issue.

Related issues

Here is a non-exhaustive list of issues fixed by the new implementation:

• Marks don’t pick up nested classes (#199).
• Markers stain on all related classes (#568).
• Combining marks - args and kwargs calculation (#2897).
• request.node.get_marker('name') returns None for markers applied in classes (#902).
• Marks applied in parametrize are stored as markdecorator (#2400).
• Fix marker interaction in a backward incompatible way (#1670).
• Refactor marks to get rid of the current “marks transfer” mechanism (#2363).
• Introduce FunctionDefinition node, use it in generate_tests (#2522).
• Remove named marker attributes and collect markers in items (#891).
• skipif mark from parametrize hides module level skipif mark (#1540).
• skipif + parametrize not skipping tests (#1296).
• Marker transfer incompatible with inheritance (#535).

More details can be found in the original PR.

Note

in a future major release of pytest we will introduce class based markers, at which point markers will no longer be limited to instances of Mark.

cache plugin integrated into the core

The functionality of the core cache plugin was previously distributed as a third party plugin named pytest-cache. The core plugin is compatible regarding command line options and API usage except that you can only store/receive data between test runs that is json-serializable.

funcargs and pytest_funcarg__

In versions prior to 2.3 there was no @pytest.fixture marker and you had to use a magic pytest_funcarg__NAME prefix for the fixture factory. This remains and will remain supported but is not anymore advertised as the primary means of declaring fixture functions.

@pytest.yield_fixture decorator

Prior to version 2.10, in order to use a yield statement to execute teardown code one had to mark a fixture using the yield_fixture marker. From 2.10 onward, normal fixtures can use yield directly so the yield_fixture decorator is no longer needed and considered deprecated.

[pytest] header in setup.cfg

Prior to 3.0, the supported section name was [pytest]. Due to how this may collide with some distutils commands, the recommended section name for setup.cfg files is now [tool:pytest].

Note that for pytest.ini and tox.ini files the section name is [pytest].

Applying marks to @pytest.mark.parametrize parameters

Prior to version 3.1 the supported mechanism for marking values used the syntax:

import pytest


@pytest.mark.parametrize(
    "test_input,expected", [("3+5", 8), ("2+4", 6), pytest.mark.xfail(("6*9", 42))]
)
def test_eval(test_input, expected):
    assert eval(test_input) == expected

This was an initial hack to support the feature but soon was demonstrated to be incomplete, broken for passing functions or applying multiple marks with the same name but different parameters.

The old syntax is planned to be removed in pytest-4.0.

@pytest.mark.parametrize argument names as a tuple

In versions prior to 2.4 one needed to specify the argument names as a tuple. This remains valid but the simpler "name1,name2,..." comma-separated-string syntax is now advertised first because it’s easier to write and produces less line noise.

setup: is now an “autouse fixture”

During development prior to the pytest-2.3 release the name pytest.setup was used but before the release it was renamed and moved to become part of the general fixture mechanism, namely Autouse fixtures (xUnit setup on steroids)

Conditions as strings instead of booleans

Prior to pytest-2.4 the only way to specify skipif/xfail conditions was to use strings:

import sys


@pytest.mark.skipif("sys.version_info >= (3,3)")
def test_function():
    ...

During test function setup the skipif condition is evaluated by calling eval('sys.version_info >= (3,0)', namespace). The namespace contains all the module globals, and os and sys as a minimum.

Since pytest-2.4 boolean conditions are considered preferable because markers can then be freely imported between test modules. With strings you need to import not only the marker but all variables used by the marker, which violates encapsulation.

The reason for specifying the condition as a string was that pytest can report a summary of skip conditions based purely on the condition string. With conditions as booleans you are required to specify a reason string.

Note that string conditions will remain fully supported and you are free to use them if you have no need for cross-importing markers.

The evaluation of a condition string in pytest.mark.skipif(conditionstring) or pytest.mark.xfail(conditionstring) takes place in a namespace dictionary which is constructed as follows:

• the namespace is initialized by putting the sys and os modules and the pytest config object into it.
• updated with the module globals of the test function for which the expression is applied.

The pytest config object allows you to skip based on a test configuration value which you might have added:

@pytest.mark.skipif("not config.getvalue('db')")
def test_function():
    ...

The equivalent with “boolean conditions” is:

@pytest.mark.skipif(not pytest.config.getvalue("db"), reason="--db was not specified")
def test_function():
    pass

Note

You cannot use pytest.config.getvalue() in code imported before pytest’s argument parsing takes place. For example, conftest.py files are imported before command line parsing and thus config.getvalue() will not execute correctly.

pytest.set_trace()

Previous to version 2.4 to set a break point in code one needed to use pytest.set_trace():

import pytest


def test_function():
    ...
    pytest.set_trace()  # invoke PDB debugger and tracing

This is no longer needed and one can use the native import pdb;pdb.set_trace() call directly.

For more details see Setting breakpoints.

“compat” properties

Access of Module, Function, Class, Instance, File and Item through Node instances have long been documented as deprecated, but started to emit warnings from pytest 3.9 and onward.

Users should just import pytest and access those objects using the pytest module.

Talks and Tutorials

Books

• pytest Quick Start Guide, by Bruno Oliveira (2018).
• Python Testing with pytest, by Brian Okken (2017).

Talks and blog postings

• pytest: recommendations, basic packages for testing in Python and Django, Andreu Vallbona, PyBCN June 2019.
• pytest: recommendations, basic packages for testing in Python and Django, Andreu Vallbona, PyconES 2017 (slides in english, video in spanish)
• pytest advanced, Andrew Svetlov (Russian, PyCon Russia, 2016).
• Pythonic testing, Igor Starikov (Russian, PyNsk, November 2016).
• pytest - Rapid Simple Testing, Florian Bruhin, Swiss Python Summit 2016.
• Improve your testing with Pytest and Mock, Gabe Hollombe, PyCon SG 2015.
• Introduction to pytest, Andreas Pelme, EuroPython 2014.
• Advanced Uses of py.test Fixtures, Floris Bruynooghe, EuroPython 2014.
• Why i use py.test and maybe you should too, Andy Todd, Pycon AU 2013
• 3-part blog series about pytest from @pydanny alias Daniel Greenfeld (January 2014)
• pytest: helps you write better Django apps, Andreas Pelme, DjangoCon Europe 2014.
• pytest fixtures: explicit, modular, scalable
• Testing Django Applications with pytest, Andreas Pelme, EuroPython 2013.
• Testes pythonics com py.test, Vinicius Belchior Assef Neto, Plone Conf 2013, Brazil.
• Introduction to py.test fixtures, FOSDEM 2013, Floris Bruynooghe.
• pytest feature and release highlights, Holger Krekel (GERMAN, October 2013)
• pytest introduction from Brian Okken (January 2013)
• pycon australia 2012 pytest talk from Brianna Laugher (video, slides, code)
• pycon 2012 US talk video from Holger Krekel
• monkey patching done right (blog post, consult monkeypatch plugin for up-to-date API)

Test parametrization:

• generating parametrized tests with fixtures.
• test generators and cached setup
• parametrizing tests, generalized (blog post)
• putting test-hooks into local or global plugins (blog post)

Assertion introspection:

• (07/2011) Behind the scenes of pytest’s new assertion rewriting

Distributed testing:

• simultaneously test your code on all platforms (blog entry)

Plugin specific examples:

• skipping slow tests by default in pytest (blog entry)
• many examples in the docs for plugins

Project examples

Here are some examples of projects using pytest (please send notes via Contact channels):

• PyPy, Python with a JIT compiler, running over 21000 tests
• the MoinMoin Wiki Engine
• sentry, realtime app-maintenance and exception tracking
• Astropy and affiliated packages
• tox, virtualenv/Hudson integration tool
• PyPM ActiveState’s package manager
• Fom a fluid object mapper for FluidDB
• applib cross-platform utilities
• six Python 2 and 3 compatibility utilities
• pediapress MediaWiki articles
• mwlib mediawiki parser and utility library
• The Translate Toolkit for localization and conversion
• execnet rapid multi-Python deployment
• pylib cross-platform path, IO, dynamic code library
• bbfreeze create standalone executables from Python scripts
• pdb++ a fancier version of PDB
• pudb full-screen console debugger for python
• py-s3fuse Amazon S3 FUSE based filesystem
• waskr WSGI Stats Middleware
• guachi global persistent configs for Python modules
• Circuits lightweight Event Driven Framework
• pygtk-helpers easy interaction with PyGTK
• QuantumCore statusmessage and repoze openid plugin
• pydataportability libraries for managing the open web
• XIST extensible HTML/XML generator
• tiddlyweb optionally headless, extensible RESTful datastore
• fancycompleter for colorful tab-completion
• Paludis tools for Gentoo Paludis package manager
• Gerald schema comparison tool
• abjad Python API for Formalized Score control
• bu a microscopic build system
• katcp Telescope communication protocol over Twisted
• kss plugin timer
• pyudev a pure Python binding to the Linux library libudev
• pytest-localserver a plugin for pytest that provides an httpserver and smtpserver
• pytest-monkeyplus a plugin that extends monkeypatch

These projects help integrate pytest into other Python frameworks:

• pytest-django for Django
• zope.pytest for Zope and Grok
• pytest_gae for Google App Engine
• There is some work underway for Kotti, a CMS built in Pyramid/Pylons

Some organisations using pytest

• Square Kilometre Array, Cape Town
• Some Mozilla QA people use pytest to distribute their Selenium tests
• Shootq
• Stups department of Heinrich Heine University Duesseldorf
• cellzome
• Open End, Gothenborg
• Laboratory of Bioinformatics, Warsaw
• merlinux, Germany
• ESSS, Brazil
• many more … (please be so kind to send a note via Contact channels)

Release announcements

pytest-5.3.5

pytest 5.3.5 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at https://docs.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Daniel Hahler
• Ran Benita

Happy testing, The pytest Development Team

pytest-5.3.4

pytest 5.3.4 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at https://docs.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Bruno Oliveira
• Daniel Hahler
• Ran Benita

Happy testing, The pytest Development Team

pytest-5.3.3

pytest 5.3.3 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at https://docs.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Adam Johnson
• Alexandre Mulatinho
• Anthony Sottile
• Bruno Oliveira
• Chris NeJame
• Daniel Hahler
• Hugo van Kemenade
• Marcelo Duarte Trevisani
• PaulC
• Ran Benita
• Ryan Barner
• Seth Junot
• marc

Happy testing, The pytest Development Team

pytest-5.3.2

pytest 5.3.2 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at https://docs.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Anthony Sottile
• Bruno Oliveira
• Claudio Madotto
• Daniel Hahler
• Jared Vasquez
• Michael Rose
• Ran Benita
• Ronny Pfannschmidt
• Zac Hatfield-Dodds

Happy testing, The pytest Development Team

pytest-5.3.1

pytest 5.3.1 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at https://docs.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Anthony Sottile
• Bruno Oliveira
• Daniel Hahler
• Felix Yan
• Florian Bruhin
• Mark Dickinson
• Nikolay Kondratyev
• Steffen Schroeder
• Zac Hatfield-Dodds

Happy testing, The pytest Development Team

pytest-5.3.0

The pytest team is proud to announce the 5.3.0 release!

pytest is a mature Python testing tool with more than a 2000 tests against itself, passing on many different interpreters and platforms.

This release contains a number of bugs fixes and improvements, so users are encouraged to take a look at the CHANGELOG:

https://docs.pytest.org/en/latest/changelog.html

For complete documentation, please visit:

https://docs.pytest.org/en/latest/

As usual, you can upgrade from pypi via:

pip install -U pytest

Thanks to all who contributed to this release, among them:

• AnjoMan
• Anthony Sottile
• Anton Lodder
• Bruno Oliveira
• Daniel Hahler
• Gregory Lee
• Josh Karpel
• JoshKarpel
• Joshua Storck
• Kale Kundert
• MarcoGorelli
• Michael Krebs
• NNRepos
• Ran Benita
• TH3CHARLie
• Tibor Arpas
• Zac Hatfield-Dodds
• 林玮

Happy testing, The Pytest Development Team

pytest-5.2.4

pytest 5.2.4 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at https://docs.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Anthony Sottile
• Bruno Oliveira
• Daniel Hahler
• Hugo
• Michael Shields

Happy testing, The pytest Development Team

pytest-5.2.3

pytest 5.2.3 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at https://docs.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Anthony Sottile
• Brett Cannon
• Bruno Oliveira
• Daniel Hahler
• Daniil Galiev
• David Szotten
• Florian Bruhin
• Patrick Harmon
• Ran Benita
• Zac Hatfield-Dodds
• Zak Hassan

Happy testing, The pytest Development Team

pytest-5.2.2

pytest 5.2.2 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at https://docs.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Albert Tugushev
• Andrzej Klajnert
• Anthony Sottile
• Bruno Oliveira
• Daniel Hahler
• Florian Bruhin
• Nattaphoom Chaipreecha
• Oliver Bestwalter
• Philipp Loose
• Ran Benita
• Victor Maryama
• Yoav Caspi

Happy testing, The pytest Development Team

pytest-5.2.1

pytest 5.2.1 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at https://docs.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Anthony Sottile
• Bruno Oliveira
• Florian Bruhin
• Hynek Schlawack
• Kevin J. Foley
• tadashigaki

Happy testing, The pytest Development Team

pytest-5.2.0

The pytest team is proud to announce the 5.2.0 release!

pytest is a mature Python testing tool with more than a 2000 tests against itself, passing on many different interpreters and platforms.

This release contains a number of bugs fixes and improvements, so users are encouraged to take a look at the CHANGELOG:

https://docs.pytest.org/en/latest/changelog.html

For complete documentation, please visit:

https://docs.pytest.org/en/latest/

As usual, you can upgrade from pypi via:

pip install -U pytest

Thanks to all who contributed to this release, among them:

• Andrzej Klajnert
• Anthony Sottile
• Bruno Oliveira
• Daniel Hahler
• James Cooke
• Michael Goerz
• Ran Benita
• Tomáš Chvátal

Happy testing, The Pytest Development Team

pytest-5.1.3

pytest 5.1.3 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at https://docs.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Anthony Sottile
• Bruno Oliveira
• Christian Neumüller
• Daniel Hahler
• Gene Wood
• Hugo

Happy testing, The pytest Development Team

pytest-5.1.2

pytest 5.1.2 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at https://docs.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Andrzej Klajnert
• Anthony Sottile
• Bruno Oliveira
• Christian Neumüller
• Robert Holt
• linchiwei123

Happy testing, The pytest Development Team

pytest-5.1.1

pytest 5.1.1 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at https://docs.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Anthony Sottile
• Bruno Oliveira
• Daniel Hahler
• Florian Bruhin
• Hugo van Kemenade
• Ran Benita
• Ronny Pfannschmidt

Happy testing, The pytest Development Team

pytest-5.1.0

The pytest team is proud to announce the 5.1.0 release!

pytest is a mature Python testing tool with more than a 2000 tests against itself, passing on many different interpreters and platforms.

This release contains a number of bugs fixes and improvements, so users are encouraged to take a look at the CHANGELOG:

https://docs.pytest.org/en/latest/changelog.html

For complete documentation, please visit:

https://docs.pytest.org/en/latest/

As usual, you can upgrade from pypi via:

pip install -U pytest

Thanks to all who contributed to this release, among them:

• Albert Tugushev
• Alexey Zankevich
• Anthony Sottile
• Bruno Oliveira
• Daniel Hahler
• David Röthlisberger
• Florian Bruhin
• Ilya Stepin
• Jon Dufresne
• Kaiqi
• Max R
• Miro Hrončok
• Oliver Bestwalter
• Ran Benita
• Ronny Pfannschmidt
• Samuel Searles-Bryant
• Semen Zhydenko
• Steffen Schroeder
• Thomas Grainger
• Tim Hoffmann
• William Woodall
• Wojtek Erbetowski
• Xixi Zhao
• Yash Todi
• boris
• dmitry.dygalo
• helloocc
• martbln
• mei-li

Happy testing, The Pytest Development Team

pytest-5.0.1

pytest 5.0.1 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at https://docs.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• AmirElkess
• Andreu Vallbona Plazas
• Anthony Sottile
• Bruno Oliveira
• Florian Bruhin
• Michael Moore
• Niklas Meinzer
• Thomas Grainger

Happy testing, The pytest Development Team

pytest-5.0.0

The pytest team is proud to announce the 5.0.0 release!

pytest is a mature Python testing tool with more than a 2000 tests against itself, passing on many different interpreters and platforms.

This release contains a number of bugs fixes and improvements, so users are encouraged to take a look at the CHANGELOG:

https://docs.pytest.org/en/latest/changelog.html

For complete documentation, please visit:

https://docs.pytest.org/en/latest/

As usual, you can upgrade from pypi via:

pip install -U pytest

Thanks to all who contributed to this release, among them:

• Anthony Sottile
• Bruno Oliveira
• Daniel Hahler
• Dirk Thomas
• Evan Kepner
• Florian Bruhin
• Hugo
• Kevin J. Foley
• Pulkit Goyal
• Ralph Giles
• Ronny Pfannschmidt
• Thomas Grainger
• Thomas Hisch
• Tim Gates
• Victor Maryama
• Yuri Apollov
• Zac Hatfield-Dodds
• curiousjazz77
• patriksevallius

Happy testing, The Pytest Development Team

pytest-4.6.9

pytest 4.6.9 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at https://docs.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Anthony Sottile
• Bruno Oliveira
• Felix Yan
• Hugo

Happy testing, The pytest Development Team

pytest-4.6.8

pytest 4.6.8 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at https://docs.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Anthony Sottile
• Bruno Oliveira
• Ryan Mast

Happy testing, The pytest Development Team

pytest-4.6.7

pytest 4.6.7 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at https://docs.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Bruno Oliveira
• Daniel Hahler

Happy testing, The pytest Development Team

pytest-4.6.6

pytest 4.6.6 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at https://docs.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Anthony Sottile
• Bruno Oliveira
• Michael Goerz

Happy testing, The pytest Development Team

pytest-4.6.5

pytest 4.6.5 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at https://docs.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Anthony Sottile
• Bruno Oliveira
• Daniel Hahler
• Thomas Grainger

Happy testing, The pytest Development Team

pytest-4.6.4

pytest 4.6.4 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at https://docs.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Anthony Sottile
• Bruno Oliveira
• Daniel Hahler
• Thomas Grainger
• Zac Hatfield-Dodds

Happy testing, The pytest Development Team

pytest-4.6.3

pytest 4.6.3 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at https://docs.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Anthony Sottile
• Bruno Oliveira
• Daniel Hahler
• Dirk Thomas

Happy testing, The pytest Development Team

pytest-4.6.2

pytest 4.6.2 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at https://docs.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Anthony Sottile

Happy testing, The pytest Development Team

pytest-4.6.1

pytest 4.6.1 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at https://docs.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Anthony Sottile
• Bruno Oliveira

Happy testing, The pytest Development Team

pytest-4.6.0

The pytest team is proud to announce the 4.6.0 release!

pytest is a mature Python testing tool with more than a 2000 tests against itself, passing on many different interpreters and platforms.

This release contains a number of bugs fixes and improvements, so users are encouraged to take a look at the CHANGELOG:

https://docs.pytest.org/en/latest/changelog.html

For complete documentation, please visit:

https://docs.pytest.org/en/latest/

As usual, you can upgrade from pypi via:

pip install -U pytest

Thanks to all who contributed to this release, among them:

• Akiomi Kamakura
• Anthony Sottile
• Bruno Oliveira
• Daniel Hahler
• David Röthlisberger
• Evan Kepner
• Jeffrey Rackauckas
• MyComputer
• Nikita Krokosh
• Raul Tambre
• Thomas Hisch
• Tim Hoffmann
• Tomer Keren
• Victor Maryama
• danielx123
• oleg-yegorov

Happy testing, The Pytest Development Team

pytest-4.5.0

The pytest team is proud to announce the 4.5.0 release!

pytest is a mature Python testing tool with more than a 2000 tests against itself, passing on many different interpreters and platforms.

This release contains a number of bugs fixes and improvements, so users are encouraged to take a look at the CHANGELOG:

https://docs.pytest.org/en/latest/changelog.html

For complete documentation, please visit:

https://docs.pytest.org/en/latest/

As usual, you can upgrade from pypi via:

pip install -U pytest

Thanks to all who contributed to this release, among them:

• Anthony Sottile
• Bruno Oliveira
• Daniel Hahler
• Floris Bruynooghe
• Pulkit Goyal
• Samuel Searles-Bryant
• Zac Hatfield-Dodds

Happy testing, The Pytest Development Team

pytest-4.4.2

pytest 4.4.2 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at https://docs.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Allan Lewis
• Anthony Sottile
• Bruno Oliveira
• DamianSkrzypczak
• Daniel Hahler
• Don Kirkby
• Douglas Thor
• Hugo
• Ilya Konstantinov
• Jon Dufresne
• Matt Cooper
• Nikolay Kondratyev
• Ondřej Súkup
• Peter Schutt
• Romain Chossart
• Sitaktif

Happy testing, The pytest Development Team

pytest-4.4.1

pytest 4.4.1 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at https://docs.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Anthony Sottile
• Bruno Oliveira
• Daniel Hahler

Happy testing, The pytest Development Team

pytest-4.4.0

The pytest team is proud to announce the 4.4.0 release!

pytest is a mature Python testing tool with more than a 2000 tests against itself, passing on many different interpreters and platforms.

This release contains a number of bugs fixes and improvements, so users are encouraged to take a look at the CHANGELOG:

https://docs.pytest.org/en/latest/changelog.html

For complete documentation, please visit:

https://docs.pytest.org/en/latest/

As usual, you can upgrade from pypi via:

pip install -U pytest

Thanks to all who contributed to this release, among them:

• Anthony Sottile
• ApaDoctor
• Bernhard M. Wiedemann
• Brian Skinn
• Bruno Oliveira
• Daniel Hahler
• Gary Tyler
• Jeong YunWon
• Miro Hrončok
• Takafumi Arakaki
• henrykironde
• smheidrich

Happy testing, The Pytest Development Team

pytest-4.3.1

pytest 4.3.1 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at https://docs.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Andras Mitzki
• Anthony Sottile
• Bruno Oliveira
• Daniel Hahler
• Danilo Horta
• Grygorii Iermolenko
• Jeff Hale
• Kyle Altendorf
• Stephan Hoyer
• Zac Hatfield-Dodds
• songbowen

Happy testing, The pytest Development Team

pytest-4.3.0

The pytest team is proud to announce the 4.3.0 release!

pytest is a mature Python testing tool with more than a 2000 tests against itself, passing on many different interpreters and platforms.

This release contains a number of bugs fixes and improvements, so users are encouraged to take a look at the CHANGELOG:

https://docs.pytest.org/en/latest/changelog.html

For complete documentation, please visit:

https://docs.pytest.org/en/latest/

As usual, you can upgrade from pypi via:

pip install -U pytest

Thanks to all who contributed to this release, among them:

• Andras Mitzki
• Anthony Sottile
• Bruno Oliveira
• Christian Fetzer
• Daniel Hahler
• Grygorii Iermolenko
• 18. Alex Matevish
• Ronny Pfannschmidt
• cclauss

Happy testing, The Pytest Development Team

pytest-4.2.1

pytest 4.2.1 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at https://docs.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Anthony Sottile
• Arel Cordero
• Bruno Oliveira
• Daniel Hahler
• Holger Kohr
• Kevin J. Foley
• Nick Murphy
• Paweł Stradomski
• Raphael Pierzina
• Ronny Pfannschmidt
• Sam Brightman
• Thomas Hisch
• Zac Hatfield-Dodds

Happy testing, The pytest Development Team

pytest-4.2.0

The pytest team is proud to announce the 4.2.0 release!

pytest is a mature Python testing tool with more than a 2000 tests against itself, passing on many different interpreters and platforms.

This release contains a number of bugs fixes and improvements, so users are encouraged to take a look at the CHANGELOG:

https://docs.pytest.org/en/latest/changelog.html

For complete documentation, please visit:

https://docs.pytest.org/en/latest/

As usual, you can upgrade from pypi via:

pip install -U pytest

Thanks to all who contributed to this release, among them:

• Adam Uhlir
• Anthony Sottile
• Bruno Oliveira
• Christopher Dignam
• Daniel Hahler
• Joseph Hunkeler
• Kristoffer Nordstroem
• Ronny Pfannschmidt
• Thomas Hisch
• wim glenn

Happy testing, The Pytest Development Team

pytest-4.1.1

pytest 4.1.1 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at https://docs.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Anthony Sottile
• Anton Lodder
• Bruno Oliveira
• Daniel Hahler
• David Vo
• Oscar Benjamin
• Ronny Pfannschmidt
• Victor Maryama
• Yoav Caspi
• dmitry.dygalo

Happy testing, The pytest Development Team

pytest-4.1.0

The pytest team is proud to announce the 4.1.0 release!

pytest is a mature Python testing tool with more than a 2000 tests against itself, passing on many different interpreters and platforms.

This release contains a number of bugs fixes and improvements, so users are encouraged to take a look at the CHANGELOG:

https://docs.pytest.org/en/latest/changelog.html

For complete documentation, please visit:

https://docs.pytest.org/en/latest/

As usual, you can upgrade from pypi via:

pip install -U pytest

Thanks to all who contributed to this release, among them:

• Adam Johnson
• Aly Sivji
• Andrey Paramonov
• Anthony Sottile
• Bruno Oliveira
• Daniel Hahler
• David Vo
• Hyunchel Kim
• Jeffrey Rackauckas
• Kanguros
• Nicholas Devenish
• Pedro Algarvio
• Randy Barlow
• Ronny Pfannschmidt
• Tomer Keren
• feuillemorte
• wim glenn

Happy testing, The Pytest Development Team

pytest-4.0.2

pytest 4.0.2 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at https://docs.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Anthony Sottile
• Bruno Oliveira
• Daniel Hahler
• Pedro Algarvio
• Ronny Pfannschmidt
• Tomer Keren
• Yash Todi

Happy testing, The pytest Development Team

pytest-4.0.1

pytest 4.0.1 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at https://docs.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Anthony Sottile
• Bruno Oliveira
• Daniel Hahler
• Michael D. Hoyle
• Ronny Pfannschmidt
• Slam

Happy testing, The pytest Development Team

pytest-4.0.0

The pytest team is proud to announce the 4.0.0 release!

pytest is a mature Python testing tool with more than a 2000 tests against itself, passing on many different interpreters and platforms.

This release contains a number of bugs fixes and improvements, so users are encouraged to take a look at the CHANGELOG:

https://docs.pytest.org/en/latest/changelog.html

For complete documentation, please visit:

https://docs.pytest.org/en/latest/

As usual, you can upgrade from pypi via:

pip install -U pytest

Thanks to all who contributed to this release, among them:

• Bruno Oliveira
• Daniel Hahler
• Ronny Pfannschmidt

Happy testing, The Pytest Development Team

pytest-3.10.1

pytest 3.10.1 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at https://docs.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Anthony Sottile
• Boris Feld
• Bruno Oliveira
• Daniel Hahler
• Fabien ZARIFIAN
• Jon Dufresne
• Ronny Pfannschmidt

Happy testing, The pytest Development Team

pytest-3.10.0

The pytest team is proud to announce the 3.10.0 release!

pytest is a mature Python testing tool with more than a 2000 tests against itself, passing on many different interpreters and platforms.

This release contains a number of bugs fixes and improvements, so users are encouraged to take a look at the CHANGELOG:

https://docs.pytest.org/en/latest/changelog.html

For complete documentation, please visit:

https://docs.pytest.org/en/latest/

As usual, you can upgrade from pypi via:

pip install -U pytest

Thanks to all who contributed to this release, among them:

• Anders Hovmöller
• Andreu Vallbona Plazas
• Ankit Goel
• Anthony Sottile
• Bernardo Gomes
• Brianna Laugher
• Bruno Oliveira
• Daniel Hahler
• David Szotten
• Mick Koch
• Niclas Olofsson
• Palash Chatterjee
• Ronny Pfannschmidt
• Sven-Hendrik Haase
• Ville Skyttä
• William Jamir Silva

Happy testing, The Pytest Development Team

pytest-3.9.3

pytest 3.9.3 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at https://docs.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Andreas Profous
• Ankit Goel
• Anthony Sottile
• Bruno Oliveira
• Daniel Hahler
• Jon Dufresne
• Ronny Pfannschmidt

Happy testing, The pytest Development Team

pytest-3.9.2

pytest 3.9.2 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at https://docs.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Ankit Goel
• Anthony Sottile
• Bruno Oliveira
• Ronny Pfannschmidt
• Vincent Barbaresi
• ykantor

Happy testing, The pytest Development Team

pytest-3.9.1

pytest 3.9.1 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at https://docs.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Bruno Oliveira
• Ronny Pfannschmidt
• Thomas Hisch

Happy testing, The pytest Development Team

pytest-3.9.0

The pytest team is proud to announce the 3.9.0 release!

pytest is a mature Python testing tool with more than a 2000 tests against itself, passing on many different interpreters and platforms.

This release contains a number of bugs fixes and improvements, so users are encouraged to take a look at the CHANGELOG:

https://docs.pytest.org/en/latest/changelog.html

For complete documentation, please visit:

https://docs.pytest.org/en/latest/

As usual, you can upgrade from pypi via:

pip install -U pytest

Thanks to all who contributed to this release, among them:

• Andrea Cimatoribus
• Ankit Goel
• Anthony Sottile
• Ben Eyal
• Bruno Oliveira
• Daniel Hahler
• Jeffrey Rackauckas
• Jose Carlos Menezes
• Kyle Altendorf
• Niklas JQ
• Palash Chatterjee
• Ronny Pfannschmidt
• Thomas Hess
• Thomas Hisch
• Tomer Keren
• Victor Maryama

Happy testing, The Pytest Development Team

pytest-3.8.2

pytest 3.8.2 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at https://docs.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Ankit Goel
• Anthony Sottile
• Bruno Oliveira
• Daniel Hahler
• Denis Otkidach
• Harry Percival
• Jeffrey Rackauckas
• Jose Carlos Menezes
• Ronny Pfannschmidt
• Zac Hatfield-Dodds
• iwanb

Happy testing, The pytest Development Team

pytest-3.8.1

pytest 3.8.1 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at https://docs.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Ankit Goel
• Anthony Sottile
• Bruno Oliveira
• Daniel Hahler
• Maximilian Albert
• Ronny Pfannschmidt
• William Jamir Silva
• wim glenn

Happy testing, The pytest Development Team

pytest-3.8.0

The pytest team is proud to announce the 3.8.0 release!

pytest is a mature Python testing tool with more than a 2000 tests against itself, passing on many different interpreters and platforms.

This release contains a number of bugs fixes and improvements, so users are encouraged to take a look at the CHANGELOG:

https://docs.pytest.org/en/latest/changelog.html

For complete documentation, please visit:

https://docs.pytest.org/en/latest/

As usual, you can upgrade from pypi via:

pip install -U pytest

Thanks to all who contributed to this release, among them:

• Anthony Sottile
• Bruno Oliveira
• CrazyMerlyn
• Daniel Hahler
• Fabio Zadrozny
• Jeffrey Rackauckas
• Ronny Pfannschmidt
• Virgil Dupras
• dhirensr
• hoefling
• wim glenn

Happy testing, The Pytest Development Team

pytest-3.7.4

pytest 3.7.4 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at https://docs.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Anthony Sottile
• Bruno Oliveira
• Daniel Hahler
• Jiri Kuncar
• Steve Piercy

Happy testing, The pytest Development Team

pytest-3.7.3

pytest 3.7.3 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at http://doc.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Andrew Champion
• Anthony Sottile
• Bruno Oliveira
• Daniel Hahler
• Gandalf Saxe
• Jennifer Rinker
• Natan Lao
• Ondřej Súkup
• Ronny Pfannschmidt
• Sankt Petersbug
• Tyler Richard
• Victor Maryama
• Vlad Shcherbina
• turturica
• wim glenn

Happy testing, The pytest Development Team

pytest-3.7.2

pytest 3.7.2 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at http://doc.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Anthony Sottile
• Bruno Oliveira
• Daniel Hahler
• Josh Holland
• Ronny Pfannschmidt
• Sankt Petersbug
• Wes Thomas
• turturica

Happy testing, The pytest Development Team

pytest-3.7.1

pytest 3.7.1 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at http://doc.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Anthony Sottile
• Bruno Oliveira
• Kale Kundert
• Ronny Pfannschmidt

Happy testing, The pytest Development Team

pytest-3.7.0

The pytest team is proud to announce the 3.7.0 release!

pytest is a mature Python testing tool with more than a 2000 tests against itself, passing on many different interpreters and platforms.

This release contains a number of bugs fixes and improvements, so users are encouraged to take a look at the CHANGELOG:

http://doc.pytest.org/en/latest/changelog.html

For complete documentation, please visit:

http://docs.pytest.org

As usual, you can upgrade from pypi via:

pip install -U pytest

Thanks to all who contributed to this release, among them:

• Alan
• Alan Brammer
• Ammar Najjar
• Anthony Sottile
• Bruno Oliveira
• Jeffrey Rackauckas
• Kale Kundert
• Ronny Pfannschmidt
• Serhii Mozghovyi
• Tadek Teleżyński
• Wil Cooley
• abrammer
• avirlrma
• turturica

Happy testing, The Pytest Development Team

pytest-3.6.4

pytest 3.6.4 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at http://doc.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Anthony Sottile
• Bernhard M. Wiedemann
• Bruno Oliveira
• Drew
• E Hershey
• Hugo Martins
• Vlad Shcherbina

Happy testing, The pytest Development Team

pytest-3.6.3

pytest 3.6.3 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at http://doc.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• AdamEr8
• Anthony Sottile
• Bruno Oliveira
• Jean-Paul Calderone
• Jon Dufresne
• Marcelo Duarte Trevisani
• Ondřej Súkup
• Ronny Pfannschmidt
• T.E.A de Souza
• Victor Maryama

Happy testing, The pytest Development Team

pytest-3.6.2

pytest 3.6.2 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at http://doc.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Alan Velasco
• Alex Barbato
• Anthony Sottile
• Bartosz Cierocki
• Bruno Oliveira
• Daniel Hahler
• Guoqiang Zhang
• Hynek Schlawack
• John T. Wodder II
• Michael Käufl
• Ronny Pfannschmidt
• Samuel Dion-Girardeau

Happy testing, The pytest Development Team

pytest-3.6.1

pytest 3.6.1 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at http://doc.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Anthony Sottile
• Bruno Oliveira
• Jeffrey Rackauckas
• Miro Hrončok
• Niklas Meinzer
• Oliver Bestwalter
• Ronny Pfannschmidt

Happy testing, The pytest Development Team

pytest-3.6.0

The pytest team is proud to announce the 3.6.0 release!

pytest is a mature Python testing tool with more than a 1600 tests against itself, passing on many different interpreters and platforms.

This release contains a number of bugs fixes and improvements, so users are encouraged to take a look at the CHANGELOG:

http://doc.pytest.org/en/latest/changelog.html

For complete documentation, please visit:

http://docs.pytest.org

As usual, you can upgrade from pypi via:

pip install -U pytest

Thanks to all who contributed to this release, among them:

• Anthony Shaw
• ApaDoctor
• Brian Maissy
• Bruno Oliveira
• Jon Dufresne
• Katerina Koukiou
• Miro Hrončok
• Rachel Kogan
• Ronny Pfannschmidt
• Tim Hughes
• Tyler Goodlet
• Ville Skyttä
• aviral1701
• feuillemorte

Happy testing, The Pytest Development Team

pytest-3.5.1

pytest 3.5.1 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at http://doc.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Brian Maissy
• Bruno Oliveira
• Darren Burns
• David Chudzicki
• Floris Bruynooghe
• Holger Kohr
• Irmen de Jong
• Jeffrey Rackauckas
• Rachel Kogan
• Ronny Pfannschmidt
• Stefan Scherfke
• Tim Strazny
• Семён Марьясин

Happy testing, The pytest Development Team

pytest-3.5.0

The pytest team is proud to announce the 3.5.0 release!

pytest is a mature Python testing tool with more than a 1600 tests against itself, passing on many different interpreters and platforms.

This release contains a number of bugs fixes and improvements, so users are encouraged to take a look at the CHANGELOG:

http://doc.pytest.org/en/latest/changelog.html

For complete documentation, please visit:

http://docs.pytest.org

As usual, you can upgrade from pypi via:

pip install -U pytest

Thanks to all who contributed to this release, among them:

• Allan Feldman
• Brian Maissy
• Bruno Oliveira
• Carlos Jenkins
• Daniel Hahler
• Florian Bruhin
• Jason R. Coombs
• Jeffrey Rackauckas
• Jordan Speicher
• Julien Palard
• Kale Kundert
• Kostis Anagnostopoulos
• Kyle Altendorf
• Maik Figura
• Pedro Algarvio
• Ronny Pfannschmidt
• Tadeu Manoel
• Tareq Alayan
• Thomas Hisch
• William Lee
• codetriage-readme-bot
• feuillemorte
• joshm91
• mike

Happy testing, The Pytest Development Team

pytest-3.4.2

pytest 3.4.2 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at http://doc.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Allan Feldman
• Bruno Oliveira
• Florian Bruhin
• Jason R. Coombs
• Kyle Altendorf
• Maik Figura
• Ronny Pfannschmidt
• codetriage-readme-bot
• feuillemorte
• joshm91
• mike

Happy testing, The pytest Development Team

pytest-3.4.1

pytest 3.4.1 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at http://doc.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Aaron
• Alan Velasco
• Andy Freeland
• Brian Maissy
• Bruno Oliveira
• Florian Bruhin
• Jason R. Coombs
• Marcin Bachry
• Pedro Algarvio
• Ronny Pfannschmidt

Happy testing, The pytest Development Team

pytest-3.4.0

The pytest team is proud to announce the 3.4.0 release!

pytest is a mature Python testing tool with more than a 1600 tests against itself, passing on many different interpreters and platforms.

This release contains a number of bugs fixes and improvements, so users are encouraged to take a look at the CHANGELOG:

http://doc.pytest.org/en/latest/changelog.html

For complete documentation, please visit:

http://docs.pytest.org

As usual, you can upgrade from pypi via:

pip install -U pytest

Thanks to all who contributed to this release, among them:

• Aaron
• Alan Velasco
• Anders Hovmöller
• Andrew Toolan
• Anthony Sottile
• Aron Coyle
• Brian Maissy
• Bruno Oliveira
• Cyrus Maden
• Florian Bruhin
• Henk-Jaap Wagenaar
• Ian Lesperance
• Jon Dufresne
• Jurko Gospodnetić
• Kate
• Kimberly
• Per A. Brodtkorb
• Pierre-Alexandre Fonta
• Raphael Castaneda
• Ronny Pfannschmidt
• ST John
• Segev Finer
• Thomas Hisch
• Tzu-ping Chung
• feuillemorte

Happy testing, The Pytest Development Team

pytest-3.3.2

pytest 3.3.2 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at http://doc.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Anthony Sottile
• Antony Lee
• Austin
• Bruno Oliveira
• Florian Bruhin
• Floris Bruynooghe
• Henk-Jaap Wagenaar
• Jurko Gospodnetić
• Ronny Pfannschmidt
• Srinivas Reddy Thatiparthy
• Thomas Hisch

Happy testing, The pytest Development Team

pytest-3.3.1

pytest 3.3.1 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at http://doc.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Bruno Oliveira
• Daniel Hahler
• Eugene Prikazchikov
• Florian Bruhin
• Roland Puntaier
• Ronny Pfannschmidt
• Sebastian Rahlf
• Tom Viner

Happy testing, The pytest Development Team

pytest-3.3.0

The pytest team is proud to announce the 3.3.0 release!

pytest is a mature Python testing tool with more than a 1600 tests against itself, passing on many different interpreters and platforms.

This release contains a number of bugs fixes and improvements, so users are encouraged to take a look at the CHANGELOG:

http://doc.pytest.org/en/latest/changelog.html

For complete documentation, please visit:

http://docs.pytest.org

As usual, you can upgrade from pypi via:

pip install -U pytest

Thanks to all who contributed to this release, among them:

• Anthony Sottile
• Bruno Oliveira
• Ceridwen
• Daniel Hahler
• Dirk Thomas
• Dmitry Malinovsky
• Florian Bruhin
• George Y. Kussumoto
• Hugo
• Jesús Espino
• Joan Massich
• Ofir
• OfirOshir
• Ronny Pfannschmidt
• Samuel Dion-Girardeau
• Srinivas Reddy Thatiparthy
• Sviatoslav Abakumov
• Tarcisio Fischer
• Thomas Hisch
• Tyler Goodlet
• hugovk
• je
• prokaktus

Happy testing, The Pytest Development Team

pytest-3.2.5

pytest 3.2.5 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at http://doc.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Bruno Oliveira

Happy testing, The pytest Development Team

pytest-3.2.4

pytest 3.2.4 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at http://doc.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Bruno Oliveira
• Christian Boelsen
• Christoph Buchner
• Daw-Ran Liou
• Florian Bruhin
• Franck Michea
• Leonard Lausen
• Matty G
• Owen Tuz
• Pavel Karateev
• Pierre GIRAUD
• Ronny Pfannschmidt
• Stephen Finucane
• Sviatoslav Abakumov
• Thomas Hisch
• Tom Dalton
• Xuan Luong
• Yorgos Pagles
• Семён Марьясин

Happy testing, The pytest Development Team

pytest-3.2.3

pytest 3.2.3 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at http://doc.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Bruno Oliveira
• Evan
• Joe Hamman
• Oliver Bestwalter
• Ronny Pfannschmidt
• Xuan Luong

Happy testing, The pytest Development Team

pytest-3.2.2

pytest 3.2.2 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at http://doc.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Andreas Pelme
• Antonio Hidalgo
• Bruno Oliveira
• Felipe Dau
• Fernando Macedo
• Jesús Espino
• Joan Massich
• Joe Talbott
• Kirill Pinchuk
• Ronny Pfannschmidt
• Xuan Luong

Happy testing, The pytest Development Team

pytest-3.2.1

pytest 3.2.1 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at http://doc.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Alex Gaynor
• Bruno Oliveira
• Florian Bruhin
• Ronny Pfannschmidt
• Srinivas Reddy Thatiparthy

Happy testing, The pytest Development Team

pytest-3.2.0

The pytest team is proud to announce the 3.2.0 release!

pytest is a mature Python testing tool with more than a 1600 tests against itself, passing on many different interpreters and platforms.

This release contains a number of bugs fixes and improvements, so users are encouraged to take a look at the CHANGELOG:

http://doc.pytest.org/en/latest/changelog.html

For complete documentation, please visit:

http://docs.pytest.org

As usual, you can upgrade from pypi via:

pip install -U pytest

Thanks to all who contributed to this release, among them:

• Alex Hartoto
• Andras Tim
• Bruno Oliveira
• Daniel Hahler
• Florian Bruhin
• Floris Bruynooghe
• John Still
• Jordan Moldow
• Kale Kundert
• Lawrence Mitchell
• Llandy Riveron Del Risco
• Maik Figura
• Martin Altmayer
• Mihai Capotă
• Nathaniel Waisbrot
• Nguyễn Hồng Quân
• Pauli Virtanen
• Raphael Pierzina
• Ronny Pfannschmidt
• Segev Finer
• V.Kuznetsov

Happy testing, The Pytest Development Team

pytest-3.1.3

pytest 3.1.3 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at http://doc.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Antoine Legrand
• Bruno Oliveira
• Max Moroz
• Raphael Pierzina
• Ronny Pfannschmidt
• Ryan Fitzpatrick

Happy testing, The pytest Development Team

pytest-3.1.2

pytest 3.1.2 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at http://doc.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Andreas Pelme
• ApaDoctor
• Bruno Oliveira
• Florian Bruhin
• Ronny Pfannschmidt
• Segev Finer

Happy testing, The pytest Development Team

pytest-3.1.1

pytest 3.1.1 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at http://doc.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Bruno Oliveira
• Florian Bruhin
• Floris Bruynooghe
• Jason R. Coombs
• Ronny Pfannschmidt
• wanghui

Happy testing, The pytest Development Team

pytest-3.1.0

The pytest team is proud to announce the 3.1.0 release!

pytest is a mature Python testing tool with more than a 1600 tests against itself, passing on many different interpreters and platforms.

This release contains a bugs fixes and improvements, so users are encouraged to take a look at the CHANGELOG:

http://doc.pytest.org/en/latest/changelog.html

For complete documentation, please visit:

http://docs.pytest.org

As usual, you can upgrade from pypi via:

pip install -U pytest

Thanks to all who contributed to this release, among them:

• Anthony Sottile
• Ben Lloyd
• Bruno Oliveira
• David Giese
• David Szotten
• Dmitri Pribysh
• Florian Bruhin
• Florian Schulze
• Floris Bruynooghe
• John Towler
• Jonas Obrist
• Katerina Koukiou
• Kodi Arfer
• Krzysztof Szularz
• Lev Maximov
• Loïc Estève
• Luke Murphy
• Manuel Krebber
• Matthew Duck
• Matthias Bussonnier
• Michael Howitz
• Michal Wajszczuk
• Paweł Adamczak
• Rafael Bertoldi
• Ravi Chandra
• Ronny Pfannschmidt
• Skylar Downes
• Thomas Kriechbaumer
• Vitaly Lashmanov
• Vlad Dragos
• Wheerd
• Xander Johnson
• mandeep
• reut

Happy testing, The Pytest Development Team

pytest-3.0.7

pytest 3.0.7 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at http://doc.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Anthony Sottile
• Barney Gale
• Bruno Oliveira
• Florian Bruhin
• Floris Bruynooghe
• Ionel Cristian Mărieș
• Katerina Koukiou
• NODA, Kai
• Omer Hadari
• Patrick Hayes
• Ran Benita
• Ronny Pfannschmidt
• Victor Uriarte
• Vidar Tonaas Fauske
• Ville Skyttä
• fbjorn
• mbyt

Happy testing, The pytest Development Team

pytest-3.0.6

pytest 3.0.6 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The full changelog is available at http://doc.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Andreas Pelme
• Bruno Oliveira
• Dmitry Malinovsky
• Eli Boyarski
• Jakub Wilk
• Jeff Widman
• Loïc Estève
• Luke Murphy
• Miro Hrončok
• Oscar Hellström
• Peter Heatwole
• Philippe Ombredanne
• Ronny Pfannschmidt
• Rutger Prins
• Stefan Scherfke

Happy testing, The pytest Development Team

pytest-3.0.5

pytest 3.0.5 has just been released to PyPI.

This is a bug-fix release, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The changelog is available at http://doc.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Ana Vojnovic
• Bruno Oliveira
• Daniel Hahler
• Duncan Betts
• Igor Starikov
• Ismail
• Luke Murphy
• Ned Batchelder
• Ronny Pfannschmidt
• Sebastian Ramacher
• nmundar

Happy testing, The pytest Development Team

pytest-3.0.4

pytest 3.0.4 has just been released to PyPI.

This release fixes some regressions and bugs reported in the last version, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The changelog is available at http://doc.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Bruno Oliveira
• Dan Wandschneider
• Florian Bruhin
• Georgy Dyuldin
• Grigorii Eremeev
• Jason R. Coombs
• Manuel Jacob
• Mathieu Clabaut
• Michael Seifert
• Nikolaus Rath
• Ronny Pfannschmidt
• Tom V

Happy testing, The pytest Development Team

pytest-3.0.3

pytest 3.0.3 has just been released to PyPI.

This release fixes some regressions and bugs reported in the last version, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The changelog is available at http://doc.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Bruno Oliveira
• Florian Bruhin
• Floris Bruynooghe
• Huayi Zhang
• Lev Maximov
• Raquel Alegre
• Ronny Pfannschmidt
• Roy Williams
• Tyler Goodlet
• mbyt

Happy testing, The pytest Development Team

pytest-3.0.2

pytest 3.0.2 has just been released to PyPI.

This release fixes some regressions and bugs reported in version 3.0.1, being a drop-in replacement. To upgrade:

pip install --upgrade pytest

The changelog is available at http://doc.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

• Ahn Ki-Wook
• Bruno Oliveira
• Florian Bruhin
• Jordan Guymon
• Raphael Pierzina
• Ronny Pfannschmidt
• mbyt

Happy testing, The pytest Development Team

pytest-3.0.1

pytest 3.0.1 has just been released to PyPI.

This release fixes some regressions reported in version 3.0.0, being a drop-in replacement. To upgrade:

pip install –upgrade pytest

The changelog is available at http://doc.pytest.org/en/latest/changelog.html.

Thanks to all who contributed to this release, among them:

Adam Chainz Andrew Svetlov Bruno Oliveira Daniel Hahler Dmitry Dygalo Florian Bruhin Marcin Bachry Ronny Pfannschmidt matthiasha

Happy testing, The py.test Development Team

pytest-3.0.0

The pytest team is proud to announce the 3.0.0 release!

pytest is a mature Python testing tool with more than a 1600 tests against itself, passing on many different interpreters and platforms.

This release contains a lot of bugs fixes and improvements, and much of the work done on it was possible because of the 2016 Sprint[1], which was funded by an indiegogo campaign which raised over US$12,000 with nearly 100 backers.

There’s a “What’s new in pytest 3.0” [2] blog post highlighting the major features in this release.

To see the complete changelog and documentation, please visit:

http://docs.pytest.org

As usual, you can upgrade from pypi via:

pip install -U pytest

Thanks to all who contributed to this release, among them:

AbdealiJK Ana Ribeiro Antony Lee Brandon W Maister Brianna Laugher Bruno Oliveira Ceridwen Christian Boelsen Daniel Hahler Danielle Jenkins Dave Hunt Diego Russo Dmitry Dygalo Edoardo Batini Eli Boyarski Florian Bruhin Floris Bruynooghe Greg Price Guyzmo HEAD KANGAROO JJ Javi Romero Javier Domingo Cansino Kale Kundert Kalle Bronsen Marius Gedminas Matt Williams Mike Lundy Oliver Bestwalter Omar Kohl Raphael Pierzina RedBeardCode Roberto Polli Romain Dorgueil Roman Bolshakov Ronny Pfannschmidt Stefan Zimmermann Steffen Allner Tareq Alayan Ted Xiao Thomas Grainger Tom Viner TomV Vasily Kuznetsov aostr marscher palaviv satoru taschini

Happy testing, The Pytest Development Team

[1] http://blog.pytest.org/2016/pytest-development-sprint/ [2] http://blog.pytest.org/2016/whats-new-in-pytest-30/

python testing sprint June 20th-26th 2016

The pytest core group held the biggest sprint in its history in June 2016, taking place in the black forest town Freiburg in Germany. In February 2016 we started a funding campaign on Indiegogo to cover expenses The page also mentions some preliminary topics:

• improving pytest-xdist test scheduling to take into account fixture setups and explicit user hints.
• provide info on fixture dependencies during –collect-only
• tying pytest-xdist to tox so that you can do “py.test -e py34” to run tests in a particular tox-managed virtualenv. Also look into making pytest-xdist use tox environments on remote ssh-sides so that remote dependency management becomes easier.
• refactoring the fixture system so more people understand it :)
• integrating PyUnit setup methods as autouse fixtures. possibly adding ways to influence ordering of same-scoped fixtures (so you can make a choice of which fixtures come before others)
• fixing bugs and issues from the tracker, really an endless source :)

Participants

Over 20 participants took part from 4 continents, including employees from Splunk, Personalkollen, Cobe.io, FanDuel and Dolby. Some newcomers mixed with developers who have worked on pytest since its beginning, and of course everyone in between.

Sprint organisation, schedule

People arrived in Freiburg on the 19th, with sprint development taking place on 20th, 21st, 22nd, 24th and 25th. On the 23rd we took a break day for some hot hiking in the Black Forest.

Sprint activity was organised heavily around pairing, with plenty of group discusssions to take advantage of the high bandwidth, and lightning talks as well.

Money / funding

The Indiegogo campaign aimed for 11000 USD and in the end raised over 12000, to reimburse travel costs, pay for a sprint venue and catering.

Excess money is reserved for further sprint/travel funding for pytest/tox contributors.

pytest-2.9.2

pytest is a mature Python testing tool with more than a 1100 tests against itself, passing on many different interpreters and platforms.

See below for the changes and see docs at:

http://pytest.org

As usual, you can upgrade from pypi via:

pip install -U pytest

Thanks to all who contributed to this release, among them:

Adam Chainz Benjamin Dopplinger Bruno Oliveira Florian Bruhin John Towler Martin Prusse Meng Jue MengJueM Omar Kohl Quentin Pradet Ronny Pfannschmidt Thomas Güttler TomV Tyler Goodlet

Happy testing, The py.test Development Team

2.9.2 (compared to 2.9.1)

Bug Fixes

• fix #510: skip tests where one parameterize dimension was empty thanks Alex Stapleton for the Report and @RonnyPfannschmidt for the PR
• Fix Xfail does not work with condition keyword argument. Thanks @astraw38 for reporting the issue (#1496) and @tomviner for PR the (#1524).
• Fix win32 path issue when putting custom config file with absolute path in pytest.main("-c your_absolute_path").
• Fix maximum recursion depth detection when raised error class is not aware of unicode/encoded bytes. Thanks @prusse-martin for the PR (#1506).
• Fix pytest.mark.skip mark when used in strict mode. Thanks @pquentin for the PR and @RonnyPfannschmidt for showing how to fix the bug.
• Minor improvements and fixes to the documentation. Thanks @omarkohl for the PR.
• Fix --fixtures to show all fixture definitions as opposed to just one per fixture name. Thanks to @hackebrot for the PR.

pytest-2.9.1

pytest is a mature Python testing tool with more than a 1100 tests against itself, passing on many different interpreters and platforms.

See below for the changes and see docs at:

http://pytest.org

As usual, you can upgrade from pypi via:

pip install -U pytest

Thanks to all who contributed to this release, among them:

Bruno Oliveira Daniel Hahler Dmitry Malinovsky Florian Bruhin Floris Bruynooghe Matt Bachmann Ronny Pfannschmidt TomV Vladimir Bolshakov Zearin palaviv

Happy testing, The py.test Development Team

2.9.1 (compared to 2.9.0)

Bug Fixes

• Improve error message when a plugin fails to load. Thanks @nicoddemus for the PR.
• Fix (#1178): pytest.fail with non-ascii characters raises an internal pytest error. Thanks @nicoddemus for the PR.
• Fix (#469): junit parses report.nodeid incorrectly, when params IDs contain ::. Thanks @tomviner for the PR (#1431).
• Fix (#578): SyntaxErrors containing non-ascii lines at the point of failure generated an internal py.test error. Thanks @asottile for the report and @nicoddemus for the PR.
• Fix (#1437): When passing in a bytestring regex pattern to parameterize attempt to decode it as utf-8 ignoring errors.
• Fix (#649): parametrized test nodes cannot be specified to run on the command line.

pytest-2.9.0

pytest is a mature Python testing tool with more than a 1100 tests against itself, passing on many different interpreters and platforms.

See below for the changes and see docs at:

http://pytest.org

As usual, you can upgrade from pypi via:

pip install -U pytest

Thanks to all who contributed to this release, among them:

Anatoly Bubenkov Bruno Oliveira Buck Golemon David Vierra Florian Bruhin Galaczi Endre Georgy Dyuldin Lukas Bednar Luke Murphy Marcin Biernat Matt Williams Michael Aquilina Raphael Pierzina Ronny Pfannschmidt Ryan Wooden Tiemo Kieft TomV holger krekel jab

Happy testing, The py.test Development Team

2.9.0 (compared to 2.8.7)

New Features

• New pytest.mark.skip mark, which unconditionally skips marked tests. Thanks @MichaelAquilina for the complete PR (#1040).
• --doctest-glob may now be passed multiple times in the command-line. Thanks @jab and @nicoddemus for the PR.
• New -rp and -rP reporting options give the summary and full output of passing tests, respectively. Thanks to @codewarrior0 for the PR.
• pytest.mark.xfail now has a strict option which makes XPASS tests to fail the test suite, defaulting to False. There’s also a xfail_strict ini option that can be used to configure it project-wise. Thanks @rabbbit for the request and @nicoddemus for the PR (#1355).
• Parser.addini now supports options of type bool. Thanks @nicoddemus for the PR.
• New ALLOW_BYTES doctest option strips b prefixes from byte strings in doctest output (similar to ALLOW_UNICODE). Thanks @jaraco for the request and @nicoddemus for the PR (#1287).
• give a hint on KeyboardInterrupt to use the –fulltrace option to show the errors, this fixes #1366. Thanks to @hpk42 for the report and @RonnyPfannschmidt for the PR.
• catch IndexError exceptions when getting exception source location. This fixes pytest internal error for dynamically generated code (fixtures and tests) where source lines are fake by intention

Changes

• Important: py.code has been merged into the pytest repository as pytest._code. This decision was made because py.code had very few uses outside pytest and the fact that it was in a different repository made it difficult to fix bugs on its code in a timely manner. The team hopes with this to be able to better refactor out and improve that code. This change shouldn’t affect users, but it is useful to let users aware if they encounter any strange behavior.

  Keep in mind that the code for pytest._code is private and experimental, so you definitely should not import it explicitly!

  Please note that the original py.code is still available in pylib.

• pytest_enter_pdb now optionally receives the pytest config object. Thanks @nicoddemus for the PR.

• Removed code and documentation for Python 2.5 or lower versions, including removal of the obsolete _pytest.assertion.oldinterpret module. Thanks @nicoddemus for the PR (#1226).

• Comparisons now always show up in full when CI or BUILD_NUMBER is found in the environment, even when -vv isn’t used. Thanks @The-Compiler for the PR.

• --lf and --ff now support long names: --last-failed and --failed-first respectively. Thanks @MichaelAquilina for the PR.

• Added expected exceptions to pytest.raises fail message

• Collection only displays progress (“collecting X items”) when in a terminal. This avoids cluttering the output when using --color=yes to obtain colors in CI integrations systems (#1397).

Bug Fixes

• The -s and -c options should now work under xdist; Config.fromdictargs now represents its input much more faithfully. Thanks to @bukzor for the complete PR (#680).
• Fix (#1290): support Python 3.5’s @ operator in assertion rewriting. Thanks @Shinkenjoe for report with test case and @tomviner for the PR.
• Fix formatting utf-8 explanation messages (#1379). Thanks @biern for the PR.
• Fix traceback style docs to describe all of the available options (auto/long/short/line/native/no), with auto being the default since v2.6. Thanks @hackebrot for the PR.
• Fix (#1422): junit record_xml_property doesn’t allow multiple records with same name.

pytest-2.8.7

This is a hotfix release to solve a regression in the builtin monkeypatch plugin that got introduced in 2.8.6.

pytest is a mature Python testing tool with more than a 1100 tests against itself, passing on many different interpreters and platforms. This release is supposed to be drop-in compatible to 2.8.5.

See below for the changes and see docs at:

http://pytest.org

As usual, you can upgrade from pypi via:

pip install -U pytest

Thanks to all who contributed to this release, among them:

Ronny Pfannschmidt

Happy testing, The py.test Development Team

2.8.7 (compared to 2.8.6)

• fix #1338: use predictable object resolution for monkeypatch

pytest-2.8.6

pytest is a mature Python testing tool with more than a 1100 tests against itself, passing on many different interpreters and platforms. This release is supposed to be drop-in compatible to 2.8.5.

See below for the changes and see docs at:

http://pytest.org

As usual, you can upgrade from pypi via:

pip install -U pytest

Thanks to all who contributed to this release, among them:

AMiT Kumar Bruno Oliveira Erik M. Bray Florian Bruhin Georgy Dyuldin Jeff Widman Kartik Singhal Loïc Estève Manu Phatak Peter Demin Rick van Hattem Ronny Pfannschmidt Ulrich Petri foxx

Happy testing, The py.test Development Team

2.8.6 (compared to 2.8.5)

• fix #1259: allow for double nodeids in junitxml, this was a regression failing plugins combinations like pytest-pep8 + pytest-flakes
• Workaround for exception that occurs in pyreadline when using --pdb with standard I/O capture enabled. Thanks Erik M. Bray for the PR.
• fix #900: Better error message in case the target of a monkeypatch call raises an ImportError.
• fix #1292: monkeypatch calls (setattr, setenv, etc.) are now O(1). Thanks David R. MacIver for the report and Bruno Oliveira for the PR.
• fix #1223: captured stdout and stderr are now properly displayed before entering pdb when --pdb is used instead of being thrown away. Thanks Cal Leeming for the PR.
• fix #1305: pytest warnings emitted during pytest_terminal_summary are now properly displayed. Thanks Ionel Maries Cristian for the report and Bruno Oliveira for the PR.
• fix #628: fixed internal UnicodeDecodeError when doctests contain unicode. Thanks Jason R. Coombs for the report and Bruno Oliveira for the PR.
• fix #1334: Add captured stdout to jUnit XML report on setup error. Thanks Georgy Dyuldin for the PR.

pytest-2.8.5

pytest is a mature Python testing tool with more than a 1100 tests against itself, passing on many different interpreters and platforms. This release is supposed to be drop-in compatible to 2.8.4.

See below for the changes and see docs at:

http://pytest.org

As usual, you can upgrade from pypi via:

pip install -U pytest

Thanks to all who contributed to this release, among them:

Alex Gaynor aselus-hub Bruno Oliveira Ronny Pfannschmidt

Happy testing, The py.test Development Team

2.8.5 (compared to 2.8.4)

• fix #1243: fixed issue where class attributes injected during collection could break pytest. PR by Alexei Kozlenok, thanks Ronny Pfannschmidt and Bruno Oliveira for the review and help.
• fix #1074: precompute junitxml chunks instead of storing the whole tree in objects Thanks Bruno Oliveira for the report and Ronny Pfannschmidt for the PR
• fix #1238: fix pytest.deprecated_call() receiving multiple arguments (Regression introduced in 2.8.4). Thanks Alex Gaynor for the report and Bruno Oliveira for the PR.

pytest-2.8.4

pytest is a mature Python testing tool with more than a 1100 tests against itself, passing on many different interpreters and platforms. This release is supposed to be drop-in compatible to 2.8.2.

See below for the changes and see docs at:

http://pytest.org

As usual, you can upgrade from pypi via:

pip install -U pytest

Thanks to all who contributed to this release, among them:

Bruno Oliveira Florian Bruhin Jeff Widman Mehdy Khoshnoody Nicholas Chammas Ronny Pfannschmidt Tim Chan

Happy testing, The py.test Development Team

2.8.4 (compared to 2.8.3)

• fix #1190: deprecated_call() now works when the deprecated function has been already called by another test in the same module. Thanks Mikhail Chernykh for the report and Bruno Oliveira for the PR.
• fix #1198: --pastebin option now works on Python 3. Thanks Mehdy Khoshnoody for the PR.
• fix #1219: --pastebin now works correctly when captured output contains non-ascii characters. Thanks Bruno Oliveira for the PR.
• fix #1204: another error when collecting with a nasty __getattr__(). Thanks Florian Bruhin for the PR.
• fix the summary printed when no tests did run. Thanks Florian Bruhin for the PR.
• a number of documentation modernizations wrt good practices. Thanks Bruno Oliveira for the PR.

pytest-2.8.3: bug fixes

pytest is a mature Python testing tool with more than a 1100 tests against itself, passing on many different interpreters and platforms. This release is supposed to be drop-in compatible to 2.8.2.

See below for the changes and see docs at:

http://pytest.org

As usual, you can upgrade from pypi via:

pip install -U pytest

Thanks to all who contributed to this release, among them:

Bruno Oliveira Florian Bruhin Gabe Hollombe Gabriel Reis Hartmut Goebel John Vandenberg Lee Kamentsky Michael Birtwell Raphael Pierzina Ronny Pfannschmidt William Martin Stewart

Happy testing, The py.test Development Team

2.8.3 (compared to 2.8.2)

• fix #1169: add __name__ attribute to testcases in TestCaseFunction to support the @unittest.skip decorator on functions and methods. Thanks Lee Kamentsky for the PR.
• fix #1035: collecting tests if test module level obj has __getattr__(). Thanks Suor for the report and Bruno Oliveira / Tom Viner for the PR.
• fix #331: don’t collect tests if their failure cannot be reported correctly e.g. they are a callable instance of a class.
• fix #1133: fixed internal error when filtering tracebacks where one entry belongs to a file which is no longer available. Thanks Bruno Oliveira for the PR.
• enhancement made to highlight in red the name of the failing tests so they stand out in the output. Thanks Gabriel Reis for the PR.
• add more talks to the documentation
• extend documentation on the –ignore cli option
• use pytest-runner for setuptools integration
• minor fixes for interaction with OS X El Capitan system integrity protection (thanks Florian)

pytest-2.8.2: bug fixes

pytest is a mature Python testing tool with more than a 1100 tests against itself, passing on many different interpreters and platforms. This release is supposed to be drop-in compatible to 2.8.1.

See below for the changes and see docs at:

http://pytest.org

As usual, you can upgrade from pypi via:

pip install -U pytest

Thanks to all who contributed to this release, among them:

Bruno Oliveira Demian Brecht Florian Bruhin Ionel Cristian Mărieș Raphael Pierzina Ronny Pfannschmidt holger krekel

Happy testing, The py.test Development Team

2.8.2 (compared to 2.7.2)

• fix #1085: proper handling of encoding errors when passing encoded byte strings to pytest.parametrize in Python 2. Thanks Themanwithoutaplan for the report and Bruno Oliveira for the PR.
• fix #1087: handling SystemError when passing empty byte strings to pytest.parametrize in Python 3. Thanks Paul Kehrer for the report and Bruno Oliveira for the PR.
• fix #995: fixed internal error when filtering tracebacks where one entry was generated by an exec() statement. Thanks Daniel Hahler, Ashley C Straw, Philippe Gauthier and Pavel Savchenko for contributing and Bruno Oliveira for the PR.

pytest-2.7.2: bug fixes

pytest is a mature Python testing tool with more than a 1100 tests against itself, passing on many different interpreters and platforms. This release is supposed to be drop-in compatible to 2.7.1.

See below for the changes and see docs at:

http://pytest.org

As usual, you can upgrade from pypi via:

pip install -U pytest

Thanks to all who contributed to this release, among them:

Bruno Oliveira Floris Bruynooghe Punyashloka Biswal Aron Curzon Benjamin Peterson Thomas De Schampheleire Edison Gustavo Muenz Holger Krekel

Happy testing, The py.test Development Team

2.7.2 (compared to 2.7.1)

• fix issue767: pytest.raises value attribute does not contain the exception instance on Python 2.6. Thanks Eric Siegerman for providing the test case and Bruno Oliveira for PR.
• Automatically create directory for junitxml and results log. Thanks Aron Curzon.
• fix issue713: JUnit XML reports for doctest failures. Thanks Punyashloka Biswal.
• fix issue735: assertion failures on debug versions of Python 3.4+ Thanks Benjamin Peterson.
• fix issue114: skipif marker reports to internal skipping plugin; Thanks Floris Bruynooghe for reporting and Bruno Oliveira for the PR.
• fix issue748: unittest.SkipTest reports to internal pytest unittest plugin. Thanks Thomas De Schampheleire for reporting and Bruno Oliveira for the PR.
• fix issue718: failed to create representation of sets containing unsortable elements in python 2. Thanks Edison Gustavo Muenz
• fix issue756, fix issue752 (and similar issues): depend on py-1.4.29 which has a refined algorithm for traceback generation.

pytest-2.7.1: bug fixes

pytest is a mature Python testing tool with more than a 1100 tests against itself, passing on many different interpreters and platforms. This release is supposed to be drop-in compatible to 2.7.0.

See below for the changes and see docs at:

http://pytest.org

As usual, you can upgrade from pypi via:

pip install -U pytest

Thanks to all who contributed to this release, among them:

Bruno Oliveira Holger Krekel Ionel Maries Cristian Floris Bruynooghe

Happy testing, The py.test Development Team

2.7.1 (compared to 2.7.0)

• fix issue731: do not get confused by the braces which may be present and unbalanced in an object’s repr while collapsing False explanations. Thanks Carl Meyer for the report and test case.
• fix issue553: properly handling inspect.getsourcelines failures in FixtureLookupError which would lead to an internal error, obfuscating the original problem. Thanks talljosh for initial diagnose/patch and Bruno Oliveira for final patch.
• fix issue660: properly report scope-mismatch-access errors independently from ordering of fixture arguments. Also avoid the pytest internal traceback which does not provide information to the user. Thanks Holger Krekel.
• streamlined and documented release process. Also all versions (in setup.py and documentation generation) are now read from _pytest/__init__.py. Thanks Holger Krekel.
• fixed docs to remove the notion that yield-fixtures are experimental. They are here to stay :) Thanks Bruno Oliveira.
• Support building wheels by using environment markers for the requirements. Thanks Ionel Maries Cristian.
• fixed regression to 2.6.4 which surfaced e.g. in lost stdout capture printing when tests raised SystemExit. Thanks Holger Krekel.
• reintroduced _pytest fixture of the pytester plugin which is used at least by pytest-xdist.

pytest-2.7.0: fixes, features, speed improvements

pytest is a mature Python testing tool with more than a 1100 tests against itself, passing on many different interpreters and platforms. This release is supposed to be drop-in compatible to 2.6.X.

See below for the changes and see docs at:

http://pytest.org

As usual, you can upgrade from pypi via:

pip install -U pytest

Thanks to all who contributed, among them:

Anatoly Bubenkoff Floris Bruynooghe Brianna Laugher Eric Siegerman Daniel Hahler Charles Cloud Tom Viner Holger Peters Ldiary Translations almarklein

have fun, holger krekel

2.7.0 (compared to 2.6.4)

• fix issue435: make reload() work when assert rewriting is active. Thanks Daniel Hahler.
• fix issue616: conftest.py files and their contained fixtures are now properly considered for visibility, independently from the exact current working directory and test arguments that are used. Many thanks to Eric Siegerman and his PR235 which contains systematic tests for conftest visibility and now passes. This change also introduces the concept of a rootdir which is printed as a new pytest header and documented in the pytest customize web page.
• change reporting of “diverted” tests, i.e. tests that are collected in one file but actually come from another (e.g. when tests in a test class come from a base class in a different file). We now show the nodeid and indicate via a postfix the other file.
• add ability to set command line options by environment variable PYTEST_ADDOPTS.
• added documentation on the new pytest-dev teams on bitbucket and github. See https://pytest.org/en/latest/contributing.html . Thanks to Anatoly for pushing and initial work on this.
• fix issue650: new option --docttest-ignore-import-errors which will turn import errors in doctests into skips. Thanks Charles Cloud for the complete PR.
• fix issue655: work around different ways that cause python2/3 to leak sys.exc_info into fixtures/tests causing failures in 3rd party code
• fix issue615: assertion rewriting did not correctly escape % signs when formatting boolean operations, which tripped over mixing booleans with modulo operators. Thanks to Tom Viner for the report, triaging and fix.
• implement issue351: add ability to specify parametrize ids as a callable to generate custom test ids. Thanks Brianna Laugher for the idea and implementation.
• introduce and document new hookwrapper mechanism useful for plugins which want to wrap the execution of certain hooks for their purposes. This supersedes the undocumented __multicall__ protocol which pytest itself and some external plugins use. Note that pytest-2.8 is scheduled to drop supporting the old __multicall__ and only support the hookwrapper protocol.
• majorly speed up invocation of plugin hooks
• use hookwrapper mechanism in builtin pytest plugins.
• add a doctest ini option for doctest flags, thanks Holger Peters.
• add note to docs that if you want to mark a parameter and the parameter is a callable, you also need to pass in a reason to disambiguate it from the “decorator” case. Thanks Tom Viner.
• “python_classes” and “python_functions” options now support glob-patterns for test discovery, as discussed in issue600. Thanks Ldiary Translations.
• allow to override parametrized fixtures with non-parametrized ones and vice versa (bubenkoff).
• fix issue463: raise specific error for ‘parameterize’ misspelling (pfctdayelise).
• On failure, the sys.last_value, sys.last_type and sys.last_traceback are set, so that a user can inspect the error via postmortem debugging (almarklein).

pytest-2.6.3: fixes and little improvements

pytest is a mature Python testing tool with more than a 1100 tests against itself, passing on many different interpreters and platforms. This release is drop-in compatible to 2.5.2 and 2.6.X. See below for the changes and see docs at:

http://pytest.org

As usual, you can upgrade from pypi via:

pip install -U pytest

Thanks to all who contributed, among them:

Floris Bruynooghe Oleg Sinyavskiy Uwe Schmitt Charles Cloud Wolfgang Schnerring

have fun, holger krekel

Changes 2.6.3

• fix issue575: xunit-xml was reporting collection errors as failures instead of errors, thanks Oleg Sinyavskiy.
• fix issue582: fix setuptools example, thanks Laszlo Papp and Ronny Pfannschmidt.
• Fix infinite recursion bug when pickling capture.EncodedFile, thanks Uwe Schmitt.
• fix issue589: fix bad interaction with numpy and others when showing exceptions. Check for precise “maximum recursion depth exceed” exception instead of presuming any RuntimeError is that one (implemented in py dep). Thanks Charles Cloud for analysing the issue.
• fix conftest related fixture visibility issue: when running with a CWD outside of a test package pytest would get fixture discovery wrong. Thanks to Wolfgang Schnerring for figuring out a reproducible example.
• Introduce pytest_enter_pdb hook (needed e.g. by pytest_timeout to cancel the timeout when interactively entering pdb). Thanks Wolfgang Schnerring.
• check xfail/skip also with non-python function test items. Thanks Floris Bruynooghe.

pytest-2.6.2: few fixes and cx_freeze support

pytest is a mature Python testing tool with more than a 1100 tests against itself, passing on many different interpreters and platforms. This release is drop-in compatible to 2.5.2 and 2.6.X. It also brings support for including pytest with cx_freeze or similar freezing tools into your single-file app distribution. For details see the CHANGELOG below.

See docs at:

http://pytest.org

As usual, you can upgrade from pypi via:

pip install -U pytest

Thanks to all who contributed, among them:

Floris Bruynooghe Benjamin Peterson Bruno Oliveira

have fun, holger krekel

2.6.2

• Added function pytest.freeze_includes(), which makes it easy to embed pytest into executables using tools like cx_freeze. See docs for examples and rationale. Thanks Bruno Oliveira.
• Improve assertion rewriting cache invalidation precision.
• fixed issue561: adapt autouse fixture example for python3.
• fixed issue453: assertion rewriting issue with __repr__ containing “n{“, “n}” and “n~”.
• fix issue560: correctly display code if an “else:” or “finally:” is followed by statements on the same line.
• Fix example in monkeypatch documentation, thanks t-8ch.
• fix issue572: correct tmpdir doc example for python3.
• Do not mark as universal wheel because Python 2.6 is different from other builds due to the extra argparse dependency. Fixes issue566. Thanks sontek.

pytest-2.6.1: fixes and new xfail feature

pytest is a mature Python testing tool with more than a 1100 tests against itself, passing on many different interpreters and platforms. The 2.6.1 release is drop-in compatible to 2.5.2 and actually fixes some regressions introduced with 2.6.0. It also brings a little feature to the xfail marker which now recognizes expected exceptions, see the CHANGELOG below.

See docs at:

http://pytest.org

As usual, you can upgrade from pypi via:

pip install -U pytest

Thanks to all who contributed, among them:

Floris Bruynooghe Bruno Oliveira Nicolas Delaby

have fun, holger krekel

Changes 2.6.1

• No longer show line numbers in the –verbose output, the output is now purely the nodeid. The line number is still shown in failure reports. Thanks Floris Bruynooghe.
• fix issue437 where assertion rewriting could cause pytest-xdist slaves to collect different tests. Thanks Bruno Oliveira.
• fix issue555: add “errors” attribute to capture-streams to satisfy some distutils and possibly other code accessing sys.stdout.errors.
• fix issue547 capsys/capfd also work when output capturing (“-s”) is disabled.
• address issue170: allow pytest.mark.xfail(…) to specify expected exceptions via an optional “raises=EXC” argument where EXC can be a single exception or a tuple of exception classes. Thanks David Mohr for the complete PR.
• fix integration of pytest with unittest.mock.patch decorator when it uses the “new” argument. Thanks Nicolas Delaby for test and PR.
• fix issue with detecting conftest files if the arguments contain “::” node id specifications (copy pasted from “-v” output)
• fix issue544 by only removing “@NUM” at the end of “::” separated parts and if the part has a “.py” extension
• don’t use py.std import helper, rather import things directly. Thanks Bruno Oliveira.

pytest-2.6.0: shorter tracebacks, new warning system, test runner compat

pytest is a mature Python testing tool with more than a 1000 tests against itself, passing on many different interpreters and platforms.

The 2.6.0 release should be drop-in backward compatible to 2.5.2 and fixes a number of bugs and brings some new features, mainly:

• shorter tracebacks by default: only the first (test function) entry and the last (failure location) entry are shown, the ones between only in “short” format. Use --tb=long to get back the old behaviour of showing “long” entries everywhere.
• a new warning system which reports oddities during collection and execution. For example, ignoring collecting Test* classes with an __init__ now produces a warning.
• various improvements to nose/mock/unittest integration

Note also that 2.6.0 departs with the “zero reported bugs” policy because it has been too hard to keep up with it, unfortunately. Instead we are for now rather bound to work on “upvoted” issues in the https://bitbucket.org/pytest-dev/pytest/issues?status=new&status=open&sort=-votes issue tracker.

See docs at:

http://pytest.org

As usual, you can upgrade from pypi via:

pip install -U pytest

Thanks to all who contributed, among them:

Benjamin Peterson Jurko Gospodnetić Floris Bruynooghe Marc Abramowitz Marc Schlaich Trevor Bekolay Bruno Oliveira Alex Groenholm

have fun, holger krekel

2.6.0

• fix issue537: Avoid importing old assertion reinterpretation code by default. Thanks Benjamin Peterson.
• fix issue364: shorten and enhance tracebacks representation by default. The new “–tb=auto” option (default) will only display long tracebacks for the first and last entry. You can get the old behaviour of printing all entries as long entries with “–tb=long”. Also short entries by default are now printed very similarly to “–tb=native” ones.
• fix issue514: teach assertion reinterpretation about private class attributes Thanks Benjamin Peterson.
• change -v output to include full node IDs of tests. Users can copy a node ID from a test run, including line number, and use it as a positional argument in order to run only a single test.
• fix issue 475: fail early and comprehensible if calling pytest.raises with wrong exception type.
• fix issue516: tell in getting-started about current dependencies.
• cleanup setup.py a bit and specify supported versions. Thanks Jurko Gospodnetic for the PR.
• change XPASS colour to yellow rather then red when tests are run with -v.
• fix issue473: work around mock putting an unbound method into a class dict when double-patching.
• fix issue498: if a fixture finalizer fails, make sure that the fixture is still invalidated.
• fix issue453: the result of the pytest_assertrepr_compare hook now gets it’s newlines escaped so that format_exception does not blow up.
• internal new warning system: pytest will now produce warnings when it detects oddities in your test collection or execution. Warnings are ultimately sent to a new pytest_logwarning hook which is currently only implemented by the terminal plugin which displays warnings in the summary line and shows more details when -rw (report on warnings) is specified.
• change skips into warnings for test classes with an __init__ and callables in test modules which look like a test but are not functions.
• fix issue436: improved finding of initial conftest files from command line arguments by using the result of parse_known_args rather than the previous flaky heuristics. Thanks Marc Abramowitz for tests and initial fixing approaches in this area.
• fix issue #479: properly handle nose/unittest(2) SkipTest exceptions during collection/loading of test modules. Thanks to Marc Schlaich for the complete PR.
• fix issue490: include pytest_load_initial_conftests in documentation and improve docstring.
• fix issue472: clarify that pytest.config.getvalue() cannot work if it’s triggered ahead of command line parsing.
• merge PR123: improved integration with mock.patch decorator on tests.
• fix issue412: messing with stdout/stderr FD-level streams is now captured without crashes.
• fix issue483: trial/py33 works now properly. Thanks Daniel Grana for PR.
• improve example for pytest integration with “python setup.py test” which now has a generic “-a” or “–pytest-args” option where you can pass additional options as a quoted string. Thanks Trevor Bekolay.
• simplified internal capturing mechanism and made it more robust against tests or setups changing FD1/FD2, also better integrated now with pytest.pdb() in single tests.
• improvements to pytest’s own test-suite leakage detection, courtesy of PRs from Marc Abramowitz
• fix issue492: avoid leak in test_writeorg. Thanks Marc Abramowitz.
• fix issue493: don’t run tests in doc directory with python setup.py test (use tox -e doctesting for that)
• fix issue486: better reporting and handling of early conftest loading failures
• some cleanup and simplification of internal conftest handling.
• work a bit harder to break reference cycles when catching exceptions. Thanks Jurko Gospodnetic.
• fix issue443: fix skip examples to use proper comparison. Thanks Alex Groenholm.
• support nose-style __test__ attribute on modules, classes and functions, including unittest-style Classes. If set to False, the test will not be collected.
• fix issue512: show “<notset>” for arguments which might not be set in monkeypatch plugin. Improves output in documentation.
• avoid importing “py.test” (an old alias module for “pytest”)

pytest-2.5.2: fixes

pytest is a mature Python testing tool with more than a 1000 tests against itself, passing on many different interpreters and platforms.

The 2.5.2 release fixes a few bugs with two maybe-bugs remaining and actively being worked on (and waiting for the bug reporter’s input). We also have a new contribution guide thanks to Piotr Banaszkiewicz and others.

See docs at:

http://pytest.org

As usual, you can upgrade from pypi via:

pip install -U pytest

Thanks to the following people who contributed to this release:

Anatoly Bubenkov Ronny Pfannschmidt Floris Bruynooghe Bruno Oliveira Andreas Pelme Jurko Gospodnetić Piotr Banaszkiewicz Simon Liedtke lakka Lukasz Balcerzak Philippe Muller Daniel Hahler

have fun, holger krekel

2.5.2

• fix issue409 – better interoperate with cx_freeze by not trying to import from collections.abc which causes problems for py27/cx_freeze. Thanks Wolfgang L. for reporting and tracking it down.
• fixed docs and code to use “pytest” instead of “py.test” almost everywhere. Thanks Jurko Gospodnetic for the complete PR.
• fix issue425: mention at end of “py.test -h” that –markers and –fixtures work according to specified test path (or current dir)
• fix issue413: exceptions with unicode attributes are now printed correctly also on python2 and with pytest-xdist runs. (the fix requires py-1.4.20)
• copy, cleanup and integrate py.io capture from pylib 1.4.20.dev2 (rev 13d9af95547e)
• address issue416: clarify docs as to conftest.py loading semantics
• fix issue429: comparing byte strings with non-ascii chars in assert expressions now work better. Thanks Floris Bruynooghe.
• make capfd/capsys.capture private, its unused and shouldn’t be exposed

pytest-2.5.1: fixes and new home page styling

pytest is a mature Python testing tool with more than a 1000 tests against itself, passing on many different interpreters and platforms.

The 2.5.1 release maintains the “zero-reported-bugs” promise by fixing the three bugs reported since the last release a few days ago. It also features a new home page styling implemented by Tobias Bieniek, based on the flask theme from Armin Ronacher:

http://pytest.org

If you have anything more to improve styling and docs, we’d be very happy to merge further pull requests.

On the coding side, the release also contains a little enhancement to fixture decorators allowing to directly influence generation of test ids, thanks to Floris Bruynooghe. Other thanks for helping with this release go to Anatoly Bubenkoff and Ronny Pfannschmidt.

As usual, you can upgrade from pypi via:

pip install -U pytest

have fun and a nice remaining “bug-free” time of the year :) holger krekel

2.5.1

• merge new documentation styling PR from Tobias Bieniek.
• fix issue403: allow parametrize of multiple same-name functions within a collection node. Thanks Andreas Kloeckner and Alex Gaynor for reporting and analysis.
• Allow parameterized fixtures to specify the ID of the parameters by adding an ids argument to pytest.fixture() and pytest.yield_fixture(). Thanks Floris Bruynooghe.
• fix issue404 by always using the binary xml escape in the junitxml plugin. Thanks Ronny Pfannschmidt.
• fix issue407: fix addoption docstring to point to argparse instead of optparse. Thanks Daniel D. Wright.

pytest-2.5.0: now down to ZERO reported bugs!

pytest-2.5.0 is a big fixing release, the result of two community bug fixing days plus numerous additional works from many people and reporters. The release should be fully compatible to 2.4.2, existing plugins and test suites. We aim at maintaining this level of ZERO reported bugs because it’s no fun if your testing tool has bugs, is it? Under a condition, though: when submitting a bug report please provide clear information about the circumstances and a simple example which reproduces the problem.

The issue tracker is of course not empty now. We have many remaining “enhacement” issues which we’ll hopefully can tackle in 2014 with your help.

For those who use older Python versions, please note that pytest is not automatically tested on python2.5 due to virtualenv, setuptools and tox not supporting it anymore. Manual verification shows that it mostly works fine but it’s not going to be part of the automated release process and thus likely to break in the future.

As usual, current docs are at

http://pytest.org

and you can upgrade from pypi via:

pip install -U pytest

Particular thanks for helping with this release go to Anatoly Bubenkoff, Floris Bruynooghe, Marc Abramowitz, Ralph Schmitt, Ronny Pfannschmidt, Donald Stufft, James Lan, Rob Dennis, Jason R. Coombs, Mathieu Agopian, Virgil Dupras, Bruno Oliveira, Alex Gaynor and others.

have fun, holger krekel

2.5.0

• dropped python2.5 from automated release testing of pytest itself which means it’s probably going to break soon (but still works with this release we believe).

• simplified and fixed implementation for calling finalizers when parametrized fixtures or function arguments are involved. finalization is now performed lazily at setup time instead of in the “teardown phase”. While this might sound odd at first, it helps to ensure that we are correctly handling setup/teardown even in complex code. User-level code should not be affected unless it’s implementing the pytest_runtest_teardown hook and expecting certain fixture instances are torn down within (very unlikely and would have been unreliable anyway).

• PR90: add –color=yes|no|auto option to force terminal coloring mode (“auto” is default). Thanks Marc Abramowitz.

• fix issue319 - correctly show unicode in assertion errors. Many thanks to Floris Bruynooghe for the complete PR. Also means we depend on py>=1.4.19 now.

• fix issue396 - correctly sort and finalize class-scoped parametrized tests independently from number of methods on the class.

• refix issue323 in a better way – parametrization should now never cause Runtime Recursion errors because the underlying algorithm for re-ordering tests per-scope/per-fixture is not recursive anymore (it was tail-call recursive before which could lead to problems for more than >966 non-function scoped parameters).

• fix issue290 - there is preliminary support now for parametrizing with repeated same values (sometimes useful to test if calling a second time works as with the first time).

• close issue240 - document precisely how pytest module importing works, discuss the two common test directory layouts, and how it interacts with PEP420-namespace packages.

• fix issue246 fix finalizer order to be LIFO on independent fixtures depending on a parametrized higher-than-function scoped fixture. (was quite some effort so please bear with the complexity of this sentence :) Thanks Ralph Schmitt for the precise failure example.

• fix issue244 by implementing special index for parameters to only use indices for paramentrized test ids

• fix issue287 by running all finalizers but saving the exception from the first failing finalizer and re-raising it so teardown will still have failed. We reraise the first failing exception because it might be the cause for other finalizers to fail.

• fix ordering when mock.patch or other standard decorator-wrappings are used with test methods. This fixes issue346 and should help with random “xdist” collection failures. Thanks to Ronny Pfannschmidt and Donald Stufft for helping to isolate it.

• fix issue357 - special case “-k” expressions to allow for filtering with simple strings that are not valid python expressions. Examples: “-k 1.3” matches all tests parametrized with 1.3. “-k None” filters all tests that have “None” in their name and conversely “-k ‘not None’”. Previously these examples would raise syntax errors.

• fix issue384 by removing the trial support code since the unittest compat enhancements allow trial to handle it on its own

• don’t hide an ImportError when importing a plugin produces one. fixes issue375.

• fix issue275 - allow usefixtures and autouse fixtures for running doctest text files.

• fix issue380 by making –resultlog only rely on longrepr instead of the “reprcrash” attribute which only exists sometimes.

• address issue122: allow @pytest.fixture(params=iterator) by exploding into a list early on.

• fix pexpect-3.0 compatibility for pytest’s own tests. (fixes issue386)

• allow nested parametrize-value markers, thanks James Lan for the PR.

• fix unicode handling with new monkeypatch.setattr(import_path, value) API. Thanks Rob Dennis. Fixes issue371.

• fix unicode handling with junitxml, fixes issue368.

• In assertion rewriting mode on Python 2, fix the detection of coding cookies. See issue #330.

• make “–runxfail” turn imperative pytest.xfail calls into no ops (it already did neutralize pytest.mark.xfail markers)

• refine pytest / pkg_resources interactions: The AssertionRewritingHook PEP302 compliant loader now registers itself with setuptools/pkg_resources properly so that the pkg_resources.resource_stream method works properly. Fixes issue366. Thanks for the investigations and full PR to Jason R. Coombs.

• pytestconfig fixture is now session-scoped as it is the same object during the whole test run. Fixes issue370.

• avoid one surprising case of marker malfunction/confusion:

  @pytest.mark.some(lambda arg: ...)
  def test_function():
  

  would not work correctly because pytest assumes @pytest.mark.some gets a function to be decorated already. We now at least detect if this arg is a lambda and thus the example will work. Thanks Alex Gaynor for bringing it up.

• xfail a test on pypy that checks wrong encoding/ascii (pypy does not error out). fixes issue385.

• internally make varnames() deal with classes’s __init__, although it’s not needed by pytest itself atm. Also fix caching. Fixes issue376.

• fix issue221 - handle importing of namespace-package with no __init__.py properly.

• refactor internal FixtureRequest handling to avoid monkeypatching. One of the positive user-facing effects is that the “request” object can now be used in closures.

• fixed version comparison in pytest.importskip(modname, minverstring)

• fix issue377 by clarifying in the nose-compat docs that pytest does not duplicate the unittest-API into the “plain” namespace.

• fix verbose reporting for @mock’d test functions

pytest-2.4.2: colorama on windows, plugin/tmpdir fixes

pytest-2.4.2 is another bug-fixing release:

• on Windows require colorama and a newer py lib so that py.io.TerminalWriter() now uses colorama instead of its own ctypes hacks. (fixes issue365) thanks Paul Moore for bringing it up.
• fix “-k” matching of tests where “repr” and “attr” and other names would cause wrong matches because of an internal implementation quirk (don’t ask) which is now properly implemented. fixes issue345.
• avoid tmpdir fixture to create too long filenames especially when parametrization is used (issue354)
• fix pytest-pep8 and pytest-flakes / pytest interactions (collection names in mark plugin was assuming an item always has a function which is not true for those plugins etc.) Thanks Andi Zeidler.
• introduce node.get_marker/node.add_marker API for plugins like pytest-pep8 and pytest-flakes to avoid the messy details of the node.keywords pseudo-dicts. Adapted docs.
• remove attempt to “dup” stdout at startup as it’s icky. the normal capturing should catch enough possibilities of tests messing up standard FDs.
• add pluginmanager.do_configure(config) as a link to config.do_configure() for plugin-compatibility

as usual, docs at http://pytest.org and upgrades via:

pip install -U pytest

have fun, holger krekel

pytest-2.4.1: fixing three regressions compared to 2.3.5

pytest-2.4.1 is a quick follow up release to fix three regressions compared to 2.3.5 before they hit more people:

• When using parser.addoption() unicode arguments to the “type” keyword should also be converted to the respective types. thanks Floris Bruynooghe, @dnozay. (fixes issue360 and issue362)
• fix dotted filename completion when using argcomplete thanks Anthon van der Neuth. (fixes issue361)
• fix regression when a 1-tuple (“arg”,) is used for specifying parametrization (the values of the parametrization were passed nested in a tuple). Thanks Donald Stufft.
• also merge doc typo fixes, thanks Andy Dirnberger

as usual, docs at http://pytest.org and upgrades via:

pip install -U pytest

have fun, holger krekel

pytest-2.4.0: new fixture features/hooks and bug fixes

The just released pytest-2.4.0 brings many improvements and numerous bug fixes while remaining plugin- and test-suite compatible apart from a few supposedly very minor incompatibilities. See below for a full list of details. A few feature highlights:

• new yield-style fixtures pytest.yield_fixture, allowing to use existing with-style context managers in fixture functions.
• improved pdb support: import pdb ; pdb.set_trace() now works without requiring prior disabling of stdout/stderr capturing. Also the --pdb options works now on collection and internal errors and we introduced a new experimental hook for IDEs/plugins to intercept debugging: pytest_exception_interact(node, call, report).
• shorter monkeypatch variant to allow specifying an import path as a target, for example: monkeypatch.setattr("requests.get", myfunc)
• better unittest/nose compatibility: all teardown methods are now only called if the corresponding setup method succeeded.
• integrate tab-completion on command line options if you have argcomplete configured.
• allow boolean expression directly with skipif/xfail if a “reason” is also specified.
• a new hook pytest_load_initial_conftests allows plugins like pytest-django to influence the environment before conftest files import django.
• reporting: color the last line red or green depending if failures/errors occurred or everything passed.

The documentation has been updated to accommodate the changes, see http://pytest.org

To install or upgrade pytest:

pip install -U pytest # or
easy_install -U pytest

Many thanks to all who helped, including Floris Bruynooghe, Brianna Laugher, Andreas Pelme, Anthon van der Neut, Anatoly Bubenkoff, Vladimir Keleshev, Mathieu Agopian, Ronny Pfannschmidt, Christian Theunert and many others.

may passing tests be with you,

holger krekel

Changes between 2.3.5 and 2.4

known incompatibilities:

• if calling –genscript from python2.7 or above, you only get a standalone script which works on python2.7 or above. Use Python2.6 to also get a python2.5 compatible version.
• all xunit-style teardown methods (nose-style, pytest-style, unittest-style) will not be called if the corresponding setup method failed, see issue322 below.
• the pytest_plugin_unregister hook wasn’t ever properly called and there is no known implementation of the hook - so it got removed.
• pytest.fixture-decorated functions cannot be generators (i.e. use yield) anymore. This change might be reversed in 2.4.1 if it causes unforeseen real-life issues. However, you can always write and return an inner function/generator and change the fixture consumer to iterate over the returned generator. This change was done in lieu of the new pytest.yield_fixture decorator, see below.

new features:

• experimentally introduce a new pytest.yield_fixture decorator which accepts exactly the same parameters as pytest.fixture but mandates a yield statement instead of a return statement from fixture functions. This allows direct integration with “with-style” context managers in fixture functions and generally avoids registering of finalization callbacks in favour of treating the “after-yield” as teardown code. Thanks Andreas Pelme, Vladimir Keleshev, Floris Bruynooghe, Ronny Pfannschmidt and many others for discussions.

• allow boolean expression directly with skipif/xfail if a “reason” is also specified. Rework skipping documentation to recommend “condition as booleans” because it prevents surprises when importing markers between modules. Specifying conditions as strings will remain fully supported.

• reporting: color the last line red or green depending if failures/errors occurred or everything passed. thanks Christian Theunert.

• make “import pdb ; pdb.set_trace()” work natively wrt capturing (no “-s” needed anymore), making pytest.set_trace() a mere shortcut.

• fix issue181: –pdb now also works on collect errors (and on internal errors) . This was implemented by a slight internal refactoring and the introduction of a new hook pytest_exception_interact hook (see next item).

• fix issue341: introduce new experimental hook for IDEs/terminals to intercept debugging: pytest_exception_interact(node, call, report).

• new monkeypatch.setattr() variant to provide a shorter invocation for patching out classes/functions from modules:

  monkeypatch.setattr(“requests.get”, myfunc)

  will replace the “get” function of the “requests” module with myfunc.

• fix issue322: tearDownClass is not run if setUpClass failed. Thanks Mathieu Agopian for the initial fix. Also make all of pytest/nose finalizer mimic the same generic behaviour: if a setupX exists and fails, don’t run teardownX. This internally introduces a new method “node.addfinalizer()” helper which can only be called during the setup phase of a node.

• simplify pytest.mark.parametrize() signature: allow to pass a CSV-separated string to specify argnames. For example: pytest.mark.parametrize("input,expected",  [(1,2), (2,3)]) works as well as the previous: pytest.mark.parametrize(("input", "expected"), ...).

• add support for setUpModule/tearDownModule detection, thanks Brian Okken.

• integrate tab-completion on options through use of “argcomplete”. Thanks Anthon van der Neut for the PR.

• change option names to be hyphen-separated long options but keep the old spelling backward compatible. py.test -h will only show the hyphenated version, for example “–collect-only” but “–collectonly” will remain valid as well (for backward-compat reasons). Many thanks to Anthon van der Neut for the implementation and to Hynek Schlawack for pushing us.

• fix issue 308 - allow to mark/xfail/skip individual parameter sets when parametrizing. Thanks Brianna Laugher.

• call new experimental pytest_load_initial_conftests hook to allow 3rd party plugins to do something before a conftest is loaded.

Bug fixes:

• fix issue358 - capturing options are now parsed more properly by using a new parser.parse_known_args method.
• pytest now uses argparse instead of optparse (thanks Anthon) which means that “argparse” is added as a dependency if installing into python2.6 environments or below.
• fix issue333: fix a case of bad unittest/pytest hook interaction.
• PR27: correctly handle nose.SkipTest during collection. Thanks Antonio Cuni, Ronny Pfannschmidt.
• fix issue355: junitxml puts name=”pytest” attribute to testsuite tag.
• fix issue336: autouse fixture in plugins should work again.
• fix issue279: improve object comparisons on assertion failure for standard datatypes and recognise collections.abc. Thanks to Brianna Laugher and Mathieu Agopian.
• fix issue317: assertion rewriter support for the is_package method
• fix issue335: document py.code.ExceptionInfo() object returned from pytest.raises(), thanks Mathieu Agopian.
• remove implicit distribute_setup support from setup.py.
• fix issue305: ignore any problems when writing pyc files.
• SO-17664702: call fixture finalizers even if the fixture function partially failed (finalizers would not always be called before)
• fix issue320 - fix class scope for fixtures when mixed with module-level functions. Thanks Anatloy Bubenkoff.
• you can specify “-q” or “-qq” to get different levels of “quieter” reporting (thanks Katarzyna Jachim)
• fix issue300 - Fix order of conftest loading when starting py.test in a subdirectory.
• fix issue323 - sorting of many module-scoped arg parametrizations
• make sessionfinish hooks execute with the same cwd-context as at session start (helps fix plugin behaviour which write output files with relative path such as pytest-cov)
• fix issue316 - properly reference collection hooks in docs
• fix issue 306 - cleanup of -k/-m options to only match markers/test names/keywords respectively. Thanks Wouter van Ackooy.
• improved doctest counting for doctests in python modules – files without any doctest items will not show up anymore and doctest examples are counted as separate test items. thanks Danilo Bellini.
• fix issue245 by depending on the released py-1.4.14 which fixes py.io.dupfile to work with files with no mode. Thanks Jason R. Coombs.
• fix junitxml generation when test output contains control characters, addressing issue267, thanks Jaap Broekhuizen
• fix issue338: honor –tb style for setup/teardown errors as well. Thanks Maho.
• fix issue307 - use yaml.safe_load in example, thanks Mark Eichin.
• better parametrize error messages, thanks Brianna Laugher
• pytest_terminal_summary(terminalreporter) hooks can now use “.section(title)” and “.line(msg)” methods to print extra information at the end of a test run.

pytest-2.3.5: bug fixes and little improvements

pytest-2.3.5 is a maintenance release with many bug fixes and little improvements. See the changelog below for details. No backward compatibility issues are foreseen and all plugins which worked with the prior version are expected to work unmodified. Speaking of which, a few interesting new plugins saw the light last month:

• pytest-instafail: show failure information while tests are running
• pytest-qt: testing of GUI applications written with QT/Pyside
• pytest-xprocess: managing external processes across test runs
• pytest-random: randomize test ordering

And several others like pytest-django saw maintenance releases. For a more complete list, check out https://pypi.org/search/?q=pytest

For general information see:

http://pytest.org/

To install or upgrade pytest:

pip install -U pytest # or easy_install -U pytest

Particular thanks to Floris, Ronny, Benjamin and the many bug reporters and fix providers.

may the fixtures be with you, holger krekel

Changes between 2.3.4 and 2.3.5

• never consider a fixture function for test function collection
• allow re-running of test items / helps to fix pytest-reruntests plugin and also help to keep less fixture/resource references alive
• put captured stdout/stderr into junitxml output even for passing tests (thanks Adam Goucher)
• Issue 265 - integrate nose setup/teardown with setupstate so it doesn’t try to teardown if it did not setup
• issue 271 - don’t write junitxml on slave nodes
• Issue 274 - don’t try to show full doctest example when doctest does not know the example location
• issue 280 - disable assertion rewriting on buggy CPython 2.6.0
• inject “getfixture()” helper to retrieve fixtures from doctests, thanks Andreas Zeidler
• issue 259 - when assertion rewriting, be consistent with the default source encoding of ASCII on Python 2
• issue 251 - report a skip instead of ignoring classes with init
• issue250 unicode/str mixes in parametrization names and values now works
• issue257, assertion-triggered compilation of source ending in a comment line doesn’t blow up in python2.5 (fixed through py>=1.4.13.dev6)
• fix –genscript option to generate standalone scripts that also work with python3.3 (importer ordering)
• issue171 - in assertion rewriting, show the repr of some global variables
• fix option help for “-k”
• move long description of distribution into README.rst
• improve docstring for metafunc.parametrize()
• fix bug where using capsys with pytest.set_trace() in a test function would break when looking at capsys.readouterr()
• allow to specify prefixes starting with “_” when customizing python_functions test discovery. (thanks Graham Horler)
• improve PYTEST_DEBUG tracing output by putting extra data on a new lines with additional indent
• ensure OutcomeExceptions like skip/fail have initialized exception attributes
• issue 260 - don’t use nose special setup on plain unittest cases
• fix issue134 - print the collect errors that prevent running specified test items
• fix issue266 - accept unicode in MarkEvaluator expressions

pytest-2.3.4: stabilization, more flexible selection via “-k expr”

pytest-2.3.4 is a small stabilization release of the py.test tool which offers uebersimple assertions, scalable fixture mechanisms and deep customization for testing with Python. This release comes with the following fixes and features:

• make “-k” option accept an expressions the same as with “-m” so that one can write: -k “name1 or name2” etc. This is a slight usage incompatibility if you used special syntax like “TestClass.test_method” which you now need to write as -k “TestClass and test_method” to match a certain method in a certain test class.
• allow to dynamically define markers via item.keywords[…]=assignment integrating with “-m” option
• yielded test functions will now have autouse-fixtures active but cannot accept fixtures as funcargs - it’s anyway recommended to rather use the post-2.0 parametrize features instead of yield, see: http://pytest.org/en/latest/example/parametrize.html
• fix autouse-issue where autouse-fixtures would not be discovered if defined in an a/conftest.py file and tests in a/tests/test_some.py
• fix issue226 - LIFO ordering for fixture teardowns
• fix issue224 - invocations with >256 char arguments now work
• fix issue91 - add/discuss package/directory level setups in example
• fixes related to autouse discovery and calling

Thanks in particular to Thomas Waldmann for spotting and reporting issues.

See

http://pytest.org/

for general information. To install or upgrade pytest:

pip install -U pytest # or easy_install -U pytest

best, holger krekel

pytest-2.3.3: integration fixes, py24 support, */** shown in traceback

pytest-2.3.3 is another stabilization release of the py.test tool which offers uebersimple assertions, scalable fixture mechanisms and deep customization for testing with Python. Particularly, this release provides:

• integration fixes and improvements related to flask, numpy, nose, unittest, mock
• makes pytest work on py24 again (yes, people sometimes still need to use it)
• show *,** args in pytest tracebacks

Thanks to Manuel Jacob, Thomas Waldmann, Ronny Pfannschmidt, Pavel Repin and Andreas Taumoefolau for providing patches and all for the issues.

See

http://pytest.org/

for general information. To install or upgrade pytest:

pip install -U pytest # or easy_install -U pytest

best, holger krekel

Changes between 2.3.2 and 2.3.3

• fix issue214 - parse modules that contain special objects like e. g. flask’s request object which blows up on getattr access if no request is active. thanks Thomas Waldmann.
• fix issue213 - allow to parametrize with values like numpy arrays that do not support an __eq__ operator
• fix issue215 - split test_python.org into multiple files
• fix issue148 - @unittest.skip on classes is now recognized and avoids calling setUpClass/tearDownClass, thanks Pavel Repin
• fix issue209 - reintroduce python2.4 support by depending on newer pylib which re-introduced statement-finding for pre-AST interpreters
• nose support: only call setup if it’s a callable, thanks Andrew Taumoefolau
• fix issue219 - add py2.4-3.3 classifiers to TROVE list
• in tracebacks ,* arg values are now shown next to normal arguments (thanks Manuel Jacob)
• fix issue217 - support mock.patch with pytest’s fixtures - note that you need either mock-1.0.1 or the python3.3 builtin unittest.mock.
• fix issue127 - improve documentation for pytest_addoption() and add a config.getoption(name) helper function for consistency.

pytest-2.3.2: some fixes and more traceback-printing speed

pytest-2.3.2 is another stabilization release:

• issue 205: fixes a regression with conftest detection
• issue 208/29: fixes traceback-printing speed in some bad cases
• fix teardown-ordering for parametrized setups
• fix unittest and trial compat behaviour with respect to runTest() methods
• issue 206 and others: some improvements to packaging
• fix issue127 and others: improve some docs

See

http://pytest.org/

for general information. To install or upgrade pytest:

pip install -U pytest # or easy_install -U pytest

best, holger krekel

Changes between 2.3.1 and 2.3.2

• fix issue208 and fix issue29 use new py version to avoid long pauses when printing tracebacks in long modules
• fix issue205 - conftests in subdirs customizing pytest_pycollect_makemodule and pytest_pycollect_makeitem now work properly
• fix teardown-ordering for parametrized setups
• fix issue127 - better documentation for pytest_addoption and related objects.
• fix unittest behaviour: TestCase.runtest only called if there are test methods defined
• improve trial support: don’t collect its empty unittest.TestCase.runTest() method
• “python setup.py test” now works with pytest itself
• fix/improve internal/packaging related bits:
  • exception message check of test_nose.py now passes on python33 as well
  • issue206 - fix test_assertrewrite.py to work when a global PYTHONDONTWRITEBYTECODE=1 is present
  • add tox.ini to pytest distribution so that ignore-dirs and others config bits are properly distributed for maintainers who run pytest-own tests

pytest-2.3.1: fix regression with factory functions

pytest-2.3.1 is a quick follow-up release:

• fix issue202 - regression with fixture functions/funcarg factories: using “self” is now safe again and works as in 2.2.4. Thanks to Eduard Schettino for the quick bug report.
• disable pexpect pytest self tests on Freebsd - thanks Koob for the quick reporting
• fix/improve interactive docs with –markers

See

http://pytest.org/

for general information. To install or upgrade pytest:

pip install -U pytest # or easy_install -U pytest

best, holger krekel

Changes between 2.3.0 and 2.3.1

• fix issue202 - fix regression: using “self” from fixture functions now works as expected (it’s the same “self” instance that a test method which uses the fixture sees)
• skip pexpect using tests (test_pdb.py mostly) on freebsd* systems due to pexpect not supporting it properly (hanging)
• link to web pages from –markers output which provides help for pytest.mark.* usage.

pytest-2.3: improved fixtures / better unittest integration

pytest-2.3 comes with many major improvements for fixture/funcarg management and parametrized testing in Python. It is now easier, more efficient and more predictable to re-run the same tests with different fixture instances. Also, you can directly declare the caching “scope” of fixtures so that dependent tests throughout your whole test suite can re-use database or other expensive fixture objects with ease. Lastly, it’s possible for fixture functions (formerly known as funcarg factories) to use other fixtures, allowing for a completely modular and re-usable fixture design.

For detailed info and tutorial-style examples, see:

http://pytest.org/en/latest/fixture.html

Moreover, there is now support for using pytest fixtures/funcargs with unittest-style suites, see here for examples:

http://pytest.org/en/latest/unittest.html

Besides, more unittest-test suites are now expected to “simply work” with pytest.

All changes are backward compatible and you should be able to continue to run your test suites and 3rd party plugins that worked with pytest-2.2.4.

If you are interested in the precise reasoning (including examples) of the pytest-2.3 fixture evolution, please consult http://pytest.org/en/latest/funcarg_compare.html

For general info on installation and getting started:

http://pytest.org/en/latest/getting-started.html

Docs and PDF access as usual at:

http://pytest.org

and more details for those already in the knowing of pytest can be found in the CHANGELOG below.

Particular thanks for this release go to Floris Bruynooghe, Alex Okrushko Carl Meyer, Ronny Pfannschmidt, Benjamin Peterson and Alex Gaynor for helping to get the new features right and well integrated. Ronny and Floris also helped to fix a number of bugs and yet more people helped by providing bug reports.

have fun, holger krekel

Changes between 2.2.4 and 2.3.0

• fix issue202 - better automatic names for parametrized test functions
• fix issue139 - introduce @pytest.fixture which allows direct scoping and parametrization of funcarg factories. Introduce new @pytest.setup marker to allow the writing of setup functions which accept funcargs.
• fix issue198 - conftest fixtures were not found on windows32 in some circumstances with nested directory structures due to path manipulation issues
• fix issue193 skip test functions with were parametrized with empty parameter sets
• fix python3.3 compat, mostly reporting bits that previously depended on dict ordering
• introduce re-ordering of tests by resource and parametrization setup which takes precedence to the usual file-ordering
• fix issue185 monkeypatching time.time does not cause pytest to fail
• fix issue172 duplicate call of pytest.setup-decoratored setup_module functions
• fix junitxml=path construction so that if tests change the current working directory and the path is a relative path it is constructed correctly from the original current working dir.
• fix “python setup.py test” example to cause a proper “errno” return
• fix issue165 - fix broken doc links and mention stackoverflow for FAQ
• catch unicode-issues when writing failure representations to terminal to prevent the whole session from crashing
• fix xfail/skip confusion: a skip-mark or an imperative pytest.skip will now take precedence before xfail-markers because we can’t determine xfail/xpass status in case of a skip. see also: http://stackoverflow.com/questions/11105828/in-py-test-when-i-explicitly-skip-a-test-that-is-marked-as-xfail-how-can-i-get
• always report installed 3rd party plugins in the header of a test run
• fix issue160: a failing setup of an xfail-marked tests should be reported as xfail (not xpass)
• fix issue128: show captured output when capsys/capfd are used
• fix issue179: properly show the dependency chain of factories
• pluginmanager.register(…) now raises ValueError if the plugin has been already registered or the name is taken
• fix issue159: improve http://pytest.org/en/latest/faq.html especially with respect to the “magic” history, also mention pytest-django, trial and unittest integration.
• make request.keywords and node.keywords writable. All descendant collection nodes will see keyword values. Keywords are dictionaries containing markers and other info.
• fix issue 178: xml binary escapes are now wrapped in py.xml.raw
• fix issue 176: correctly catch the builtin AssertionError even when we replaced AssertionError with a subclass on the python level
• factory discovery no longer fails with magic global callables that provide no sane __code__ object (mock.call for example)
• fix issue 182: testdir.inprocess_run now considers passed plugins

• fix issue 188: ensure sys.exc_info is clear on python2

  before calling into a test

• fix issue 191: add unittest TestCase runTest method support
• fix issue 156: monkeypatch correctly handles class level descriptors
• reporting refinements:
  • pytest_report_header now receives a “startdir” so that you can use startdir.bestrelpath(yourpath) to show nice relative path
  • allow plugins to implement both pytest_report_header and pytest_sessionstart (sessionstart is invoked first).
  • don’t show deselected reason line if there is none
  • py.test -vv will show all of assert comparisons instead of truncating

pytest-2.2.4: bug fixes, better junitxml/unittest/python3 compat

pytest-2.2.4 is a minor backward-compatible release of the versatile py.test testing tool. It contains bug fixes and a few refinements to junitxml reporting, better unittest- and python3 compatibility.

For general information see here:

http://pytest.org/

To install or upgrade pytest:

pip install -U pytest # or easy_install -U pytest

Special thanks for helping on this release to Ronny Pfannschmidt and Benjamin Peterson and the contributors of issues.

best, holger krekel

Changes between 2.2.3 and 2.2.4

• fix error message for rewritten assertions involving the % operator
• fix issue 126: correctly match all invalid xml characters for junitxml binary escape
• fix issue with unittest: now @unittest.expectedFailure markers should be processed correctly (you can also use @pytest.mark markers)
• document integration with the extended distribute/setuptools test commands
• fix issue 140: properly get the real functions of bound classmethods for setup/teardown_class
• fix issue #141: switch from the deceased paste.pocoo.org to bpaste.net
• fix issue #143: call unconfigure/sessionfinish always when configure/sessionstart where called
• fix issue #144: better mangle test ids to junitxml classnames
• upgrade distribute_setup.py to 0.6.27

pytest-2.2.2: bug fixes

pytest-2.2.2 (updated to 2.2.3 to fix packaging issues) is a minor backward-compatible release of the versatile py.test testing tool. It contains bug fixes and a few refinements particularly to reporting with “–collectonly”, see below for betails.

For general information see here:

http://pytest.org/

To install or upgrade pytest:

pip install -U pytest # or easy_install -U pytest

Special thanks for helping on this release to Ronny Pfannschmidt and Ralf Schmitt and the contributors of issues.

best, holger krekel

Changes between 2.2.1 and 2.2.2

• fix issue101: wrong args to unittest.TestCase test function now produce better output
• fix issue102: report more useful errors and hints for when a test directory was renamed and some pyc/__pycache__ remain
• fix issue106: allow parametrize to be applied multiple times e.g. from module, class and at function level.
• fix issue107: actually perform session scope finalization
• don’t check in parametrize if indirect parameters are funcarg names
• add chdir method to monkeypatch funcarg
• fix crash resulting from calling monkeypatch undo a second time
• fix issue115: make –collectonly robust against early failure (missing files/directories)
• “-qq –collectonly” now shows only files and the number of tests in them
• “-q –collectonly” now shows test ids
• allow adding of attributes to test reports such that it also works with distributed testing (no upgrade of pytest-xdist needed)

pytest-2.2.1: bug fixes, perfect teardowns

pytest-2.2.1 is a minor backward-compatible release of the py.test testing tool. It contains bug fixes and little improvements, including documentation fixes. If you are using the distributed testing pluginmake sure to upgrade it to pytest-xdist-1.8.

For general information see here:

http://pytest.org/

To install or upgrade pytest:

pip install -U pytest # or easy_install -U pytest

Special thanks for helping on this release to Ronny Pfannschmidt, Jurko Gospodnetic and Ralf Schmitt.

best, holger krekel

Changes between 2.2.0 and 2.2.1

• fix issue99 (in pytest and py) internallerrors with resultlog now produce better output - fixed by normalizing pytest_internalerror input arguments.
• fix issue97 / traceback issues (in pytest and py) improve traceback output in conjunction with jinja2 and cython which hack tracebacks
• fix issue93 (in pytest and pytest-xdist) avoid “delayed teardowns”: the final test in a test node will now run its teardown directly instead of waiting for the end of the session. Thanks Dave Hunt for the good reporting and feedback. The pytest_runtest_protocol as well as the pytest_runtest_teardown hooks now have “nextitem” available which will be None indicating the end of the test run.
• fix collection crash due to unknown-source collected items, thanks to Ralf Schmitt (fixed by depending on a more recent pylib)

py.test 2.2.0: test marking++, parametrization++ and duration profiling

pytest-2.2.0 is a test-suite compatible release of the popular py.test testing tool. Plugins might need upgrades. It comes with these improvements:

• easier and more powerful parametrization of tests:
  • new @pytest.mark.parametrize decorator to run tests with different arguments
  • new metafunc.parametrize() API for parametrizing arguments independently
  • see examples at http://pytest.org/en/latest/example/parametrize.html
  • NOTE that parametrize() related APIs are still a bit experimental and might change in future releases.
• improved handling of test markers and refined marking mechanism:
  • “-m markexpr” option for selecting tests according to their mark
  • a new “markers” ini-variable for registering test markers for your project
  • the new “–strict” bails out with an error if using unregistered markers.
  • see examples at http://pytest.org/en/latest/example/markers.html
• duration profiling: new “–duration=N” option showing the N slowest test execution or setup/teardown calls. This is most useful if you want to find out where your slowest test code is.
• also 2.2.0 performs more eager calling of teardown/finalizers functions resulting in better and more accurate reporting when they fail

Besides there is the usual set of bug fixes along with a cleanup of pytest’s own test suite allowing it to run on a wider range of environments.

For general information, see extensive docs with examples here:

http://pytest.org/

If you want to install or upgrade pytest you might just type:

pip install -U pytest # or
easy_install -U pytest

Thanks to Ronny Pfannschmidt, David Burns, Jeff Donner, Daniel Nouri, Alfredo Deza and all who gave feedback or sent bug reports.

best, holger krekel

notes on incompatibility

While test suites should work unchanged you might need to upgrade plugins:

• You need a new version of the pytest-xdist plugin (1.7) for distributing test runs.
• Other plugins might need an upgrade if they implement the pytest_runtest_logreport hook which now is called unconditionally for the setup/teardown fixture phases of a test. You may choose to ignore setup/teardown failures by inserting “if rep.when != ‘call’: return” or something similar. Note that most code probably “just” works because the hook was already called for failing setup/teardown phases of a test so a plugin should have been ready to grok such reports already.

Changes between 2.1.3 and 2.2.0

• fix issue90: introduce eager tearing down of test items so that teardown function are called earlier.
• add an all-powerful metafunc.parametrize function which allows to parametrize test function arguments in multiple steps and therefore from independent plugins and places.
• add a @pytest.mark.parametrize helper which allows to easily call a test function with different argument values.
• Add examples to the “parametrize” example page, including a quick port of Test scenarios and the new parametrize function and decorator.
• introduce registration for “pytest.mark.*” helpers via ini-files or through plugin hooks. Also introduce a “–strict” option which will treat unregistered markers as errors allowing to avoid typos and maintain a well described set of markers for your test suite. See examples at http://pytest.org/en/latest/mark.html and its links.
• issue50: introduce “-m marker” option to select tests based on markers (this is a stricter and more predictable version of “-k” in that “-m” only matches complete markers and has more obvious rules for and/or semantics.
• new feature to help optimizing the speed of your tests: –durations=N option for displaying N slowest test calls and setup/teardown methods.
• fix issue87: –pastebin now works with python3
• fix issue89: –pdb with unexpected exceptions in doctest work more sensibly
• fix and cleanup pytest’s own test suite to not leak FDs
• fix issue83: link to generated funcarg list
• fix issue74: pyarg module names are now checked against imp.find_module false positives
• fix compatibility with twisted/trial-11.1.0 use cases

py.test 2.1.3: just some more fixes

pytest-2.1.3 is a minor backward compatible maintenance release of the popular py.test testing tool. It is commonly used for unit, functional- and integration testing. See extensive docs with examples here:

http://pytest.org/

The release contains another fix to the perfected assertions introduced with the 2.1 series as well as the new possibility to customize reporting for assertion expressions on a per-directory level.

If you want to install or upgrade pytest, just type one of:

pip install -U pytest # or
easy_install -U pytest

Thanks to the bug reporters and to Ronny Pfannschmidt, Benjamin Peterson and Floris Bruynooghe who implemented the fixes.

best, holger krekel

Changes between 2.1.2 and 2.1.3

• fix issue79: assertion rewriting failed on some comparisons in boolops,
• correctly handle zero length arguments (a la pytest ‘’)
• fix issue67 / junitxml now contains correct test durations
• fix issue75 / skipping test failure on jython
• fix issue77 / Allow assertrepr_compare hook to apply to a subset of tests

py.test 2.1.2: bug fixes and fixes for jython

pytest-2.1.2 is a minor backward compatible maintenance release of the popular py.test testing tool. pytest is commonly used for unit, functional- and integration testing. See extensive docs with examples here:

http://pytest.org/

Most bug fixes address remaining issues with the perfected assertions introduced in the 2.1 series - many thanks to the bug reporters and to Benjamin Peterson for helping to fix them. pytest should also work better with Jython-2.5.1 (and Jython trunk).

If you want to install or upgrade pytest, just type one of:

pip install -U pytest # or
easy_install -U pytest

best, holger krekel / http://merlinux.eu

Changes between 2.1.1 and 2.1.2

• fix assertion rewriting on files with windows newlines on some Python versions
• refine test discovery by package/module name (–pyargs), thanks Florian Mayer
• fix issue69 / assertion rewriting fixed on some boolean operations
• fix issue68 / packages now work with assertion rewriting
• fix issue66: use different assertion rewriting caches when the -O option is passed
• don’t try assertion rewriting on Jython, use reinterp

py.test 2.1.1: assertion fixes and improved junitxml output

pytest-2.1.1 is a backward compatible maintenance release of the popular py.test testing tool. See extensive docs with examples here:

http://pytest.org/

Most bug fixes address remaining issues with the perfected assertions introduced with 2.1.0 - many thanks to the bug reporters and to Benjamin Peterson for helping to fix them. Also, junitxml output now produces system-out/err tags which lead to better displays of tracebacks with Jenkins.

Also a quick note to package maintainers and others interested: there now is a “pytest” man page which can be generated with “make man” in doc/.

If you want to install or upgrade pytest, just type one of:

pip install -U pytest # or
easy_install -U pytest

best, holger krekel / http://merlinux.eu

Changes between 2.1.0 and 2.1.1

• fix issue64 / pytest.set_trace now works within pytest_generate_tests hooks
• fix issue60 / fix error conditions involving the creation of __pycache__
• fix issue63 / assertion rewriting on inserts involving strings containing ‘%’
• fix assertion rewriting on calls with a ** arg
• don’t cache rewritten modules if bytecode generation is disabled
• fix assertion rewriting in read-only directories
• fix issue59: provide system-out/err tags for junitxml output
• fix issue61: assertion rewriting on boolean operations with 3 or more operands
• you can now build a man page with “cd doc ; make man”

py.test 2.1.0: perfected assertions and bug fixes

Welcome to the release of pytest-2.1, a mature testing tool for Python, supporting CPython 2.4-3.2, Jython and latest PyPy interpreters. See the improved extensive docs (now also as PDF!) with tested examples here:

http://pytest.org/

The single biggest news about this release are perfected assertions courtesy of Benjamin Peterson. You can now safely use assert statements in test modules without having to worry about side effects or python optimization (“-OO”) options. This is achieved by rewriting assert statements in test modules upon import, using a PEP302 hook. See https://docs.pytest.org/en/latest/assert.html for detailed information. The work has been partly sponsored by my company, merlinux GmbH.

For further details on bug fixes and smaller enhancements see below.

If you want to install or upgrade pytest, just type one of:

pip install -U pytest # or
easy_install -U pytest

best, holger krekel / http://merlinux.eu

Changes between 2.0.3 and 2.1.0

• fix issue53 call nosestyle setup functions with correct ordering
• fix issue58 and issue59: new assertion code fixes
• merge Benjamin’s assertionrewrite branch: now assertions for test modules on python 2.6 and above are done by rewriting the AST and saving the pyc file before the test module is imported. see doc/assert.txt for more info.
• fix issue43: improve doctests with better traceback reporting on unexpected exceptions
• fix issue47: timing output in junitxml for test cases is now correct
• fix issue48: typo in MarkInfo repr leading to exception
• fix issue49: avoid confusing error when initialization partially fails
• fix issue44: env/username expansion for junitxml file path
• show releaselevel information in test runs for pypy
• reworked doc pages for better navigation and PDF generation
• report KeyboardInterrupt even if interrupted during session startup
• fix issue 35 - provide PDF doc version and download link from index page

py.test 2.0.3: bug fixes and speed ups

Welcome to pytest-2.0.3, a maintenance and bug fix release of pytest, a mature testing tool for Python, supporting CPython 2.4-3.2, Jython and latest PyPy interpreters. See the extensive docs with tested examples here:

http://pytest.org/

If you want to install or upgrade pytest, just type one of:

pip install -U pytest # or
easy_install -U pytest

There also is a bugfix release 1.6 of pytest-xdist, the plugin that enables seamless distributed and “looponfail” testing for Python.

best, holger krekel

Changes between 2.0.2 and 2.0.3

• fix issue38: nicer tracebacks on calls to hooks, particularly early configure/sessionstart ones
• fix missing skip reason/meta information in junitxml files, reported via http://lists.idyll.org/pipermail/testing-in-python/2011-March/003928.html
• fix issue34: avoid collection failure with “test” prefixed classes deriving from object.
• don’t require zlib (and other libs) for genscript plugin without –genscript actually being used.
• speed up skips (by not doing a full traceback representation internally)
• fix issue37: avoid invalid characters in junitxml’s output

py.test 2.0.2: bug fixes, improved xfail/skip expressions, speed ups

Welcome to pytest-2.0.2, a maintenance and bug fix release of pytest, a mature testing tool for Python, supporting CPython 2.4-3.2, Jython and latest PyPy interpreters. See the extensive docs with tested examples here:

http://pytest.org/

If you want to install or upgrade pytest, just type one of:

pip install -U pytest # or
easy_install -U pytest

Many thanks to all issue reporters and people asking questions or complaining, particularly Jurko for his insistence, Laura, Victor and Brianna for helping with improving and Ronny for his general advise.

best, holger krekel

Changes between 2.0.1 and 2.0.2

• tackle issue32 - speed up test runs of very quick test functions by reducing the relative overhead

• fix issue30 - extended xfail/skipif handling and improved reporting. If you have a syntax error in your skip/xfail expressions you now get nice error reports.

  Also you can now access module globals from xfail/skipif expressions so that this for example works now:

  import pytest
  import mymodule
  @pytest.mark.skipif("mymodule.__version__[0] == "1")
  def test_function():
      pass
  

  This will not run the test function if the module’s version string does not start with a “1”. Note that specifying a string instead of a boolean expressions allows py.test to report meaningful information when summarizing a test run as to what conditions lead to skipping (or xfail-ing) tests.

• fix issue28 - setup_method and pytest_generate_tests work together The setup_method fixture method now gets called also for test function invocations generated from the pytest_generate_tests hook.

• fix issue27 - collectonly and keyword-selection (-k) now work together Also, if you do “py.test –collectonly -q” you now get a flat list of test ids that you can use to paste to the py.test commandline in order to execute a particular test.

• fix issue25 avoid reported problems with –pdb and python3.2/encodings output

• fix issue23 - tmpdir argument now works on Python3.2 and WindowsXP Starting with Python3.2 os.symlink may be supported. By requiring a newer py lib version the py.path.local() implementation acknowledges this.

• fixed typos in the docs (thanks Victor Garcia, Brianna Laugher) and particular thanks to Laura Creighton who also reviewed parts of the documentation.

• fix slightly wrong output of verbose progress reporting for classes (thanks Amaury)

• more precise (avoiding of) deprecation warnings for node.Class|Function accesses

• avoid std unittest assertion helper code in tracebacks (thanks Ronny)

py.test 2.0.1: bug fixes

Welcome to pytest-2.0.1, a maintenance and bug fix release of pytest, a mature testing tool for Python, supporting CPython 2.4-3.2, Jython and latest PyPy interpreters. See extensive docs with tested examples here:

http://pytest.org/

If you want to install or upgrade pytest, just type one of:

pip install -U pytest # or
easy_install -U pytest

Many thanks to all issue reporters and people asking questions or complaining. Particular thanks to Floris Bruynooghe and Ronny Pfannschmidt for their great coding contributions and many others for feedback and help.

best, holger krekel

Changes between 2.0.0 and 2.0.1

• refine and unify initial capturing so that it works nicely even if the logging module is used on an early-loaded conftest.py file or plugin.
• fix issue12 - show plugin versions with “–version” and “–traceconfig” and also document how to add extra information to reporting test header
• fix issue17 (import-* reporting issue on python3) by requiring py>1.4.0 (1.4.1 is going to include it)
• fix issue10 (numpy arrays truth checking) by refining assertion interpretation in py lib
• fix issue15: make nose compatibility tests compatible with python3 (now that nose-1.0 supports python3)
• remove somewhat surprising “same-conftest” detection because it ignores conftest.py when they appear in several subdirs.
• improve assertions (“not in”), thanks Floris Bruynooghe
• improve behaviour/warnings when running on top of “python -OO” (assertions and docstrings are turned off, leading to potential false positives)
• introduce a pytest_cmdline_processargs(args) hook to allow dynamic computation of command line arguments. This fixes a regression because py.test prior to 2.0 allowed to set command line options from conftest.py files which so far pytest-2.0 only allowed from ini-files now.
• fix issue7: assert failures in doctest modules. unexpected failures in doctests will not generally show nicer, i.e. within the doctest failing context.
• fix issue9: setup/teardown functions for an xfail-marked test will report as xfail if they fail but report as normally passing (not xpassing) if they succeed. This only is true for “direct” setup/teardown invocations because teardown_class/ teardown_module cannot closely relate to a single test.
• fix issue14: no logging errors at process exit
• refinements to “collecting” output on non-ttys
• refine internal plugin registration and –traceconfig output
• introduce a mechanism to prevent/unregister plugins from the command line, see http://pytest.org/en/latest/plugins.html#cmdunregister
• activate resultlog plugin by default
• fix regression wrt yielded tests which due to the collection-before-running semantics were not setup as with pytest 1.3.4. Note, however, that the recommended and much cleaner way to do test parametrization remains the “pytest_generate_tests” mechanism, see the docs.

py.test 2.0.0: asserts++, unittest++, reporting++, config++, docs++

Welcome to pytest-2.0.0, a major new release of “py.test”, the rapid easy Python testing tool. There are many new features and enhancements, see below for summary and detailed lists. A lot of long-deprecated code has been removed, resulting in a much smaller and cleaner implementation. See the new docs with examples here:

http://pytest.org/en/latest/index.html

A note on packaging: pytest used to part of the “py” distribution up until version py-1.3.4 but this has changed now: pytest-2.0.0 only contains py.test related code and is expected to be backward-compatible to existing test code. If you want to install pytest, just type one of:

pip install -U pytest
easy_install -U pytest

Many thanks to all issue reporters and people asking questions or complaining. Particular thanks to Floris Bruynooghe and Ronny Pfannschmidt for their great coding contributions and many others for feedback and help.

best, holger krekel

New Features

• new invocations through Python interpreter and from Python:

  python -m pytest      # on all pythons >= 2.5
  

  or from a python program:

  import pytest ; pytest.main(arglist, pluginlist)
  

  see http://pytest.org/en/latest/usage.html for details.

• new and better reporting information in assert expressions if comparing lists, sequences or strings.

  see http://pytest.org/en/latest/assert.html#newreport

• new configuration through ini-files (setup.cfg or tox.ini recognized), for example:

  [pytest]
  norecursedirs = .hg data*  # don't ever recurse in such dirs
  addopts = -x --pyargs      # add these command line options by default
  

  see http://pytest.org/en/latest/customize.html

• improved standard unittest support. In general py.test should now better be able to run custom unittest.TestCases like twisted trial or Django based TestCases. Also you can now run the tests of an installed ‘unittest’ package with py.test:

  py.test --pyargs unittest
  

• new “-q” option which decreases verbosity and prints a more nose/unittest-style “dot” output.

• many many more detailed improvements details

Fixes

• fix issue126 - introduce py.test.set_trace() to trace execution via PDB during the running of tests even if capturing is ongoing.
• fix issue124 - make reporting more resilient against tests opening files on filedescriptor 1 (stdout).
• fix issue109 - sibling conftest.py files will not be loaded. (and Directory collectors cannot be customized anymore from a Directory’s conftest.py - this needs to happen at least one level up).
• fix issue88 (finding custom test nodes from command line arg)
• fix issue93 stdout/stderr is captured while importing conftest.py
• fix bug: unittest collected functions now also can have “pytestmark” applied at class/module level

Important Notes

• The usual way in pre-2.0 times to use py.test in python code was to import “py” and then e.g. use “py.test.raises” for the helper. This remains valid and is not planned to be deprecated. However, in most examples and internal code you’ll find “import pytest” and “pytest.raises” used as the recommended default way.
• pytest now first performs collection of the complete test suite before running any test. This changes for example the semantics of when pytest_collectstart/pytest_collectreport are called. Some plugins may need upgrading.
• The pytest package consists of a 400 LOC core.py and about 20 builtin plugins, summing up to roughly 5000 LOCs, including docstrings. To be fair, it also uses generic code from the “pylib”, and the new “py” package to help with filesystem and introspection/code manipulation.

(Incompatible) Removals

• py.test.config is now only available if you are in a test run.

• the following (mostly already deprecated) functionality was removed:

  • removed support for Module/Class/… collection node definitions in conftest.py files. They will cause nothing special.
  • removed support for calling the pre-1.0 collection API of “run()” and “join”
  • removed reading option values from conftest.py files or env variables. This can now be done much much better and easier through the ini-file mechanism and the “addopts” entry in particular.
  • removed the “disabled” attribute in test classes. Use the skipping and pytestmark mechanism to skip or xfail a test class.

• py.test.collect.Directory does not exist anymore and it is not possible to provide an own “Directory” object. If you have used this and don’t know what to do, get in contact. We’ll figure something out.

  Note that pytest_collect_directory() is still called but any return value will be ignored. This allows to keep old code working that performed for example “py.test.skip()” in collect() to prevent recursion into directory trees if a certain dependency or command line option is missing.

see Changelog for more detailed changes.

Changelog

Versions follow Semantic Versioning (<major>.<minor>.<patch>).

Backward incompatible (breaking) changes will only be introduced in major versions with advance notice in the Deprecations section of releases.

pytest 5.3.5 (2020-01-29)

Bug Fixes

• #6517: Fix regression in pytest 5.3.4 causing an INTERNALERROR due to a wrong assertion.

pytest 5.3.4 (2020-01-20)

Bug Fixes

• #6496: Revert #6436: unfortunately this change has caused a number of regressions in many suites, so the team decided to revert this change and make a new release while we continue to look for a solution.

pytest 5.3.3 (2020-01-16)

Bug Fixes

• #2780: Captured output during teardown is shown with -rP.
• #5971: Fix a pytest-xdist crash when dealing with exceptions raised in subprocesses created by the multiprocessing module.
• #6436: FixtureDef objects now properly register their finalizers with autouse and parameterized fixtures that execute before them in the fixture stack so they are torn down at the right times, and in the right order.
• #6532: Fix parsing of outcomes containing multiple errors with testdir results (regression in 5.3.0).

Trivial/Internal Changes

• #6350: Optimized automatic renaming of test parameter IDs.

pytest 5.3.2 (2019-12-13)

Improvements

• #4639: Revert “A warning is now issued when assertions are made for None”.

  The warning proved to be less useful than initially expected and had quite a few false positive cases.

Bug Fixes

• #5430: junitxml: Logs for failed test are now passed to junit report in case the test fails during call phase.
• #6290: The supporting files in the .pytest_cache directory are kept with --cache-clear, which only clears cached values now.
• #6301: Fix assertion rewriting for egg-based distributions and editable installs (pip install --editable).

pytest 5.3.1 (2019-11-25)

Improvements

• #6231: Improve check for misspelling of pytest.mark.parametrize.
• #6257: Handle _pytest.outcomes.exit() being used via pytest_internalerror(), e.g. when quitting pdb from post mortem.

Bug Fixes

• #5914: pytester: fix no_fnmatch_line() when used after positive matching.
• #6082: Fix line detection for doctest samples inside property docstrings, as a workaround to bpo-17446.
• #6254: Fix compatibility with pytest-parallel (regression in pytest 5.3.0).
• #6255: Clear the sys.last_traceback, sys.last_type and sys.last_value attributes by deleting them instead of setting them to None. This better matches the behaviour of the Python standard library.

pytest 5.3.0 (2019-11-19)

Deprecations

• #6179: The default value of junit_family option will change to "xunit2" in pytest 6.0, given that this is the version supported by default in modern tools that manipulate this type of file.

  In order to smooth the transition, pytest will issue a warning in case the --junitxml option is given in the command line but junit_family is not explicitly configured in pytest.ini.

  For more information, see the docs.

Features

• #4488: The pytest team has created the pytest-reportlog plugin, which provides a new --report-log=FILE option that writes report logs into a file as the test session executes.

  Each line of the report log contains a self contained JSON object corresponding to a testing event, such as a collection or a test result report. The file is guaranteed to be flushed after writing each line, so systems can read and process events in real-time.

  The plugin is meant to replace the --resultlog option, which is deprecated and meant to be removed in a future release. If you use --resultlog, please try out pytest-reportlog and provide feedback.

• #4730: When sys.pycache_prefix (Python 3.8+) is set, it will be used by pytest to cache test files changed by the assertion rewriting mechanism.

  This makes it easier to benefit of cached .pyc files even on file systems without permissions.

• #5515: Allow selective auto-indentation of multiline log messages.

  Adds command line option --log-auto-indent, config option log_auto_indent and support for per-entry configuration of indentation behavior on calls to logging.log().

  Alters the default for auto-indention from "on" to "off". This restores the older behavior that existed prior to v4.6.0. This reversion to earlier behavior was done because it is better to activate new features that may lead to broken tests explicitly rather than implicitly.

• #5914: testdir learned two new functions, no_fnmatch_line() and no_re_match_line().

  The functions are used to ensure the captured text does not match the given pattern.

  The previous idiom was to use re.match():

  result = testdir.runpytest()
  assert re.match(pat, result.stdout.str()) is None
  

  Or the in operator:

  result = testdir.runpytest()
  assert text in result.stdout.str()
  

  But the new functions produce best output on failure.

• #6057: Added tolerances to complex values when printing pytest.approx.

  For example, repr(pytest.approx(3+4j)) returns (3+4j) ± 5e-06 ∠ ±180°. This is polar notation indicating a circle around the expected value, with a radius of 5e-06. For approx comparisons to return True, the actual value should fall within this circle.

• #6061: Added the pluginmanager as an argument to pytest_addoption so that hooks can be invoked when setting up command line options. This is useful for having one plugin communicate things to another plugin, such as default values or which set of command line options to add.

Improvements

• #5061: Use multiple colors with terminal summary statistics.

• #5630: Quitting from debuggers is now properly handled in doctest items.

• #5924: Improved verbose diff output with sequences.

  Before:

  E   AssertionError: assert ['version', '...version_info'] == ['version', '...version', ...]
  E     Right contains 3 more items, first extra item: ' '
  E     Full diff:
  E     - ['version', 'version_info', 'sys.version', 'sys.version_info']
  E     + ['version',
  E     +  'version_info',
  E     +  'sys.version',
  E     +  'sys.version_info',
  E     +  ' ',
  E     +  'sys.version',
  E     +  'sys.version_info']
  

  After:

  E   AssertionError: assert ['version', '...version_info'] == ['version', '...version', ...]
  E     Right contains 3 more items, first extra item: ' '
  E     Full diff:
  E       [
  E        'version',
  E        'version_info',
  E        'sys.version',
  E        'sys.version_info',
  E     +  ' ',
  E     +  'sys.version',
  E     +  'sys.version_info',
  E       ]
  

• #5934: repr of ExceptionInfo objects has been improved to honor the __repr__ method of the underlying exception.

• #5936: Display untruncated assertion message with -vv.

• #5990: Fixed plurality mismatch in test summary (e.g. display “1 error” instead of “1 errors”).

• #6008: Config.InvocationParams.args is now always a tuple to better convey that it should be immutable and avoid accidental modifications.

• #6023: pytest.main returns a pytest.ExitCode instance now, except for when custom exit codes are used (where it returns int then still).

• #6026: Align prefixes in output of pytester’s LineMatcher.

• #6059: Collection errors are reported as errors (and not failures like before) in the terminal’s short test summary.

• #6069: pytester.spawn does not skip/xfail tests on FreeBSD anymore unconditionally.

• #6097: The “[…%]” indicator in the test summary is now colored according to the final (new) multi-colored line’s main color.

• #6116: Added --co as a synonym to --collect-only.

• #6148: atomicwrites is now only used on Windows, fixing a performance regression with assertion rewriting on Unix.

• #6152: Now parametrization will use the __name__ attribute of any object for the id, if present. Previously it would only use __name__ for functions and classes.

• #6176: Improved failure reporting with pytester’s Hookrecorder.assertoutcome.

• #6181: The reason for a stopped session, e.g. with --maxfail / -x, now gets reported in the test summary.

• #6206: Improved cache.set robustness and performance.

Bug Fixes

• #2049: Fixed --setup-plan showing inaccurate information about fixture lifetimes.

• #2548: Fixed line offset mismatch of skipped tests in terminal summary.

• #6039: The PytestDoctestRunner is now properly invalidated when unconfiguring the doctest plugin.

  This is important when used with pytester’s runpytest_inprocess.

• #6047: BaseExceptions are now handled in saferepr, which includes pytest.fail.Exception etc.

• #6074: pytester: fixed order of arguments in rm_rf warning when cleaning up temporary directories, and do not emit warnings for errors with os.open.

• #6189: Fixed result of getmodpath method.

Trivial/Internal Changes

• #4901: RunResult from pytester now displays the mnemonic of the ret attribute when it is a valid pytest.ExitCode value.

pytest 5.2.4 (2019-11-15)

Bug Fixes

• #6194: Fix incorrect discovery of non-test __init__.py files.
• #6197: Revert “The first test in a package (__init__.py) marked with @pytest.mark.skip is now correctly skipped.”.

pytest 5.2.3 (2019-11-14)

Bug Fixes

• #5830: The first test in a package (__init__.py) marked with @pytest.mark.skip is now correctly skipped.
• #6099: Fix --trace when used with parametrized functions.
• #6183: Using request as a parameter name in @pytest.mark.parametrize now produces a more user-friendly error.

pytest 5.2.2 (2019-10-24)

Bug Fixes

• #5206: Fix --nf to not forget about known nodeids with partial test selection.
• #5906: Fix crash with KeyboardInterrupt during --setup-show.
• #5946: Fixed issue when parametrizing fixtures with numpy arrays (and possibly other sequence-like types).
• #6044: Properly ignore FileNotFoundError exceptions when trying to remove old temporary directories, for instance when multiple processes try to remove the same directory (common with pytest-xdist for example).

pytest 5.2.1 (2019-10-06)

Bug Fixes

• #5902: Fix warnings about deprecated cmp attribute in attrs>=19.2.

pytest 5.2.0 (2019-09-28)

Deprecations

• #1682: Passing arguments to pytest.fixture() as positional arguments is deprecated - pass them as a keyword argument instead.

Features

• #1682: The scope parameter of @pytest.fixture can now be a callable that receives the fixture name and the config object as keyword-only parameters. See the docs for more information.
• #5764: New behavior of the --pastebin option: failures to connect to the pastebin server are reported, without failing the pytest run

Bug Fixes

• #5806: Fix “lexer” being used when uploading to bpaste.net from --pastebin to “text”.
• #5884: Fix --setup-only and --setup-show for custom pytest items.

Trivial/Internal Changes

• #5056: The HelpFormatter uses py.io.get_terminal_width for better width detection.

pytest 5.1.3 (2019-09-18)

Bug Fixes

• #5807: Fix pypy3.6 (nightly) on windows.
• #5811: Handle --fulltrace correctly with pytest.raises.
• #5819: Windows: Fix regression with conftest whose qualified name contains uppercase characters (introduced by #5792).

pytest 5.1.2 (2019-08-30)

Bug Fixes

• #2270: Fixed self reference in function-scoped fixtures defined plugin classes: previously self would be a reference to a test class, not the plugin class.
• #570: Fixed long standing issue where fixture scope was not respected when indirect fixtures were used during parametrization.
• #5782: Fix decoding error when printing an error response from --pastebin.
• #5786: Chained exceptions in test and collection reports are now correctly serialized, allowing plugins like pytest-xdist to display them properly.
• #5792: Windows: Fix error that occurs in certain circumstances when loading conftest.py from a working directory that has casing other than the one stored in the filesystem (e.g., c:\test instead of C:\test).

pytest 5.1.1 (2019-08-20)

Bug Fixes

• #5751: Fixed TypeError when importing pytest on Python 3.5.0 and 3.5.1.

pytest 5.1.0 (2019-08-15)

Removals

• #5180: As per our policy, the following features have been deprecated in the 4.X series and are now removed:

  • Request.getfuncargvalue: use Request.getfixturevalue instead.
  • pytest.raises and pytest.warns no longer support strings as the second argument.
  • message parameter of pytest.raises.
  • pytest.raises, pytest.warns and ParameterSet.param now use native keyword-only syntax. This might change the exception message from previous versions, but they still raise TypeError on unknown keyword arguments as before.
  • pytest.config global variable.
  • tmpdir_factory.ensuretemp method.
  • pytest_logwarning hook.
  • RemovedInPytest4Warning warning type.
  • request is now a reserved name for fixtures.

  For more information consult Deprecations and Removals in the docs.

• #5565: Removed unused support code for unittest2.

  The unittest2 backport module is no longer necessary since Python 3.3+, and the small amount of code in pytest to support it also doesn’t seem to be used: after removed, all tests still pass unchanged.

  Although our policy is to introduce a deprecation period before removing any features or support for third party libraries, because this code is apparently not used at all (even if unittest2 is used by a test suite executed by pytest), it was decided to remove it in this release.

  If you experience a regression because of this, please file an issue.

• #5615: pytest.fail, pytest.xfail and pytest.skip no longer support bytes for the message argument.

  This was supported for Python 2 where it was tempting to use "message" instead of u"message".

  Python 3 code is unlikely to pass bytes to these functions. If you do, please decode it to an str beforehand.

Features

• #5564: New Config.invocation_args attribute containing the unchanged arguments passed to pytest.main().
• #5576: New NUMBER option for doctests to ignore irrelevant differences in floating-point numbers. Inspired by Sébastien Boisgérault’s numtest extension for doctest.

Improvements

• #5471: JUnit XML now includes a timestamp and hostname in the testsuite tag.

• #5707: Time taken to run the test suite now includes a human-readable representation when it takes over 60 seconds, for example:

  ===== 2 failed in 102.70s (0:01:42) =====
  

Bug Fixes

• #4344: Fix RuntimeError/StopIteration when trying to collect package with “__init__.py” only.
• #5115: Warnings issued during pytest_configure are explicitly not treated as errors, even if configured as such, because it otherwise completely breaks pytest.
• #5477: The XML file produced by --junitxml now correctly contain a <testsuites> root element.
• #5524: Fix issue where tmp_path and tmpdir would not remove directories containing files marked as read-only, which could lead to pytest crashing when executed a second time with the --basetemp option.
• #5537: Replace importlib_metadata backport with importlib.metadata from the standard library on Python 3.8+.
• #5578: Improve type checking for some exception-raising functions (pytest.xfail, pytest.skip, etc) so they provide better error messages when users meant to use marks (for example @pytest.xfail instead of @pytest.mark.xfail).
• #5606: Fixed internal error when test functions were patched with objects that cannot be compared for truth values against others, like numpy arrays.
• #5634: pytest.exit is now correctly handled in unittest cases. This makes unittest cases handle quit from pytest’s pdb correctly.
• #5650: Improved output when parsing an ini configuration file fails.
• #5701: Fix collection of staticmethod objects defined with functools.partial.
• #5734: Skip async generator test functions, and update the warning message to refer to async def functions.

Improved Documentation

• #5669: Add docstring for Testdir.copy_example.

Trivial/Internal Changes

• #5095: XML files of the xunit2 family are now validated against the schema by pytest’s own test suite to avoid future regressions.
• #5516: Cache node splitting function which can improve collection performance in very large test suites.
• #5603: Simplified internal SafeRepr class and removed some dead code.
• #5664: When invoking pytest’s own testsuite with PYTHONDONTWRITEBYTECODE=1, the test_xfail_handling test no longer fails.
• #5684: Replace manual handling of OSError.errno in the codebase by new OSError subclasses (PermissionError, FileNotFoundError, etc.).

pytest 5.0.1 (2019-07-04)

Bug Fixes

• #5479: Improve quoting in raises match failure message.
• #5523: Fixed using multiple short options together in the command-line (for example -vs) in Python 3.8+.
• #5547: --step-wise now handles xfail(strict=True) markers properly.

Improved Documentation

• #5517: Improve “Declaring new hooks” section in chapter “Writing Plugins”

pytest 5.0.0 (2019-06-28)

Important

This release is a Python3.5+ only release.

For more details, see our Python 2.7 and 3.4 support plan.

Removals

• #1149: Pytest no longer accepts prefixes of command-line arguments, for example typing pytest --doctest-mod inplace of --doctest-modules. This was previously allowed where the ArgumentParser thought it was unambiguous, but this could be incorrect due to delayed parsing of options for plugins. See for example issues #1149, #3413, and #4009.

• #5402: PytestDeprecationWarning are now errors by default.

  Following our plan to remove deprecated features with as little disruption as possible, all warnings of type PytestDeprecationWarning now generate errors instead of warning messages.

  The affected features will be effectively removed in pytest 5.1, so please consult the Deprecations and Removals section in the docs for directions on how to update existing code.

  In the pytest 5.0.X series, it is possible to change the errors back into warnings as a stop gap measure by adding this to your pytest.ini file:

  [pytest]
  filterwarnings =
      ignore::pytest.PytestDeprecationWarning
  

  But this will stop working when pytest 5.1 is released.

  If you have concerns about the removal of a specific feature, please add a comment to #5402.

• #5412: ExceptionInfo objects (returned by pytest.raises) now have the same str representation as repr, which avoids some confusion when users use print(e) to inspect the object.

  This means code like:

  with pytest.raises(SomeException) as e:
      ...
  assert "some message" in str(e)
  

  Needs to be changed to:

  with pytest.raises(SomeException) as e:
      ...
  assert "some message" in str(e.value)
  

Deprecations

• #4488: The removal of the --result-log option and module has been postponed to (tentatively) pytest 6.0 as the team has not yet got around to implement a good alternative for it.
• #466: The funcargnames attribute has been an alias for fixturenames since pytest 2.3, and is now deprecated in code too.

Features

• #3457: New pytest_assertion_pass hook, called with context information when an assertion passes.

  This hook is still experimental so use it with caution.

• #5440: The faulthandler standard library module is now enabled by default to help users diagnose crashes in C modules.

  This functionality was provided by integrating the external pytest-faulthandler plugin into the core, so users should remove that plugin from their requirements if used.

  For more information see the docs: https://docs.pytest.org/en/latest/usage.html#fault-handler

• #5452: When warnings are configured as errors, pytest warnings now appear as originating from pytest. instead of the internal _pytest.warning_types. module.

• #5125: Session.exitcode values are now coded in pytest.ExitCode, an IntEnum. This makes the exit code available for consumer code and are more explicit other than just documentation. User defined exit codes are still valid, but should be used with caution.

  The team doesn’t expect this change to break test suites or plugins in general, except in esoteric/specific scenarios.

  pytest-xdist users should upgrade to 1.29.0 or later, as pytest-xdist required a compatibility fix because of this change.

Bug Fixes

• #1403: Switch from imp to importlib.

• #1671: The name of the .pyc files cached by the assertion writer now includes the pytest version to avoid stale caches.

• #2761: Honor PEP 235 on case-insensitive file systems.

• #5078: Test module is no longer double-imported when using --pyargs.

• #5260: Improved comparison of byte strings.

  When comparing bytes, the assertion message used to show the byte numeric value when showing the differences:

      def test():
  >       assert b'spam' == b'eggs'
  E       AssertionError: assert b'spam' == b'eggs'
  E         At index 0 diff: 115 != 101
  E         Use -v to get the full diff
  

  It now shows the actual ascii representation instead, which is often more useful:

      def test():
  >       assert b'spam' == b'eggs'
  E       AssertionError: assert b'spam' == b'eggs'
  E         At index 0 diff: b's' != b'e'
  E         Use -v to get the full diff
  

• #5335: Colorize level names when the level in the logging format is formatted using ‘%(levelname).Xs’ (truncated fixed width alignment), where X is an integer.

• #5354: Fix pytest.mark.parametrize when the argvalues is an iterator.

• #5370: Revert unrolling of all() to fix NameError on nested comprehensions.

• #5371: Revert unrolling of all() to fix incorrect handling of generators with if.

• #5372: Revert unrolling of all() to fix incorrect assertion when using all() in an expression.

• #5383: -q has again an impact on the style of the collected items (--collect-only) when --log-cli-level is used.

• #5389: Fix regressions of #5063 for importlib_metadata.PathDistribution which have their files attribute being None.

• #5390: Fix regression where the obj attribute of TestCase items was no longer bound to methods.

• #5404: Emit a warning when attempting to unwrap a broken object raises an exception, for easier debugging (#5080).

• #5432: Prevent “already imported” warnings from assertion rewriter when invoking pytest in-process multiple times.

• #5433: Fix assertion rewriting in packages (__init__.py).

• #5444: Fix --stepwise mode when the first file passed on the command-line fails to collect.

• #5482: Fix bug introduced in 4.6.0 causing collection errors when passing more than 2 positional arguments to pytest.mark.parametrize.

• #5505: Fix crash when discovery fails while using -p no:terminal.

Improved Documentation

• #5315: Expand docs on mocking classes and dictionaries with monkeypatch.
• #5416: Fix PytestUnknownMarkWarning in run/skip example.

pytest 4.6.9 (2020-01-04)

Bug Fixes

• #6301: Fix assertion rewriting for egg-based distributions and editable installs (pip install --editable).

pytest 4.6.8 (2019-12-19)

Features

• #5471: JUnit XML now includes a timestamp and hostname in the testsuite tag.

Bug Fixes

• #5430: junitxml: Logs for failed test are now passed to junit report in case the test fails during call phase.

Trivial/Internal Changes

• #6345: Pin colorama to 0.4.1 only for Python 3.4 so newer Python versions can still receive colorama updates.

pytest 4.6.7 (2019-12-05)

Bug Fixes

• #5477: The XML file produced by --junitxml now correctly contain a <testsuites> root element.
• #6044: Properly ignore FileNotFoundError (OSError.errno == NOENT in Python 2) exceptions when trying to remove old temporary directories, for instance when multiple processes try to remove the same directory (common with pytest-xdist for example).

pytest 4.6.6 (2019-10-11)

Bug Fixes

• #5523: Fixed using multiple short options together in the command-line (for example -vs) in Python 3.8+.
• #5537: Replace importlib_metadata backport with importlib.metadata from the standard library on Python 3.8+.
• #5806: Fix “lexer” being used when uploading to bpaste.net from --pastebin to “text”.
• #5902: Fix warnings about deprecated cmp attribute in attrs>=19.2.

Trivial/Internal Changes

• #5801: Fixes python version checks (detected by flake8-2020) in case python4 becomes a thing.

pytest 4.6.5 (2019-08-05)

Bug Fixes

• #4344: Fix RuntimeError/StopIteration when trying to collect package with “__init__.py” only.
• #5478: Fix encode error when using unicode strings in exceptions with pytest.raises.
• #5524: Fix issue where tmp_path and tmpdir would not remove directories containing files marked as read-only, which could lead to pytest crashing when executed a second time with the --basetemp option.
• #5547: --step-wise now handles xfail(strict=True) markers properly.
• #5650: Improved output when parsing an ini configuration file fails.

pytest 4.6.4 (2019-06-28)

Bug Fixes

• #5404: Emit a warning when attempting to unwrap a broken object raises an exception, for easier debugging (#5080).
• #5444: Fix --stepwise mode when the first file passed on the command-line fails to collect.
• #5482: Fix bug introduced in 4.6.0 causing collection errors when passing more than 2 positional arguments to pytest.mark.parametrize.
• #5505: Fix crash when discovery fails while using -p no:terminal.

pytest 4.6.3 (2019-06-11)

Bug Fixes

• #5383: -q has again an impact on the style of the collected items (--collect-only) when --log-cli-level is used.
• #5389: Fix regressions of #5063 for importlib_metadata.PathDistribution which have their files attribute being None.
• #5390: Fix regression where the obj attribute of TestCase items was no longer bound to methods.

pytest 4.6.2 (2019-06-03)

Bug Fixes

• #5370: Revert unrolling of all() to fix NameError on nested comprehensions.
• #5371: Revert unrolling of all() to fix incorrect handling of generators with if.
• #5372: Revert unrolling of all() to fix incorrect assertion when using all() in an expression.

pytest 4.6.1 (2019-06-02)

Bug Fixes

• #5354: Fix pytest.mark.parametrize when the argvalues is an iterator.
• #5358: Fix assertion rewriting of all() calls to deal with non-generators.

pytest 4.6.0 (2019-05-31)

Important

The 4.6.X series will be the last series to support Python 2 and Python 3.4.

For more details, see our Python 2.7 and 3.4 support plan.

Features

• #4559: Added the junit_log_passing_tests ini value which can be used to enable or disable logging of passing test output in the Junit XML file.
• #4956: pytester’s testdir.spawn uses tmpdir as HOME/USERPROFILE directory.
• #5062: Unroll calls to all to full for-loops with assertion rewriting for better failure messages, especially when using Generator Expressions.
• #5063: Switch from pkg_resources to importlib-metadata for entrypoint detection for improved performance and import time.
• #5091: The output for ini options in --help has been improved.
• #5269: pytest.importorskip includes the ImportError now in the default reason.
• #5311: Captured logs that are output for each failing test are formatted using the ColoredLevelFormatter.
• #5312: Improved formatting of multiline log messages in Python 3.

Bug Fixes

• #2064: The debugging plugin imports the wrapped Pdb class (--pdbcls) on-demand now.
• #4908: The pytest_enter_pdb hook gets called with post-mortem (--pdb).
• #5036: Fix issue where fixtures dependent on other parametrized fixtures would be erroneously parametrized.
• #5256: Handle internal error due to a lone surrogate unicode character not being representable in Jython.
• #5257: Ensure that sys.stdout.mode does not include 'b' as it is a text stream.
• #5278: Pytest’s internal python plugin can be disabled using -p no:python again.
• #5286: Fix issue with disable_test_id_escaping_and_forfeit_all_rights_to_community_support option not working when using a list of test IDs in parametrized tests.
• #5330: Show the test module being collected when emitting PytestCollectionWarning messages for test classes with __init__ and __new__ methods to make it easier to pin down the problem.
• #5333: Fix regression in 4.5.0 with --lf not re-running all tests with known failures from non-selected tests.

Improved Documentation

• #5250: Expand docs on use of setenv and delenv with monkeypatch.

pytest 4.5.0 (2019-05-11)

Features

• #4826: A warning is now emitted when unknown marks are used as a decorator. This is often due to a typo, which can lead to silently broken tests.

• #4907: Show XFail reason as part of JUnitXML message field.

• #5013: Messages from crash reports are displayed within test summaries now, truncated to the terminal width.

• #5023: New flag --strict-markers that triggers an error when unknown markers (e.g. those not registered using the markers option in the configuration file) are used in the test suite.

  The existing --strict option has the same behavior currently, but can be augmented in the future for additional checks.

• #5026: Assertion failure messages for sequences and dicts contain the number of different items now.

• #5034: Improve reporting with --lf and --ff (run-last-failure).

• #5035: The --cache-show option/action accepts an optional glob to show only matching cache entries.

• #5059: Standard input (stdin) can be given to pytester’s Testdir.run() and Testdir.popen().

• #5068: The -r option learnt about A to display all reports (including passed ones) in the short test summary.

• #5108: The short test summary is displayed after passes with output (-rP).

• #5172: The --last-failed (--lf) option got smarter and will now skip entire files if all tests of that test file have passed in previous runs, greatly speeding up collection.

• #5177: Introduce new specific warning PytestWarning subclasses to make it easier to filter warnings based on the class, rather than on the message. The new subclasses are:

  • PytestAssertRewriteWarning
  • PytestCacheWarning
  • PytestCollectionWarning
  • PytestConfigWarning
  • PytestUnhandledCoroutineWarning
  • PytestUnknownMarkWarning

• #5202: New record_testsuite_property session-scoped fixture allows users to log <property> tags at the testsuite level with the junitxml plugin.

  The generated XML is compatible with the latest xunit standard, contrary to the properties recorded by record_property and record_xml_attribute.

• #5214: The default logging format has been changed to improve readability. Here is an example of a previous logging message:

  test_log_cli_enabled_disabled.py    3 CRITICAL critical message logged by test
  

  This has now become:

  CRITICAL root:test_log_cli_enabled_disabled.py:3 critical message logged by test
  

  The formatting can be changed through the log_format configuration option.

• #5220: --fixtures now also shows fixture scope for scopes other than "function".

Bug Fixes

• #5113: Deselected items from plugins using pytest_collect_modifyitems as a hookwrapper are correctly reported now.
• #5144: With usage errors exitstatus is set to EXIT_USAGEERROR in the pytest_sessionfinish hook now as expected.
• #5235: outcome.exit is not used with EOF in the pdb wrapper anymore, but only with quit.

Improved Documentation

• #4935: Expand docs on registering marks and the effect of --strict.

Trivial/Internal Changes

• #4942: logging.raiseExceptions is not set to False anymore.
• #5013: pytest now depends on wcwidth to properly track unicode character sizes for more precise terminal output.
• #5059: pytester’s Testdir.popen() uses stdout and stderr via keyword arguments with defaults now (subprocess.PIPE).
• #5069: The code for the short test summary in the terminal was moved to the terminal plugin.
• #5082: Improved validation of kwargs for various methods in the pytester plugin.
• #5202: record_property now emits a PytestWarning when used with junit_family=xunit2: the fixture generates property tags as children of testcase, which is not permitted according to the most recent schema.
• #5239: Pin pluggy to < 1.0 so we don’t update to 1.0 automatically when it gets released: there are planned breaking changes, and we want to ensure pytest properly supports pluggy 1.0.

pytest 4.4.2 (2019-05-08)

Bug Fixes

• #5089: Fix crash caused by error in __repr__ function with both showlocals and verbose output enabled.
• #5139: Eliminate core dependency on ‘terminal’ plugin.
• #5229: Require pluggy>=0.11.0 which reverts a dependency to importlib-metadata added in 0.10.0. The importlib-metadata package cannot be imported when installed as an egg and causes issues when relying on setup.py to install test dependencies.

Improved Documentation

• #5171: Doc: pytest_ignore_collect, pytest_collect_directory, pytest_collect_file and pytest_pycollect_makemodule hooks’s ‘path’ parameter documented type is now py.path.local
• #5188: Improve help for --runxfail flag.

Trivial/Internal Changes

• #5182: Removed internal and unused _pytest.deprecated.MARK_INFO_ATTRIBUTE.

pytest 4.4.1 (2019-04-15)

Bug Fixes

• #5031: Environment variables are properly restored when using pytester’s testdir fixture.
• #5039: Fix regression with --pdbcls, which stopped working with local modules in 4.0.0.
• #5092: Produce a warning when unknown keywords are passed to pytest.param(...).
• #5098: Invalidate import caches with monkeypatch.syspath_prepend, which is required with namespace packages being used.

pytest 4.4.0 (2019-03-29)

Features

• #2224: async test functions are skipped and a warning is emitted when a suitable async plugin is not installed (such as pytest-asyncio or pytest-trio).

  Previously async functions would not execute at all but still be marked as “passed”.

• #2482: Include new disable_test_id_escaping_and_forfeit_all_rights_to_community_support option to disable ascii-escaping in parametrized values. This may cause a series of problems and as the name makes clear, use at your own risk.

• #4718: The -p option can now be used to early-load plugins also by entry-point name, instead of just by module name.

  This makes it possible to early load external plugins like pytest-cov in the command-line:

  pytest -p pytest_cov
  

• #4855: The --pdbcls option handles classes via module attributes now (e.g. pdb:pdb.Pdb with pdb++), and its validation was improved.

• #4875: The testpaths configuration option is now displayed next to the rootdir and inifile lines in the pytest header if the option is in effect, i.e., directories or file names were not explicitly passed in the command line.

  Also, inifile is only displayed if there’s a configuration file, instead of an empty inifile: string.

• #4911: Doctests can be skipped now dynamically using pytest.skip().

• #4920: Internal refactorings have been made in order to make the implementation of the pytest-subtests plugin possible, which adds unittest sub-test support and a new subtests fixture as discussed in #1367.

  For details on the internal refactorings, please see the details on the related PR.

• #4931: pytester’s LineMatcher asserts that the passed lines are a sequence.

• #4936: Handle -p plug after -p no:plug.

  This can be used to override a blocked plugin (e.g. in “addopts”) from the command line etc.

• #4951: Output capturing is handled correctly when only capturing via fixtures (capsys, capfs) with pdb.set_trace().

• #4956: pytester sets $HOME and $USERPROFILE to the temporary directory during test runs.

  This ensures to not load configuration files from the real user’s home directory.

• #4980: Namespace packages are handled better with monkeypatch.syspath_prepend and testdir.syspathinsert (via pkg_resources.fixup_namespace_packages).

• #4993: The stepwise plugin reports status information now.

• #5008: If a setup.cfg file contains [tool:pytest] and also the no longer supported [pytest] section, pytest will use [tool:pytest] ignoring [pytest]. Previously it would unconditionally error out.

  This makes it simpler for plugins to support old pytest versions.

Bug Fixes

• #1895: Fix bug where fixtures requested dynamically via request.getfixturevalue() might be teardown before the requesting fixture.

• #4851: pytester unsets PYTEST_ADDOPTS now to not use outer options with testdir.runpytest().

• #4903: Use the correct modified time for years after 2038 in rewritten .pyc files.

• #4928: Fix line offsets with ScopeMismatch errors.

• #4957: -p no:plugin is handled correctly for default (internal) plugins now, e.g. with -p no:capture.

  Previously they were loaded (imported) always, making e.g. the capfd fixture available.

• #4968: The pdb quit command is handled properly when used after the debug command with pdb++.

• #4975: Fix the interpretation of -qq option where it was being considered as -v instead.

• #4978: outcomes.Exit is not swallowed in assertrepr_compare anymore.

• #4988: Close logging’s file handler explicitly when the session finishes.

• #5003: Fix line offset with mark collection error (off by one).

Improved Documentation

• #4974: Update docs for pytest_cmdline_parse hook to note availability liminations

Trivial/Internal Changes

• #4718: pluggy>=0.9 is now required.

• #4815: funcsigs>=1.0 is now required for Python 2.7.

• #4829: Some left-over internal code related to yield tests has been removed.

• #4890: Remove internally unused anypython fixture from the pytester plugin.

• #4912: Remove deprecated Sphinx directive, add_description_unit(), pin sphinx-removed-in to >= 0.2.0 to support Sphinx 2.0.

• #4913: Fix pytest tests invocation with custom PYTHONPATH.

• #4965: New pytest_report_to_serializable and pytest_report_from_serializable experimental hooks.

  These hooks will be used by pytest-xdist, pytest-subtests, and the replacement for resultlog to serialize and customize reports.

  They are experimental, meaning that their details might change or even be removed completely in future patch releases without warning.

  Feedback is welcome from plugin authors and users alike.

• #4987: Collector.repr_failure respects the --tb option, but only defaults to short now (with auto).

pytest 4.3.1 (2019-03-11)

Bug Fixes

• #4810: Logging messages inside pytest_runtest_logreport() are now properly captured and displayed.
• #4861: Improve validation of contents written to captured output so it behaves the same as when capture is disabled.
• #4898: Fix AttributeError: FixtureRequest has no 'confg' attribute bug in testdir.copy_example.

Trivial/Internal Changes

• #4768: Avoid pkg_resources import at the top-level.

pytest 4.3.0 (2019-02-16)

Deprecations

• #4724: pytest.warns() now emits a warning when it receives unknown keyword arguments.

  This will be changed into an error in the future.

Features

• #2753: Usage errors from argparse are mapped to pytest’s UsageError.

• #3711: Add the --ignore-glob parameter to exclude test-modules with Unix shell-style wildcards. Add the collect_ignore_glob for conftest.py to exclude test-modules with Unix shell-style wildcards.

• #4698: The warning about Python 2.7 and 3.4 not being supported in pytest 5.0 has been removed.

  In the end it was considered to be more of a nuisance than actual utility and users of those Python versions shouldn’t have problems as pip will not install pytest 5.0 on those interpreters.

• #4707: With the help of new set_log_path() method there is a way to set log_file paths from hooks.

Bug Fixes

• #4651: --help and --version are handled with UsageError.
• #4782: Fix AssertionError with collection of broken symlinks with packages.

pytest 4.2.1 (2019-02-12)

Bug Fixes

• #2895: The pytest_report_collectionfinish hook now is also called with --collect-only.
• #3899: Do not raise UsageError when an imported package has a pytest_plugins.py child module.
• #4347: Fix output capturing when using pdb++ with recursive debugging.
• #4592: Fix handling of collect_ignore via parent conftest.py.
• #4700: Fix regression where setUpClass would always be called in subclasses even if all tests were skipped by a unittest.skip() decorator applied in the subclass.
• #4739: Fix parametrize(... ids=<function>) when the function returns non-strings.
• #4745: Fix/improve collection of args when passing in __init__.py and a test file.
• #4770: more_itertools is now constrained to <6.0.0 when required for Python 2.7 compatibility.
• #526: Fix “ValueError: Plugin already registered” exceptions when running in build directories that symlink to actual source.

Improved Documentation

• #3899: Add note to plugins.rst that pytest_plugins should not be used as a name for a user module containing plugins.
• #4324: Document how to use raises and does_not_raise to write parametrized tests with conditional raises.
• #4709: Document how to customize test failure messages when using pytest.warns.

Trivial/Internal Changes

• #4741: Some verbosity related attributes of the TerminalReporter plugin are now read only properties.

pytest 4.2.0 (2019-01-30)

Features

• #3094: Classic xunit-style functions and methods now obey the scope of autouse fixtures.

  This fixes a number of surprising issues like setup_method being called before session-scoped autouse fixtures (see #517 for an example).

• #4627: Display a message at the end of the test session when running under Python 2.7 and 3.4 that pytest 5.0 will no longer support those Python versions.

• #4660: The number of selected tests now are also displayed when the -k or -m flags are used.

• #4688: pytest_report_teststatus hook now can also receive a config parameter.

• #4691: pytest_terminal_summary hook now can also receive a config parameter.

Bug Fixes

• #3547: --junitxml can emit XML compatible with Jenkins xUnit. junit_family INI option accepts legacy|xunit1, which produces old style output, and xunit2 that conforms more strictly to https://github.com/jenkinsci/xunit-plugin/blob/xunit-2.3.2/src/main/resources/org/jenkinsci/plugins/xunit/types/model/xsd/junit-10.xsd

• #4280: Improve quitting from pdb, especially with --trace.

  Using q[quit] after pdb.set_trace() will quit pytest also.

• #4402: Warning summary now groups warnings by message instead of by test id.

  This makes the output more compact and better conveys the general idea of how much code is actually generating warnings, instead of how many tests call that code.

• #4536: monkeypatch.delattr handles class descriptors like staticmethod/classmethod.

• #4649: Restore marks being considered keywords for keyword expressions.

• #4653: tmp_path fixture and other related ones provides resolved path (a.k.a real path)

• #4667: pytest_terminal_summary uses result from pytest_report_teststatus hook, rather than hardcoded strings.

• #4669: Correctly handle unittest.SkipTest exception containing non-ascii characters on Python 2.

• #4680: Ensure the tmpdir and the tmp_path fixtures are the same folder.

• #4681: Ensure tmp_path is always a real path.

Trivial/Internal Changes

• #4643: Use a.item() instead of the deprecated np.asscalar(a) in pytest.approx.

  np.asscalar has been deprecated in numpy 1.16..

• #4657: Copy saferepr from pylib

• #4668: The verbose word for expected failures in the teststatus report changes from xfail to XFAIL to be consistent with other test outcomes.

pytest 4.1.1 (2019-01-12)

Bug Fixes

• #2256: Show full repr with assert a==b and -vv.
• #3456: Extend Doctest-modules to ignore mock objects.
• #4617: Fixed pytest.warns bug when context manager is reused (e.g. multiple parametrization).
• #4631: Don’t rewrite assertion when __getattr__ is broken

Improved Documentation

• #3375: Document that using setup.cfg may crash other tools or cause hard to track down problems because it uses a different parser than pytest.ini or tox.ini files.

Trivial/Internal Changes

• #4602: Uninstall hypothesis in regen tox env.

pytest 4.1.0 (2019-01-05)

Removals

• #2169: pytest.mark.parametrize: in previous versions, errors raised by id functions were suppressed and changed into warnings. Now the exceptions are propagated, along with a pytest message informing the node, parameter value and index where the exception occurred.

• #3078: Remove legacy internal warnings system: config.warn, Node.warn. The pytest_logwarning now issues a warning when implemented.

  See our docs on information on how to update your code.

• #3079: Removed support for yield tests - they are fundamentally broken because they don’t support fixtures properly since collection and test execution were separated.

  See our docs on information on how to update your code.

• #3082: Removed support for applying marks directly to values in @pytest.mark.parametrize. Use pytest.param instead.

  See our docs on information on how to update your code.

• #3083: Removed Metafunc.addcall. This was the predecessor mechanism to @pytest.mark.parametrize.

  See our docs on information on how to update your code.

• #3085: Removed support for passing strings to pytest.main. Now, always pass a list of strings instead.

  See our docs on information on how to update your code.

• #3086: [pytest] section in setup.cfg files is no longer supported, use [tool:pytest] instead. setup.cfg files are meant for use with distutils, and a section named pytest has notoriously been a source of conflicts and bugs.

  Note that for pytest.ini and tox.ini files the section remains [pytest].

• #3616: Removed the deprecated compat properties for node.Class/Function/Module - use pytest.Class/Function/Module now.

  See our docs on information on how to update your code.

• #4421: Removed the implementation of the pytest_namespace hook.

  See our docs on information on how to update your code.

• #4489: Removed request.cached_setup. This was the predecessor mechanism to modern fixtures.

  See our docs on information on how to update your code.

• #4535: Removed the deprecated PyCollector.makeitem method. This method was made public by mistake a long time ago.

• #4543: Removed support to define fixtures using the pytest_funcarg__ prefix. Use the @pytest.fixture decorator instead.

  See our docs on information on how to update your code.

• #4545: Calling fixtures directly is now always an error instead of a warning.

  See our docs on information on how to update your code.

• #4546: Remove Node.get_marker(name) the return value was not usable for more than a existence check.

  Use Node.get_closest_marker(name) as a replacement.

• #4547: The deprecated record_xml_property fixture has been removed, use the more generic record_property instead.

  See our docs for more information.

• #4548: An error is now raised if the pytest_plugins variable is defined in a non-top-level conftest.py file (i.e., not residing in the rootdir).

  See our docs for more information.

• #891: Remove testfunction.markername attributes - use Node.iter_markers(name=None) to iterate them.

Deprecations

• #3050: Deprecated the pytest.config global.

  See https://docs.pytest.org/en/latest/deprecations.html#pytest-config-global for rationale.

• #3974: Passing the message parameter of pytest.raises now issues a DeprecationWarning.

  It is a common mistake to think this parameter will match the exception message, while in fact it only serves to provide a custom message in case the pytest.raises check fails. To avoid this mistake and because it is believed to be little used, pytest is deprecating it without providing an alternative for the moment.

  If you have concerns about this, please comment on issue #3974.

• #4435: Deprecated raises(..., 'code(as_a_string)') and warns(..., 'code(as_a_string)').

  See https://docs.pytest.org/en/latest/deprecations.html#raises-warns-exec for rationale and examples.

Features

• #3191: A warning is now issued when assertions are made for None.

  This is a common source of confusion among new users, which write:

  assert mocked_object.assert_called_with(3, 4, 5, key="value")
  

  When they should write:

  mocked_object.assert_called_with(3, 4, 5, key="value")
  

  Because the assert_called_with method of mock objects already executes an assertion.

  This warning will not be issued when None is explicitly checked. An assertion like:

  assert variable is None
  

  will not issue the warning.

• #3632: Richer equality comparison introspection on AssertionError for objects created using attrs or dataclasses (Python 3.7+, backported to 3.6).

• #4278: CACHEDIR.TAG files are now created inside cache directories.

  Those files are part of the Cache Directory Tagging Standard, and can be used by backup or synchronization programs to identify pytest’s cache directory as such.

• #4292: pytest.outcomes.Exit is derived from SystemExit instead of KeyboardInterrupt. This allows us to better handle pdb exiting.

• #4371: Updated the --collect-only option to display test descriptions when ran using --verbose.

• #4386: Restructured ExceptionInfo object construction and ensure incomplete instances have a repr/str.

• #4416: pdb: added support for keyword arguments with pdb.set_trace.

  It handles header similar to Python 3.7 does it, and forwards any other keyword arguments to the Pdb constructor.

  This allows for __import__("pdb").set_trace(skip=["foo.*"]).

• #4483: Added ini parameter junit_duration_report to optionally report test call durations, excluding setup and teardown times.

  The JUnit XML specification and the default pytest behavior is to include setup and teardown times in the test duration report. You can include just the call durations instead (excluding setup and teardown) by adding this to your pytest.ini file:

  [pytest]
  junit_duration_report = call
  

• #4532: -ra now will show errors and failures last, instead of as the first items in the summary.

  This makes it easier to obtain a list of errors and failures to run tests selectively.

• #4599: pytest.importorskip now supports a reason parameter, which will be shown when the requested module cannot be imported.

Bug Fixes

• #3532: -p now accepts its argument without a space between the value, for example -pmyplugin.
• #4327: approx again works with more generic containers, more precisely instances of Iterable and Sized instead of more restrictive Sequence.
• #4397: Ensure that node ids are printable.
• #4435: Fixed raises(..., 'code(string)') frame filename.
• #4458: Display actual test ids in --collect-only.

Improved Documentation

• #4557: Markers example documentation page updated to support latest pytest version.
• #4558: Update cache documentation example to correctly show cache hit and miss.
• #4580: Improved detailed summary report documentation.

Trivial/Internal Changes

• #4447: Changed the deprecation type of --result-log to PytestDeprecationWarning.

  It was decided to remove this feature at the next major revision.

pytest 4.0.2 (2018-12-13)

Bug Fixes

• #4265: Validate arguments from the PYTEST_ADDOPTS environment variable and the addopts ini option separately.
• #4435: Fix raises(..., 'code(string)') frame filename.
• #4500: When a fixture yields and a log call is made after the test runs, and, if the test is interrupted, capture attributes are None.
• #4538: Raise TypeError for with raises(..., match=<non-None falsey value>).

Improved Documentation

• #1495: Document common doctest fixture directory tree structure pitfalls

pytest 4.0.1 (2018-11-23)

Bug Fixes

• #3952: Display warnings before “short test summary info” again, but still later warnings in the end.
• #4386: Handle uninitialized exceptioninfo in repr/str.
• #4393: Do not create .gitignore/README.md files in existing cache directories.
• #4400: Rearrange warning handling for the yield test errors so the opt-out in 4.0.x correctly works.
• #4405: Fix collection of testpaths with --pyargs.
• #4412: Fix assertion rewriting involving Starred + side-effects.
• #4425: Ensure we resolve the absolute path when the given --basetemp is a relative path.

Trivial/Internal Changes

• #4315: Use pkg_resources.parse_version instead of LooseVersion in minversion check.
• #4440: Adjust the stack level of some internal pytest warnings.

pytest 4.0.0 (2018-11-13)

Removals

• #3737: RemovedInPytest4Warnings are now errors by default.

  Following our plan to remove deprecated features with as little disruption as possible, all warnings of type RemovedInPytest4Warnings now generate errors instead of warning messages.

  The affected features will be effectively removed in pytest 4.1, so please consult the Deprecations and Removals section in the docs for directions on how to update existing code.

  In the pytest 4.0.X series, it is possible to change the errors back into warnings as a stop gap measure by adding this to your pytest.ini file:

  [pytest]
  filterwarnings =
      ignore::pytest.RemovedInPytest4Warning
  

  But this will stop working when pytest 4.1 is released.

  If you have concerns about the removal of a specific feature, please add a comment to #4348.

• #4358: Remove the ::() notation to denote a test class instance in node ids.

  Previously, node ids that contain test instances would use ::() to denote the instance like this:

  test_foo.py::Test::()::test_bar
  

  The extra ::() was puzzling to most users and has been removed, so that the test id becomes now:

  test_foo.py::Test::test_bar
  

  This change could not accompany a deprecation period as is usual when user-facing functionality changes because it was not really possible to detect when the functionality was being used explicitly.

  The extra ::() might have been removed in some places internally already, which then led to confusion in places where it was expected, e.g. with --deselect (#4127).

  Test class instances are also not listed with --collect-only anymore.

Features

• #4270: The cache_dir option uses $TOX_ENV_DIR as prefix (if set in the environment).

  This uses a different cache per tox environment by default.

Bug Fixes

• #3554: Fix CallInfo.__repr__ for when the call is not finished yet.

pytest 3.10.1 (2018-11-11)

Bug Fixes

• #4287: Fix nested usage of debugging plugin (pdb), e.g. with pytester’s testdir.runpytest.
• #4304: Block the stepwise plugin if cacheprovider is also blocked, as one depends on the other.
• #4306: Parse minversion as an actual version and not as dot-separated strings.
• #4310: Fix duplicate collection due to multiple args matching the same packages.
• #4321: Fix item.nodeid with resolved symlinks.
• #4325: Fix collection of direct symlinked files, where the target does not match python_files.
• #4329: Fix TypeError in report_collect with _collect_report_last_write.

Trivial/Internal Changes

• #4305: Replace byte/unicode helpers in test_capture with python level syntax.

pytest 3.10.0 (2018-11-03)

Features

• #2619: Resume capturing output after continue with __import__("pdb").set_trace().

  This also adds a new pytest_leave_pdb hook, and passes in pdb to the existing pytest_enter_pdb hook.

• #4147: Add --sw, --stepwise as an alternative to --lf -x for stopping at the first failure, but starting the next test invocation from that test. See the documentation for more info.

• #4188: Make --color emit colorful dots when not running in verbose mode. Earlier, it would only colorize the test-by-test output if --verbose was also passed.

• #4225: Improve performance with collection reporting in non-quiet mode with terminals.

  The “collecting …” message is only printed/updated every 0.5s.

Bug Fixes

• #2701: Fix false RemovedInPytest4Warning: usage of Session... is deprecated, please use pytest warnings.
• #4046: Fix problems with running tests in package __init__.py files.
• #4260: Swallow warnings during anonymous compilation of source.
• #4262: Fix access denied error when deleting stale directories created by tmpdir / tmp_path.
• #611: Naming a fixture request will now raise a warning: the request fixture is internal and should not be overwritten as it will lead to internal errors.
• #4266: Handle (ignore) exceptions raised during collection, e.g. with Django’s LazySettings proxy class.

Improved Documentation

• #4255: Added missing documentation about the fact that module names passed to filter warnings are not regex-escaped.

Trivial/Internal Changes

• #4272: Display cachedir also in non-verbose mode if non-default.

• #4277: pdb: improve message about output capturing with set_trace.

  Do not display “IO-capturing turned off/on” when -s is used to avoid confusion.

• #4279: Improve message and stack level of warnings issued by monkeypatch.setenv when the value of the environment variable is not a str.

pytest 3.9.3 (2018-10-27)

Bug Fixes

• #4174: Fix “ValueError: Plugin already registered” with conftest plugins via symlink.
• #4181: Handle race condition between creation and deletion of temporary folders.
• #4221: Fix bug where the warning summary at the end of the test session was not showing the test where the warning was originated.
• #4243: Fix regression when stacklevel for warnings was passed as positional argument on python2.

Improved Documentation

• #3851: Add reference to empty_parameter_set_mark ini option in documentation of @pytest.mark.parametrize

Trivial/Internal Changes

• #4028: Revert patching of sys.breakpointhook since it appears to do nothing.
• #4233: Apply an import sorter (reorder-python-imports) to the codebase.
• #4248: Remove use of unnecessary compat shim, six.binary_type

pytest 3.9.2 (2018-10-22)

Bug Fixes

• #2909: Improve error message when a recursive dependency between fixtures is detected.
• #3340: Fix logging messages not shown in hooks pytest_sessionstart() and pytest_sessionfinish().
• #3533: Fix unescaped XML raw objects in JUnit report for skipped tests
• #3691: Python 2: safely format warning message about passing unicode strings to warnings.warn, which may cause surprising MemoryError exception when monkey patching warnings.warn itself.
• #4026: Improve error message when it is not possible to determine a function’s signature.
• #4177: Pin setuptools>=40.0 to support py_modules in setup.cfg
• #4179: Restore the tmpdir behaviour of symlinking the current test run.
• #4192: Fix filename reported by warnings.warn when using recwarn under python2.

pytest 3.9.1 (2018-10-16)

Features

• #4159: For test-suites containing test classes, the information about the subclassed module is now output only if a higher verbosity level is specified (at least “-vv”).

pytest 3.9.0 (2018-10-15 - not published due to a release automation bug)

Deprecations

• #3616: The following accesses have been documented as deprecated for years, but are now actually emitting deprecation warnings.

  • Access of Module, Function, Class, Instance, File and Item through Node instances. Now users will this warning:

    usage of Function.Module is deprecated, please use pytest.Module instead
    

    Users should just import pytest and access those objects using the pytest module.

  • request.cached_setup, this was the precursor of the setup/teardown mechanism available to fixtures. You can consult funcarg comparison section in the docs.

  • Using objects named "Class" as a way to customize the type of nodes that are collected in Collector subclasses has been deprecated. Users instead should use pytest_collect_make_item to customize node types during collection.

    This issue should affect only advanced plugins who create new collection types, so if you see this warning message please contact the authors so they can change the code.

  • The warning that produces the message below has changed to RemovedInPytest4Warning:

    getfuncargvalue is deprecated, use getfixturevalue
    

• #3988: Add a Deprecation warning for pytest.ensuretemp as it was deprecated since a while.

Features

• #2293: Improve usage errors messages by hiding internal details which can be distracting and noisy.

  This has the side effect that some error conditions that previously raised generic errors (such as ValueError for unregistered marks) are now raising Failed exceptions.

• #3332: Improve the error displayed when a conftest.py file could not be imported.

  In order to implement this, a new chain parameter was added to ExceptionInfo.getrepr to show or hide chained tracebacks in Python 3 (defaults to True).

• #3849: Add empty_parameter_set_mark=fail_at_collect ini option for raising an exception when parametrize collects an empty set.

• #3964: Log messages generated in the collection phase are shown when live-logging is enabled and/or when they are logged to a file.

• #3985: Introduce tmp_path as a fixture providing a Path object. Also introduce tmp_path_factory as a session-scoped fixture for creating arbitrary temporary directories from any other fixture or test.

• #4013: Deprecation warnings are now shown even if you customize the warnings filters yourself. In the previous version any customization would override pytest’s filters and deprecation warnings would fall back to being hidden by default.

• #4073: Allow specification of timeout for Testdir.runpytest_subprocess() and Testdir.run().

• #4098: Add returncode argument to pytest.exit() to exit pytest with a specific return code.

• #4102: Reimplement pytest.deprecated_call using pytest.warns so it supports the match='...' keyword argument.

  This has the side effect that pytest.deprecated_call now raises pytest.fail.Exception instead of AssertionError.

• #4149: Require setuptools>=30.3 and move most of the metadata to setup.cfg.

Bug Fixes

• #2535: Improve error message when test functions of unittest.TestCase subclasses use a parametrized fixture.

• #3057: request.fixturenames now correctly returns the name of fixtures created by request.getfixturevalue().

• #3946: Warning filters passed as command line options using -W now take precedence over filters defined in ini configuration files.

• #4066: Fix source reindenting by using textwrap.dedent directly.

• #4102: pytest.warn will capture previously-warned warnings in Python 2. Previously they were never raised.

• #4108: Resolve symbolic links for args.

  This fixes running pytest tests/test_foo.py::test_bar, where tests is a symlink to project/app/tests: previously project/app/conftest.py would be ignored for fixtures then.

• #4132: Fix duplicate printing of internal errors when using --pdb.

• #4135: pathlib based tmpdir cleanup now correctly handles symlinks in the folder.

• #4152: Display the filename when encountering SyntaxWarning.

Improved Documentation

• #3713: Update usefixtures documentation to clarify that it can’t be used with fixture functions.
• #4058: Update fixture documentation to specify that a fixture can be invoked twice in the scope it’s defined for.
• #4064: According to unittest.rst, setUpModule and tearDownModule were not implemented, but it turns out they are. So updated the documentation for unittest.
• #4151: Add tempir testing example to CONTRIBUTING.rst guide

Trivial/Internal Changes

• #2293: The internal MarkerError exception has been removed.
• #3988: Port the implementation of tmpdir to pathlib.
• #4063: Exclude 0.00 second entries from --duration output unless -vv is passed on the command-line.
• #4093: Fixed formatting of string literals in internal tests.

pytest 3.8.2 (2018-10-02)

Deprecations and Removals

• #4036: The item parameter of pytest_warning_captured hook is now documented as deprecated. We realized only after the 3.8 release that this parameter is incompatible with pytest-xdist.

  Our policy is to not deprecate features during bugfix releases, but in this case we believe it makes sense as we are only documenting it as deprecated, without issuing warnings which might potentially break test suites. This will get the word out that hook implementers should not use this parameter at all.

  In a future release item will always be None and will emit a proper warning when a hook implementation makes use of it.

Bug Fixes

• #3539: Fix reload on assertion rewritten modules.

• #4034: The .user_properties attribute of TestReport objects is a list of (name, value) tuples, but could sometimes be instantiated as a tuple of tuples. It is now always a list.

• #4039: No longer issue warnings about using pytest_plugins in non-top-level directories when using --pyargs: the current --pyargs mechanism is not reliable and might give false negatives.

• #4040: Exclude empty reports for passed tests when -rP option is used.

• #4051: Improve error message when an invalid Python expression is passed to the -m option.

• #4056: MonkeyPatch.setenv and MonkeyPatch.delenv issue a warning if the environment variable name is not str on Python 2.

  In Python 2, adding unicode keys to os.environ causes problems with subprocess (and possible other modules), making this a subtle bug specially susceptible when used with from __future__ import unicode_literals.

Improved Documentation

• #3928: Add possible values for fixture scope to docs.

pytest 3.8.1 (2018-09-22)

Bug Fixes

• #3286: .pytest_cache directory is now automatically ignored by Git. Users who would like to contribute a solution for other SCMs please consult/comment on this issue.

• #3749: Fix the following error during collection of tests inside packages:

  TypeError: object of type 'Package' has no len()
  

• #3941: Fix bug where indirect parametrization would consider the scope of all fixtures used by the test function to determine the parametrization scope, and not only the scope of the fixtures being parametrized.

• #3973: Fix crash of the assertion rewriter if a test changed the current working directory without restoring it afterwards.

• #3998: Fix issue that prevented some caplog properties (for example record_tuples) from being available when entering the debugger with --pdb.

• #3999: Fix UnicodeDecodeError in python2.x when a class returns a non-ascii binary __repr__ in an assertion which also contains non-ascii text.

Improved Documentation

• #3996: New Deprecations and Removals page shows all currently deprecated features, the rationale to do so, and alternatives to update your code. It also list features removed from pytest in past major releases to help those with ancient pytest versions to upgrade.

Trivial/Internal Changes

• #3955: Improve pre-commit detection for changelog filenames
• #3975: Remove legacy code around im_func as that was python2 only

pytest 3.8.0 (2018-09-05)

Deprecations and Removals

• #2452: Config.warn and Node.warn have been deprecated, see https://docs.pytest.org/en/latest/deprecations.html#config-warn-and-node-warn for rationale and examples.

• #3936: @pytest.mark.filterwarnings second parameter is no longer regex-escaped, making it possible to actually use regular expressions to check the warning message.

  Note: regex-escaping the match string was an implementation oversight that might break test suites which depend on the old behavior.

Features

• #2452: Internal pytest warnings are now issued using the standard warnings module, making it possible to use the standard warnings filters to manage those warnings. This introduces PytestWarning, PytestDeprecationWarning and RemovedInPytest4Warning warning types as part of the public API.

  Consult the documentation for more info.

• #2908: DeprecationWarning and PendingDeprecationWarning are now shown by default if no other warning filter is configured. This makes pytest more compliant with PEP-0506. See the docs for more info.

• #3251: Warnings are now captured and displayed during test collection.

• #3784: PYTEST_DISABLE_PLUGIN_AUTOLOAD environment variable disables plugin auto-loading when set.

• #3829: Added the count option to console_output_style to enable displaying the progress as a count instead of a percentage.

• #3837: Added support for ‘xfailed’ and ‘xpassed’ outcomes to the pytester.RunResult.assert_outcomes signature.

Bug Fixes

• #3911: Terminal writer now takes into account unicode character width when writing out progress.
• #3913: Pytest now returns with correct exit code (EXIT_USAGEERROR, 4) when called with unknown arguments.
• #3918: Improve performance of assertion rewriting.

Improved Documentation

• #3566: Added a blurb in usage.rst for the usage of -r flag which is used to show an extra test summary info.
• #3907: Corrected type of the exceptions collection passed to xfail: raises argument accepts a tuple instead of list.

Trivial/Internal Changes

• #3853: Removed "run all (no recorded failures)" message printed with --failed-first and --last-failed when there are no failed tests.

pytest 3.7.4 (2018-08-29)

Bug Fixes

• #3506: Fix possible infinite recursion when writing .pyc files.
• #3853: Cache plugin now obeys the -q flag when --last-failed and --failed-first flags are used.
• #3883: Fix bad console output when using console_output_style=classic.
• #3888: Fix macOS specific code using capturemanager plugin in doctests.

Improved Documentation

• #3902: Fix pytest.org links

pytest 3.7.3 (2018-08-26)

Bug Fixes

• #3033: Fixtures during teardown can again use capsys and capfd to inspect output captured during tests.
• #3773: Fix collection of tests from __init__.py files if they match the python_files configuration option.
• #3796: Fix issue where teardown of fixtures of consecutive sub-packages were executed once, at the end of the outer package.
• #3816: Fix bug where --show-capture=no option would still show logs printed during fixture teardown.
• #3819: Fix stdout/stderr not getting captured when real-time cli logging is active.
• #3843: Fix collection error when specifying test functions directly in the command line using test.py::test syntax together with --doctest-modules.
• #3848: Fix bugs where unicode arguments could not be passed to testdir.runpytest on Python 2.
• #3854: Fix double collection of tests within packages when the filename starts with a capital letter.

Improved Documentation

• #3824: Added example for multiple glob pattern matches in python_files.
• #3833: Added missing docs for pytester.Testdir.
• #3870: Correct documentation for setuptools integration.

Trivial/Internal Changes

• #3826: Replace broken type annotations with type comments.
• #3845: Remove a reference to issue #568 from the documentation, which has since been fixed.

pytest 3.7.2 (2018-08-16)

Bug Fixes

• #3671: Fix filterwarnings not being registered as a builtin mark.
• #3768, #3789: Fix test collection from packages mixed with normal directories.
• #3771: Fix infinite recursion during collection if a pytest_ignore_collect hook returns False instead of None.
• #3774: Fix bug where decorated fixtures would lose functionality (for example @mock.patch).
• #3775: Fix bug where importing modules or other objects with prefix pytest_ prefix would raise a PluginValidationError.
• #3788: Fix AttributeError during teardown of TestCase subclasses which raise an exception during __init__.
• #3804: Fix traceback reporting for exceptions with __cause__ cycles.

Improved Documentation

• #3746: Add documentation for metafunc.config that had been mistakenly hidden.

pytest 3.7.1 (2018-08-02)

Bug Fixes

• #3473: Raise immediately if approx() is given an expected value of a type it doesn’t understand (e.g. strings, nested dicts, etc.).
• #3712: Correctly represent the dimensions of a numpy array when calling repr() on approx().
• #3742: Fix incompatibility with third party plugins during collection, which produced the error object has no attribute '_collectfile'.
• #3745: Display the absolute path if cache_dir is not relative to the rootdir instead of failing.
• #3747: Fix compatibility problem with plugins and the warning code issued by fixture functions when they are called directly.
• #3748: Fix infinite recursion in pytest.approx with arrays in numpy<1.13.
• #3757: Pin pathlib2 to >=2.2.0 as we require __fspath__ support.
• #3763: Fix TypeError when the assertion message is bytes in python 3.

pytest 3.7.0 (2018-07-30)

Deprecations and Removals

• #2639: pytest_namespace has been deprecated.
• #3661: Calling a fixture function directly, as opposed to request them in a test function, now issues a RemovedInPytest4Warning. See the documentation for rationale and examples.

Features

• #2283: New package fixture scope: fixtures are finalized when the last test of a package finishes. This feature is considered experimental, so use it sparingly.
• #3576: Node.add_marker now supports an append=True/False parameter to determine whether the mark comes last (default) or first.
• #3579: Fixture caplog now has a messages property, providing convenient access to the format-interpolated log messages without the extra data provided by the formatter/handler.
• #3610: New --trace option to enter the debugger at the start of a test.
• #3623: Introduce pytester.copy_example as helper to do acceptance tests against examples from the project.

Bug Fixes

• #2220: Fix a bug where fixtures overridden by direct parameters (for example parametrization) were being instantiated even if they were not being used by a test.
• #3695: Fix ApproxNumpy initialisation argument mixup, abs and rel tolerances were flipped causing strange comparison results. Add tests to check abs and rel tolerances for np.array and test for expecting nan with np.array()
• #980: Fix truncated locals output in verbose mode.

Improved Documentation

• #3295: Correct the usage documentation of --last-failed-no-failures by adding the missing --last-failed argument in the presented examples, because they are misleading and lead to think that the missing argument is not needed.

Trivial/Internal Changes

• #3519: Now a README.md file is created in .pytest_cache to make it clear why the directory exists.

pytest 3.6.4 (2018-07-28)

Bug Fixes

• Invoke pytest using -mpytest so sys.path does not get polluted by packages installed in site-packages. (#742)

Improved Documentation

• Use smtp_connection instead of smtp in fixtures documentation to avoid possible confusion. (#3592)

Trivial/Internal Changes

• Remove obsolete __future__ imports. (#2319)
• Add CITATION to provide information on how to formally cite pytest. (#3402)
• Replace broken type annotations with type comments. (#3635)
• Pin pluggy to <0.8. (#3727)

pytest 3.6.3 (2018-07-04)

Bug Fixes

• Fix ImportWarning triggered by explicit relative imports in assertion-rewritten package modules. (#3061)
• Fix error in pytest.approx when dealing with 0-dimension numpy arrays. (#3593)
• No longer raise ValueError when using the get_marker API. (#3605)
• Fix problem where log messages with non-ascii characters would not appear in the output log file. (#3630)
• No longer raise AttributeError when legacy marks can’t be stored in functions. (#3631)

Improved Documentation

• The description above the example for @pytest.mark.skipif now better matches the code. (#3611)

Trivial/Internal Changes

• Internal refactoring: removed unused CallSpec2tox ._globalid_args attribute and metafunc parameter from CallSpec2.copy(). (#3598)
• Silence usage of reduce warning in Python 2 (#3609)
• Fix usage of attr.ib deprecated convert parameter. (#3653)

pytest 3.6.2 (2018-06-20)

Bug Fixes

• Fix regression in Node.add_marker by extracting the mark object of a MarkDecorator. (#3555)
• Warnings without location were reported as None. This is corrected to now report <undetermined location>. (#3563)
• Continue to call finalizers in the stack when a finalizer in a former scope raises an exception. (#3569)
• Fix encoding error with print statements in doctests (#3583)

Improved Documentation

• Add documentation for the --strict flag. (#3549)

Trivial/Internal Changes

• Update old quotation style to parens in fixture.rst documentation. (#3525)
• Improve display of hint about --fulltrace with KeyboardInterrupt. (#3545)
• pytest’s testsuite is no longer runnable through python setup.py test – instead invoke pytest or tox directly. (#3552)
• Fix typo in documentation (#3567)

pytest 3.6.1 (2018-06-05)

Bug Fixes

• Fixed a bug where stdout and stderr were logged twice by junitxml when a test was marked xfail. (#3491)
• Fix usefixtures mark applyed to unittest tests by correctly instantiating FixtureInfo. (#3498)
• Fix assertion rewriter compatibility with libraries that monkey patch file objects. (#3503)

Improved Documentation

• Added a section on how to use fixtures as factories to the fixture documentation. (#3461)

Trivial/Internal Changes

• Enable caching for pip/pre-commit in order to reduce build time on travis/appveyor. (#3502)
• Switch pytest to the src/ layout as we already suggested it for good practice - now we implement it as well. (#3513)
• Fix if in tests to support 3.7.0b5, where a docstring handling in AST got reverted. (#3530)
• Remove some python2.5 compatibility code. (#3529)

pytest 3.6.0 (2018-05-23)

Features

• Revamp the internals of the pytest.mark implementation with correct per node handling which fixes a number of long standing bugs caused by the old design. This introduces new Node.iter_markers(name) and Node.get_closest_marker(name) APIs. Users are strongly encouraged to read the reasons for the revamp in the docs, or jump over to details about updating existing code to use the new APIs. (#3317)
• Now when @pytest.fixture is applied more than once to the same function a ValueError is raised. This buggy behavior would cause surprising problems and if was working for a test suite it was mostly by accident. (#2334)
• Support for Python 3.7’s builtin breakpoint() method, see Using the builtin breakpoint function for details. (#3180)
• monkeypatch now supports a context() function which acts as a context manager which undoes all patching done within the with block. (#3290)
• The --pdb option now causes KeyboardInterrupt to enter the debugger, instead of stopping the test session. On python 2.7, hitting CTRL+C again exits the debugger. On python 3.2 and higher, use CTRL+D. (#3299)
• pytest no longer changes the log level of the root logger when the log-level parameter has greater numeric value than that of the level of the root logger, which makes it play better with custom logging configuration in user code. (#3307)

Bug Fixes

• A rare race-condition which might result in corrupted .pyc files on Windows has been hopefully solved. (#3008)
• Also use iter_marker for discovering the marks applying for marker expressions from the cli to avoid the bad data from the legacy mark storage. (#3441)
• When showing diffs of failed assertions where the contents contain only whitespace, escape them using repr() first to make it easy to spot the differences. (#3443)

Improved Documentation

• Change documentation copyright year to a range which auto-updates itself each time it is published. (#3303)

Trivial/Internal Changes

• pytest now depends on the python-atomicwrites library. (#3008)
• Update all pypi.python.org URLs to pypi.org. (#3431)
• Detect pytest_ prefixed hooks using the internal plugin manager since pluggy is deprecating the implprefix argument to PluginManager. (#3487)
• Import Mapping and Sequence from _pytest.compat instead of directly from collections in python_api.py::approx. Add Mapping to _pytest.compat, import it from collections on python 2, but from collections.abc on Python 3 to avoid a DeprecationWarning on Python 3.7 or newer. (#3497)

pytest 3.5.1 (2018-04-23)

Bug Fixes

• Reset sys.last_type, sys.last_value and sys.last_traceback before each test executes. Those attributes are added by pytest during the test run to aid debugging, but were never reset so they would create a leaking reference to the last failing test’s frame which in turn could never be reclaimed by the garbage collector. (#2798)
• pytest.raises now raises TypeError when receiving an unknown keyword argument. (#3348)
• pytest.raises now works with exception classes that look like iterables. (#3372)

Improved Documentation

• Fix typo in caplog fixture documentation, which incorrectly identified certain attributes as methods. (#3406)

Trivial/Internal Changes

• Added a more indicative error message when parametrizing a function whose argument takes a default value. (#3221)
• Remove internal _pytest.terminal.flatten function in favor of more_itertools.collapse. (#3330)
• Import some modules from collections.abc instead of collections as the former modules trigger DeprecationWarning in Python 3.7. (#3339)
• record_property is no longer experimental, removing the warnings was forgotten. (#3360)
• Mention in documentation and CLI help that fixtures with leading _ are printed by pytest --fixtures only if the -v option is added. (#3398)

pytest 3.5.0 (2018-03-21)

Deprecations and Removals

• record_xml_property fixture is now deprecated in favor of the more generic record_property. (#2770)
• Defining pytest_plugins is now deprecated in non-top-level conftest.py files, because they “leak” to the entire directory tree. See the docs for the rationale behind this decision (#3084)

Features

• New --show-capture command-line option that allows to specify how to display captured output when tests fail: no, stdout, stderr, log or all (the default). (#1478)
• New --rootdir command-line option to override the rules for discovering the root directory. See customize in the documentation for details. (#1642)
• Fixtures are now instantiated based on their scopes, with higher-scoped fixtures (such as session) being instantiated first than lower-scoped fixtures (such as function). The relative order of fixtures of the same scope is kept unchanged, based in their declaration order and their dependencies. (#2405)
• record_xml_property renamed to record_property and is now compatible with xdist, markers and any reporter. record_xml_property name is now deprecated. (#2770)
• New --nf, --new-first options: run new tests first followed by the rest of the tests, in both cases tests are also sorted by the file modified time, with more recent files coming first. (#3034)
• New --last-failed-no-failures command-line option that allows to specify the behavior of the cache plugin’s `--last-failed feature when no tests failed in the last run (or no cache was found): none or all (the default). (#3139)
• New --doctest-continue-on-failure command-line option to enable doctests to show multiple failures for each snippet, instead of stopping at the first failure. (#3149)
• Captured log messages are added to the <system-out> tag in the generated junit xml file if the junit_logging ini option is set to system-out. If the value of this ini option is system-err, the logs are written to <system-err>. The default value for junit_logging is no, meaning captured logs are not written to the output file. (#3156)
• Allow the logging plugin to handle pytest_runtest_logstart and pytest_runtest_logfinish hooks when live logs are enabled. (#3189)
• Passing --log-cli-level in the command-line now automatically activates live logging. (#3190)
• Add command line option --deselect to allow deselection of individual tests at collection time. (#3198)
• Captured logs are printed before entering pdb. (#3204)
• Deselected item count is now shown before tests are run, e.g. collected X items / Y deselected. (#3213)
• The builtin module platform is now available for use in expressions in pytest.mark. (#3236)
• The short test summary info section now is displayed after tracebacks and warnings in the terminal. (#3255)
• New --verbosity flag to set verbosity level explicitly. (#3296)
• pytest.approx now accepts comparing a numpy array with a scalar. (#3312)

Bug Fixes

• Suppress IOError when closing the temporary file used for capturing streams in Python 2.7. (#2370)
• Fixed clear() method on caplog fixture which cleared records, but not the text property. (#3297)
• During test collection, when stdin is not allowed to be read, the DontReadFromStdin object still allow itself to be iterable and resolved to an iterator without crashing. (#3314)

Improved Documentation

• Added a reference page to the docs. (#1713)

Trivial/Internal Changes

• Change minimum requirement of attrs to 17.4.0. (#3228)
• Renamed example directories so all tests pass when ran from the base directory. (#3245)
• Internal mark.py module has been turned into a package. (#3250)
• pytest now depends on the more-itertools package. (#3265)
• Added warning when [pytest] section is used in a .cfg file passed with -c (#3268)
• nodeids can now be passed explicitly to FSCollector and Node constructors. (#3291)
• Internal refactoring of FormattedExcinfo to use attrs facilities and remove old support code for legacy Python versions. (#3292)
• Refactoring to unify how verbosity is handled internally. (#3296)
• Internal refactoring to better integrate with argparse. (#3304)
• Fix a python example when calling a fixture in doc/en/usage.rst (#3308)

pytest 3.4.2 (2018-03-04)

Bug Fixes

• Removed progress information when capture option is no. (#3203)
• Refactor check of bindir from exists to isdir. (#3241)
• Fix TypeError issue when using approx with a Decimal value. (#3247)
• Fix reference cycle generated when using the request fixture. (#3249)
• [tool:pytest] sections in *.cfg files passed by the -c option are now properly recognized. (#3260)

Improved Documentation

• Add logging plugin to plugins list. (#3209)

Trivial/Internal Changes

• Fix minor typo in fixture.rst (#3259)

pytest 3.4.1 (2018-02-20)

Bug Fixes

• Move import of doctest.UnexpectedException to top-level to avoid possible errors when using --pdb. (#1810)
• Added printing of captured stdout/stderr before entering pdb, and improved a test which was giving false negatives about output capturing. (#3052)
• Fix ordering of tests using parametrized fixtures which can lead to fixtures being created more than necessary. (#3161)
• Fix bug where logging happening at hooks outside of “test run” hooks would cause an internal error. (#3184)
• Detect arguments injected by unittest.mock.patch decorator correctly when pypi mock.patch is installed and imported. (#3206)
• Errors shown when a pytest.raises() with match= fails are now cleaner on what happened: When no exception was raised, the “matching ‘…’” part got removed as it falsely implies that an exception was raised but it didn’t match. When a wrong exception was raised, it’s now thrown (like pytest.raised() without match= would) instead of complaining about the unmatched text. (#3222)
• Fixed output capture handling in doctests on macOS. (#985)

Improved Documentation

• Add Sphinx parameter docs for match and message args to pytest.raises. (#3202)

Trivial/Internal Changes

• pytest has changed the publication procedure and is now being published to PyPI directly from Travis. (#3060)
• Rename ParameterSet._for_parameterize() to _for_parametrize() in order to comply with the naming convention. (#3166)
• Skip failing pdb/doctest test on mac. (#985)

pytest 3.4.0 (2018-01-30)

Deprecations and Removals

• All pytest classes now subclass object for better Python 2/3 compatibility. This should not affect user code except in very rare edge cases. (#2147)

Features

• Introduce empty_parameter_set_mark ini option to select which mark to apply when @pytest.mark.parametrize is given an empty set of parameters. Valid options are skip (default) and xfail. Note that it is planned to change the default to xfail in future releases as this is considered less error prone. (#2527)
• Incompatible change: after community feedback the logging functionality has undergone some changes. Please consult the logging documentation for details. (#3013)
• Console output falls back to “classic” mode when capturing is disabled (-s), otherwise the output gets garbled to the point of being useless. (#3038)
• New pytest_runtest_logfinish hook which is called when a test item has finished executing, analogous to pytest_runtest_logstart. (#3101)
• Improve performance when collecting tests using many fixtures. (#3107)
• New caplog.get_records(when) method which provides access to the captured records for the "setup", "call" and "teardown" testing stages. (#3117)
• New fixture record_xml_attribute that allows modifying and inserting attributes on the <testcase> xml node in JUnit reports. (#3130)
• The default cache directory has been renamed from .cache to .pytest_cache after community feedback that the name .cache did not make it clear that it was used by pytest. (#3138)
• Colorize the levelname column in the live-log output. (#3142)

Bug Fixes

• Fix hanging pexpect test on MacOS by using flush() instead of wait(). (#2022)
• Fix restoring Python state after in-process pytest runs with the pytester plugin; this may break tests using multiple inprocess pytest runs if later ones depend on earlier ones leaking global interpreter changes. (#3016)
• Fix skipping plugin reporting hook when test aborted before plugin setup hook. (#3074)
• Fix progress percentage reported when tests fail during teardown. (#3088)
• Incompatible change: -o/--override option no longer eats all the remaining options, which can lead to surprising behavior: for example, pytest -o foo=1 /path/to/test.py would fail because /path/to/test.py would be considered as part of the -o command-line argument. One consequence of this is that now multiple configuration overrides need multiple -o flags: pytest -o foo=1 -o bar=2. (#3103)

Improved Documentation

• Document hooks (defined with historic=True) which cannot be used with hookwrapper=True. (#2423)
• Clarify that warning capturing doesn’t change the warning filter by default. (#2457)
• Clarify a possible confusion when using pytest_fixture_setup with fixture functions that return None. (#2698)
• Fix the wording of a sentence on doctest flags used in pytest. (#3076)
• Prefer https://*.readthedocs.io over http://*.rtfd.org for links in the documentation. (#3092)
• Improve readability (wording, grammar) of Getting Started guide (#3131)
• Added note that calling pytest.main multiple times from the same process is not recommended because of import caching. (#3143)

Trivial/Internal Changes

• Show a simple and easy error when keyword expressions trigger a syntax error (for example, "-k foo and import" will show an error that you can not use the import keyword in expressions). (#2953)
• Change parametrized automatic test id generation to use the __name__ attribute of functions instead of the fallback argument name plus counter. (#2976)
• Replace py.std with stdlib imports. (#3067)
• Corrected ‘you’ to ‘your’ in logging docs. (#3129)

pytest 3.3.2 (2017-12-25)

Bug Fixes

• pytester: ignore files used to obtain current user metadata in the fd leak detector. (#2784)
• Fix memory leak where objects returned by fixtures were never destructed by the garbage collector. (#2981)
• Fix conversion of pyargs to filename to not convert symlinks on Python 2. (#2985)
• PYTEST_DONT_REWRITE is now checked for plugins too rather than only for test modules. (#2995)

Improved Documentation

• Add clarifying note about behavior of multiple parametrized arguments (#3001)

Trivial/Internal Changes

• Code cleanup. (#3015, #3021)
• Clean up code by replacing imports and references of _ast to ast. (#3018)

pytest 3.3.1 (2017-12-05)

Bug Fixes

• Fix issue about -p no:<plugin> having no effect. (#2920)
• Fix regression with warnings that contained non-strings in their arguments in Python 2. (#2956)
• Always escape null bytes when setting PYTEST_CURRENT_TEST. (#2957)
• Fix ZeroDivisionError when using the testmon plugin when no tests were actually collected. (#2971)
• Bring back TerminalReporter.writer as an alias to TerminalReporter._tw. This alias was removed by accident in the 3.3.0 release. (#2984)
• The pytest-capturelog plugin is now also blacklisted, avoiding errors when running pytest with it still installed. (#3004)

Improved Documentation

• Fix broken link to plugin pytest-localserver. (#2963)

Trivial/Internal Changes

• Update github “bugs” link in CONTRIBUTING.rst (#2949)

pytest 3.3.0 (2017-11-23)

Deprecations and Removals

• pytest no longer supports Python 2.6 and 3.3. Those Python versions are EOL for some time now and incur maintenance and compatibility costs on the pytest core team, and following up with the rest of the community we decided that they will no longer be supported starting on this version. Users which still require those versions should pin pytest to <3.3. (#2812)
• Remove internal _preloadplugins() function. This removal is part of the pytest_namespace() hook deprecation. (#2636)
• Internally change CallSpec2 to have a list of marks instead of a broken mapping of keywords. This removes the keywords attribute of the internal CallSpec2 class. (#2672)
• Remove ParameterSet.deprecated_arg_dict - its not a public api and the lack of the underscore was a naming error. (#2675)
• Remove the internal multi-typed attribute Node._evalskip and replace it with the boolean Node._skipped_by_mark. (#2767)
• The params list passed to pytest.fixture is now for all effects considered immutable and frozen at the moment of the pytest.fixture call. Previously the list could be changed before the first invocation of the fixture allowing for a form of dynamic parametrization (for example, updated from command-line options), but this was an unwanted implementation detail which complicated the internals and prevented some internal cleanup. See issue #2959 for details and a recommended workaround.

Features

• pytest_fixture_post_finalizer hook can now receive a request argument. (#2124)
• Replace the old introspection code in compat.py that determines the available arguments of fixtures with inspect.signature on Python 3 and funcsigs.signature on Python 2. This should respect __signature__ declarations on functions. (#2267)
• Report tests with global pytestmark variable only once. (#2549)
• Now pytest displays the total progress percentage while running tests. The previous output style can be set by configuring the console_output_style setting to classic. (#2657)
• Match warns signature to raises by adding match keyword. (#2708)
• pytest now captures and displays output from the standard logging module. The user can control the logging level to be captured by specifying options in pytest.ini, the command line and also during individual tests using markers. Also, a caplog fixture is available that enables users to test the captured log during specific tests (similar to capsys for example). For more information, please see the logging docs. This feature was introduced by merging the popular pytest-catchlog plugin, thanks to Thomas Hisch. Be advised that during the merging the backward compatibility interface with the defunct pytest-capturelog has been dropped. (#2794)
• Add allow_module_level kwarg to pytest.skip(), enabling to skip the whole module. (#2808)
• Allow setting file_or_dir, -c, and -o in PYTEST_ADDOPTS. (#2824)
• Return stdout/stderr capture results as a namedtuple, so out and err can be accessed by attribute. (#2879)
• Add capfdbinary, a version of capfd which returns bytes from readouterr(). (#2923)
• Add capsysbinary a version of capsys which returns bytes from readouterr(). (#2934)
• Implement feature to skip setup.py files when run with --doctest-modules. (#502)

Bug Fixes

• Resume output capturing after capsys/capfd.disabled() context manager. (#1993)
• pytest_fixture_setup and pytest_fixture_post_finalizer hooks are now called for all conftest.py files. (#2124)
• If an exception happens while loading a plugin, pytest no longer hides the original traceback. In Python 2 it will show the original traceback with a new message that explains in which plugin. In Python 3 it will show 2 canonized exceptions, the original exception while loading the plugin in addition to an exception that pytest throws about loading a plugin. (#2491)
• capsys and capfd can now be used by other fixtures. (#2709)
• Internal pytester plugin properly encodes bytes arguments to utf-8. (#2738)
• testdir now uses use the same method used by tmpdir to create its temporary directory. This changes the final structure of the testdir directory slightly, but should not affect usage in normal scenarios and avoids a number of potential problems. (#2751)
• pytest no longer complains about warnings with unicode messages being non-ascii compatible even for ascii-compatible messages. As a result of this, warnings with unicode messages are converted first to an ascii representation for safety. (#2809)
• Change return value of pytest command when --maxfail is reached from 2 (interrupted) to 1 (failed). (#2845)
• Fix issue in assertion rewriting which could lead it to rewrite modules which should not be rewritten. (#2939)
• Handle marks without description in pytest.ini. (#2942)

Trivial/Internal Changes

• pytest now depends on attrs for internal structures to ease code maintainability. (#2641)
• Refactored internal Python 2/3 compatibility code to use six. (#2642)
• Stop vendoring pluggy - we’re missing out on its latest changes for not much benefit (#2719)
• Internal refactor: simplify ascii string escaping by using the backslashreplace error handler in newer Python 3 versions. (#2734)
• Remove unnecessary mark evaluator in unittest plugin (#2767)
• Calls to Metafunc.addcall now emit a deprecation warning. This function is scheduled to be removed in pytest-4.0. (#2876)
• Internal move of the parameterset extraction to a more maintainable place. (#2877)
• Internal refactoring to simplify scope node lookup. (#2910)
• Configure pytest to prevent pip from installing pytest in unsupported Python versions. (#2922)

pytest 3.2.5 (2017-11-15)

Bug Fixes

• Remove py<1.5 restriction from pytest as this can cause version conflicts in some installations. (#2926)

pytest 3.2.4 (2017-11-13)

Bug Fixes

• Fix the bug where running with --pyargs will result in items with empty parent.nodeid if run from a different root directory. (#2775)
• Fix issue with @pytest.parametrize if argnames was specified as keyword arguments. (#2819)
• Strip whitespace from marker names when reading them from INI config. (#2856)
• Show full context of doctest source in the pytest output, if the line number of failed example in the docstring is < 9. (#2882)
• Match fixture paths against actual path segments in order to avoid matching folders which share a prefix. (#2836)

Improved Documentation

• Introduce a dedicated section about conftest.py. (#1505)
• Explicitly mention xpass in the documentation of xfail. (#1997)
• Append example for pytest.param in the example/parametrize document. (#2658)
• Clarify language of proposal for fixtures parameters (#2893)
• List python 3.6 in the documented supported versions in the getting started document. (#2903)
• Clarify the documentation of available fixture scopes. (#538)
• Add documentation about the python -m pytest invocation adding the current directory to sys.path. (#911)

pytest 3.2.3 (2017-10-03)

Bug Fixes

• Fix crash in tab completion when no prefix is given. (#2748)
• The equality checking function (__eq__) of MarkDecorator returns False if one object is not an instance of MarkDecorator. (#2758)
• When running pytest --fixtures-per-test: don’t crash if an item has no _fixtureinfo attribute (e.g. doctests) (#2788)

Improved Documentation

• In help text of -k option, add example of using not to not select certain tests whose names match the provided expression. (#1442)
• Add note in parametrize.rst about calling metafunc.parametrize multiple times. (#1548)

Trivial/Internal Changes

• Set xfail_strict=True in pytest’s own test suite to catch expected failures as soon as they start to pass. (#2722)
• Fix typo in example of passing a callable to markers (in example/markers.rst) (#2765)

pytest 3.2.2 (2017-09-06)

Bug Fixes

• Calling the deprecated request.getfuncargvalue() now shows the source of the call. (#2681)
• Allow tests declared as @staticmethod to use fixtures. (#2699)
• Fixed edge-case during collection: attributes which raised pytest.fail when accessed would abort the entire collection. (#2707)
• Fix ReprFuncArgs with mixed unicode and UTF-8 args. (#2731)

Improved Documentation

• In examples on working with custom markers, add examples demonstrating the usage of pytest.mark.MARKER_NAME.with_args in comparison with pytest.mark.MARKER_NAME.__call__ (#2604)
• In one of the simple examples, use pytest_collection_modifyitems() to skip tests based on a command-line option, allowing its sharing while preventing a user error when acessing pytest.config before the argument parsing. (#2653)

Trivial/Internal Changes

• Fixed minor error in ‘Good Practices/Manual Integration’ code snippet. (#2691)
• Fixed typo in goodpractices.rst. (#2721)
• Improve user guidance regarding --resultlog deprecation. (#2739)

pytest 3.2.1 (2017-08-08)

Bug Fixes

• Fixed small terminal glitch when collecting a single test item. (#2579)
• Correctly consider / as the file separator to automatically mark plugin files for rewrite on Windows. (#2591)
• Properly escape test names when setting PYTEST_CURRENT_TEST environment variable. (#2644)
• Fix error on Windows and Python 3.6+ when sys.stdout has been replaced with a stream-like object which does not implement the full io module buffer protocol. In particular this affects pytest-xdist users on the aforementioned platform. (#2666)

Improved Documentation

• Explicitly document which pytest features work with unittest. (#2626)

pytest 3.2.0 (2017-07-30)

Deprecations and Removals

• pytest.approx no longer supports >, >=, < and <= operators to avoid surprising/inconsistent behavior. See the approx docs for more information. (#2003)
• All old-style specific behavior in current classes in the pytest’s API is considered deprecated at this point and will be removed in a future release. This affects Python 2 users only and in rare situations. (#2147)
• A deprecation warning is now raised when using marks for parameters in pytest.mark.parametrize. Use pytest.param to apply marks to parameters instead. (#2427)

Features

• Add support for numpy arrays (and dicts) to approx. (#1994)
• Now test function objects have a pytestmark attribute containing a list of marks applied directly to the test function, as opposed to marks inherited from parent classes or modules. (#2516)
• Collection ignores local virtualenvs by default; --collect-in-virtualenv overrides this behavior. (#2518)
• Allow class methods decorated as @staticmethod to be candidates for collection as a test function. (Only for Python 2.7 and above. Python 2.6 will still ignore static methods.) (#2528)
• Introduce mark.with_args in order to allow passing functions/classes as sole argument to marks. (#2540)
• New cache_dir ini option: sets the directory where the contents of the cache plugin are stored. Directory may be relative or absolute path: if relative path, then directory is created relative to rootdir, otherwise it is used as is. Additionally path may contain environment variables which are expanded during runtime. (#2543)
• Introduce the PYTEST_CURRENT_TEST environment variable that is set with the nodeid and stage (setup, call and teardown) of the test being currently executed. See the documentation for more info. (#2583)
• Introduced @pytest.mark.filterwarnings mark which allows overwriting the warnings filter on a per test, class or module level. See the docs for more information. (#2598)
• --last-failed now remembers forever when a test has failed and only forgets it if it passes again. This makes it easy to fix a test suite by selectively running files and fixing tests incrementally. (#2621)
• New pytest_report_collectionfinish hook which allows plugins to add messages to the terminal reporting after collection has been finished successfully. (#2622)
• Added support for PEP-415’s Exception.__suppress_context__. Now if a raise exception from None is caught by pytest, pytest will no longer chain the context in the test report. The behavior now matches Python’s traceback behavior. (#2631)
• Exceptions raised by pytest.fail, pytest.skip and pytest.xfail now subclass BaseException, making them harder to be caught unintentionally by normal code. (#580)

Bug Fixes

• Set stdin to a closed PIPE in pytester.py.Testdir.popen() for avoid unwanted interactive pdb (#2023)
• Add missing encoding attribute to sys.std* streams when using capsys capture mode. (#2375)
• Fix terminal color changing to black on Windows if colorama is imported in a conftest.py file. (#2510)
• Fix line number when reporting summary of skipped tests. (#2548)
• capture: ensure that EncodedFile.name is a string. (#2555)
• The options --fixtures and --fixtures-per-test will now keep indentation within docstrings. (#2574)
• doctests line numbers are now reported correctly, fixing pytest-sugar#122. (#2610)
• Fix non-determinism in order of fixture collection. Adds new dependency (ordereddict) for Python 2.6. (#920)

Improved Documentation

• Clarify pytest_configure hook call order. (#2539)
• Extend documentation for testing plugin code with the pytester plugin. (#971)

Trivial/Internal Changes

• Update help message for --strict to make it clear it only deals with unregistered markers, not warnings. (#2444)
• Internal code move: move code for pytest.approx/pytest.raises to own files in order to cut down the size of python.py (#2489)
• Renamed the utility function _pytest.compat._escape_strings to _ascii_escaped to better communicate the function’s purpose. (#2533)
• Improve error message for CollectError with skip/skipif. (#2546)
• Emit warning about yield tests being deprecated only once per generator. (#2562)
• Ensure final collected line doesn’t include artifacts of previous write. (#2571)
• Fixed all flake8 errors and warnings. (#2581)
• Added fix-lint tox environment to run automatic pep8 fixes on the code. (#2582)
• Turn warnings into errors in pytest’s own test suite in order to catch regressions due to deprecations more promptly. (#2588)
• Show multiple issue links in CHANGELOG entries. (#2620)

pytest 3.1.3 (2017-07-03)

Bug Fixes

• Fix decode error in Python 2 for doctests in docstrings. (#2434)
• Exceptions raised during teardown by finalizers are now suppressed until all finalizers are called, with the initial exception reraised. (#2440)
• Fix incorrect “collected items” report when specifying tests on the command- line. (#2464)
• deprecated_call in context-manager form now captures deprecation warnings even if the same warning has already been raised. Also, deprecated_call will always produce the same error message (previously it would produce different messages in context-manager vs. function-call mode). (#2469)
• Fix issue where paths collected by pytest could have triple leading / characters. (#2475)
• Fix internal error when trying to detect the start of a recursive traceback. (#2486)

Improved Documentation

• Explicitly state for which hooks the calls stop after the first non-None result. (#2493)

Trivial/Internal Changes

• Create invoke tasks for updating the vendored packages. (#2474)
• Update copyright dates in LICENSE, README.rst and in the documentation. (#2499)

pytest 3.1.2 (2017-06-08)

Bug Fixes

• Required options added via pytest_addoption will no longer prevent using –help without passing them. (#1999)
• Respect python_files in assertion rewriting. (#2121)
• Fix recursion error detection when frames in the traceback contain objects that can’t be compared (like numpy arrays). (#2459)
• UnicodeWarning is issued from the internal pytest warnings plugin only when the message contains non-ascii unicode (Python 2 only). (#2463)
• Added a workaround for Python 3.6 WindowsConsoleIO breaking due to Pytests’s FDCapture. Other code using console handles might still be affected by the very same issue and might require further workarounds/fixes, i.e. colorama. (#2467)

Improved Documentation

• Fix internal API links to pluggy objects. (#2331)
• Make it clear that pytest.xfail stops test execution at the calling point and improve overall flow of the skipping docs. (#810)

pytest 3.1.1 (2017-05-30)

Bug Fixes

• pytest warning capture no longer overrides existing warning filters. The previous behaviour would override all filters and caused regressions in test suites which configure warning filters to match their needs. Note that as a side-effect of this is that DeprecationWarning and PendingDeprecationWarning are no longer shown by default. (#2430)
• Fix issue with non-ascii contents in doctest text files. (#2434)
• Fix encoding errors for unicode warnings in Python 2. (#2436)
• pytest.deprecated_call now captures PendingDeprecationWarning in context manager form. (#2441)

Improved Documentation

• Addition of towncrier for changelog management. (#2390)

3.1.0 (2017-05-22)

New Features

• The pytest-warnings plugin has been integrated into the core and now pytest automatically captures and displays warnings at the end of the test session.

  Warning

  This feature may disrupt test suites which apply and treat warnings themselves, and can be disabled in your pytest.ini:

  [pytest]
  addopts = -p no:warnings
  

  See the warnings documentation page for more information.

  Thanks @nicoddemus for the PR.

• Added junit_suite_name ini option to specify root <testsuite> name for JUnit XML reports (#533).

• Added an ini option doctest_encoding to specify which encoding to use for doctest files. Thanks @wheerd for the PR (#2101).

• pytest.warns now checks for subclass relationship rather than class equality. Thanks @lesteve for the PR (#2166)

• pytest.raises now asserts that the error message matches a text or regex with the match keyword argument. Thanks @Kriechi for the PR.

• pytest.param can be used to declare test parameter sets with marks and test ids. Thanks @RonnyPfannschmidt for the PR.

Changes

• remove all internal uses of pytest_namespace hooks, this is to prepare the removal of preloadconfig in pytest 4.0 Thanks to @RonnyPfannschmidt for the PR.
• pytest now warns when a callable ids raises in a parametrized test. Thanks @fogo for the PR.
• It is now possible to skip test classes from being collected by setting a __test__ attribute to False in the class body (#2007). Thanks to @syre for the report and @lwm for the PR.
• Change junitxml.py to produce reports that comply with Junitxml schema. If the same test fails with failure in call and then errors in teardown we split testcase element into two, one containing the error and the other the failure. (#2228) Thanks to @kkoukiou for the PR.
• Testcase reports with a url attribute will now properly write this to junitxml. Thanks @fushi for the PR (#1874).
• Remove common items from dict comparison output when verbosity=1. Also update the truncation message to make it clearer that pytest truncates all assertion messages if verbosity < 2 (#1512). Thanks @mattduck for the PR
• --pdbcls no longer implies --pdb. This makes it possible to use addopts=--pdbcls=module.SomeClass on pytest.ini. Thanks @davidszotten for the PR (#1952).
• fix #2013: turn RecordedWarning into namedtuple, to give it a comprehensible repr while preventing unwarranted modification.
• fix #2208: ensure an iteration limit for _pytest.compat.get_real_func. Thanks @RonnyPfannschmidt for the report and PR.
• Hooks are now verified after collection is complete, rather than right after loading installed plugins. This makes it easy to write hooks for plugins which will be loaded during collection, for example using the pytest_plugins special variable (#1821). Thanks @nicoddemus for the PR.
• Modify pytest_make_parametrize_id() hook to accept argname as an additional parameter. Thanks @unsignedint for the PR.
• Add venv to the default norecursedirs setting. Thanks @The-Compiler for the PR.
• PluginManager.import_plugin now accepts unicode plugin names in Python 2. Thanks @reutsharabani for the PR.
• fix #2308: When using both --lf and --ff, only the last failed tests are run. Thanks @ojii for the PR.
• Replace minor/patch level version numbers in the documentation with placeholders. This significantly reduces change-noise as different contributors regnerate the documentation on different platforms. Thanks @RonnyPfannschmidt for the PR.
• fix #2391: consider pytest_plugins on all plugin modules Thanks @RonnyPfannschmidt for the PR.

Bug Fixes

• Fix AttributeError on sys.stdout.buffer / sys.stderr.buffer while using capsys fixture in python 3. (#1407). Thanks to @asottile.
• Change capture.py’s DontReadFromInput class to throw io.UnsupportedOperation errors rather than ValueErrors in the fileno method (#2276). Thanks @metasyn and @vlad-dragos for the PR.
• Fix exception formatting while importing modules when the exception message contains non-ascii characters (#2336). Thanks @fabioz for the report and @nicoddemus for the PR.
• Added documentation related to issue (#1937) Thanks @skylarjhdownes for the PR.
• Allow collecting files with any file extension as Python modules (#2369). Thanks @Kodiologist for the PR.
• Show the correct error message when collect “parametrize” func with wrong args (#2383). Thanks @The-Compiler for the report and @robin0371 for the PR.

3.0.7 (2017-03-14)

• Fix issue in assertion rewriting breaking due to modules silently discarding other modules when importing fails Notably, importing the anydbm module is fixed. (#2248). Thanks @pfhayes for the PR.
• junitxml: Fix problematic case where system-out tag occurred twice per testcase element in the XML report. Thanks @kkoukiou for the PR.
• Fix regression, pytest now skips unittest correctly if run with --pdb (#2137). Thanks to @gst for the report and @mbyt for the PR.
• Ignore exceptions raised from descriptors (e.g. properties) during Python test collection (#2234). Thanks to @bluetech.
• --override-ini now correctly overrides some fundamental options like python_files (#2238). Thanks @sirex for the report and @nicoddemus for the PR.
• Replace raise StopIteration usages in the code by simple returns to finish generators, in accordance to PEP-479 (#2160). Thanks to @nicoddemus for the PR.
• Fix internal errors when an unprintable AssertionError is raised inside a test. Thanks @omerhadari for the PR.
• Skipping plugin now also works with test items generated by custom collectors (#2231). Thanks to @vidartf.
• Fix trailing whitespace in console output if no .ini file presented (#2281). Thanks @fbjorn for the PR.
• Conditionless xfail markers no longer rely on the underlying test item being an instance of PyobjMixin, and can therefore apply to tests not collected by the built-in python test collector. Thanks @barneygale for the PR.

3.0.6 (2017-01-22)

• pytest no longer generates PendingDeprecationWarning from its own operations, which was introduced by mistake in version 3.0.5 (#2118). Thanks to @nicoddemus for the report and @RonnyPfannschmidt for the PR.
• pytest no longer recognizes coroutine functions as yield tests (#2129). Thanks to @malinoff for the PR.
• Plugins loaded by the PYTEST_PLUGINS environment variable are now automatically considered for assertion rewriting (#2185). Thanks @nicoddemus for the PR.
• Improve error message when pytest.warns fails (#2150). The type(s) of the expected warnings and the list of caught warnings is added to the error message. Thanks @lesteve for the PR.
• Fix pytester internal plugin to work correctly with latest versions of zope.interface (#1989). Thanks @nicoddemus for the PR.
• Assert statements of the pytester plugin again benefit from assertion rewriting (#1920). Thanks @RonnyPfannschmidt for the report and @nicoddemus for the PR.
• Specifying tests with colons like test_foo.py::test_bar for tests in subdirectories with ini configuration files now uses the correct ini file (#2148). Thanks @pelme.
• Fail testdir.runpytest().assert_outcomes() explicitly if the pytest terminal output it relies on is missing. Thanks to @eli-b for the PR.

3.0.5 (2016-12-05)

• Add warning when not passing option=value correctly to -o/--override-ini (#2105). Also improved the help documentation. Thanks to @mbukatov for the report and @lwm for the PR.
• Now --confcutdir and --junit-xml are properly validated if they are directories and filenames, respectively (#2089 and #2078). Thanks to @lwm for the PR.
• Add hint to error message hinting possible missing __init__.py (#478). Thanks @DuncanBetts.
• More accurately describe when fixture finalization occurs in documentation (#687). Thanks @DuncanBetts.
• Provide :ref: targets for recwarn.rst so we can use intersphinx referencing. Thanks to @dupuy for the report and @lwm for the PR.
• In Python 2, use a simple +- ASCII string in the string representation of pytest.approx (for example "4 +- 4.0e-06") because it is brittle to handle that in different contexts and representations internally in pytest which can result in bugs such as #2111. In Python 3, the representation still uses ± (for example 4 ± 4.0e-06). Thanks @kerrick-lyft for the report and @nicoddemus for the PR.
• Using item.Function, item.Module, etc., is now issuing deprecation warnings, prefer pytest.Function, pytest.Module, etc., instead (#2034). Thanks @nmundar for the PR.
• Fix error message using approx with complex numbers (#2082). Thanks @adler-j for the report and @nicoddemus for the PR.
• Fixed false-positives warnings from assertion rewrite hook for modules imported more than once by the pytest_plugins mechanism. Thanks @nicoddemus for the PR.
• Remove an internal cache which could cause hooks from conftest.py files in sub-directories to be called in other directories incorrectly (#2016). Thanks @d-b-w for the report and @nicoddemus for the PR.
• Remove internal code meant to support earlier Python 3 versions that produced the side effect of leaving None in sys.modules when expressions were evaluated by pytest (for example passing a condition as a string to pytest.mark.skipif)(#2103). Thanks @jaraco for the report and @nicoddemus for the PR.
• Cope gracefully with a .pyc file with no matching .py file (#2038). Thanks @nedbat.

3.0.4 (2016-11-09)

• Import errors when collecting test modules now display the full traceback (#1976). Thanks @cwitty for the report and @nicoddemus for the PR.
• Fix confusing command-line help message for custom options with two or more metavar properties (#2004). Thanks @okulynyak and @davehunt for the report and @nicoddemus for the PR.
• When loading plugins, import errors which contain non-ascii messages are now properly handled in Python 2 (#1998). Thanks @nicoddemus for the PR.
• Fixed cyclic reference when pytest.raises is used in context-manager form (#1965). Also as a result of this fix, sys.exc_info() is left empty in both context-manager and function call usages. Previously, sys.exc_info would contain the exception caught by the context manager, even when the expected exception occurred. Thanks @MSeifert04 for the report and the PR.
• Fixed false-positives warnings from assertion rewrite hook for modules that were rewritten but were later marked explicitly by pytest.register_assert_rewrite or implicitly as a plugin (#2005). Thanks @RonnyPfannschmidt for the report and @nicoddemus for the PR.
• Report teardown output on test failure (#442). Thanks @matclab for the PR.
• Fix teardown error message in generated xUnit XML. Thanks @gdyuldin for the PR.
• Properly handle exceptions in multiprocessing tasks (#1984). Thanks @adborden for the report and @nicoddemus for the PR.
• Clean up unittest TestCase objects after tests are complete (#1649). Thanks @d_b_w for the report and PR.

3.0.3 (2016-09-28)

• The ids argument to parametrize again accepts unicode strings in Python 2 (#1905). Thanks @philpep for the report and @nicoddemus for the PR.
• Assertions are now being rewritten for plugins in development mode (pip install -e) (#1934). Thanks @nicoddemus for the PR.
• Fix pkg_resources import error in Jython projects (#1853). Thanks @raquel-ucl for the PR.
• Got rid of AttributeError: 'Module' object has no attribute '_obj' exception in Python 3 (#1944). Thanks @axil for the PR.
• Explain a bad scope value passed to @fixture declarations or a MetaFunc.parametrize() call.
• This version includes pluggy-0.4.0, which correctly handles VersionConflict errors in plugins (#704). Thanks @nicoddemus for the PR.

3.0.2 (2016-09-01)

• Improve error message when passing non-string ids to pytest.mark.parametrize (#1857). Thanks @okken for the report and @nicoddemus for the PR.
• Add buffer attribute to stdin stub class pytest.capture.DontReadFromInput Thanks @joguSD for the PR.
• Fix UnicodeEncodeError when string comparison with unicode has failed. (#1864) Thanks @AiOO for the PR.
• pytest_plugins is now handled correctly if defined as a string (as opposed as a sequence of strings) when modules are considered for assertion rewriting. Due to this bug, much more modules were being rewritten than necessary if a test suite uses pytest_plugins to load internal plugins (#1888). Thanks @jaraco for the report and @nicoddemus for the PR (#1891).
• Do not call tearDown and cleanups when running tests from unittest.TestCase subclasses with --pdb enabled. This allows proper post mortem debugging for all applications which have significant logic in their tearDown machinery (#1890). Thanks @mbyt for the PR.
• Fix use of deprecated getfuncargvalue method in the internal doctest plugin. Thanks @ViviCoder for the report (#1898).

3.0.1 (2016-08-23)

• Fix regression when importorskip is used at module level (#1822). Thanks @jaraco and @The-Compiler for the report and @nicoddemus for the PR.
• Fix parametrization scope when session fixtures are used in conjunction with normal parameters in the same call (#1832). Thanks @The-Compiler for the report, @Kingdread and @nicoddemus for the PR.
• Fix internal error when parametrizing tests or fixtures using an empty ids argument (#1849). Thanks @OPpuolitaival for the report and @nicoddemus for the PR.
• Fix loader error when running pytest embedded in a zipfile. Thanks @mbachry for the PR.

3.0.0 (2016-08-18)

Incompatible changes

A number of incompatible changes were made in this release, with the intent of removing features deprecated for a long time or change existing behaviors in order to make them less surprising/more useful.

• Reinterpretation mode has now been removed. Only plain and rewrite mode are available, consequently the --assert=reinterp option is no longer available. This also means files imported from plugins or conftest.py will not benefit from improved assertions by default, you should use pytest.register_assert_rewrite() to explicitly turn on assertion rewriting for those files. Thanks @flub for the PR.

• The following deprecated commandline options were removed:

  • --genscript: no longer supported;
  • --no-assert: use --assert=plain instead;
  • --nomagic: use --assert=plain instead;
  • --report: use -r instead;

  Thanks to @RedBeardCode for the PR (#1664).

• ImportErrors in plugins now are a fatal error instead of issuing a pytest warning (#1479). Thanks to @The-Compiler for the PR.

• Removed support code for Python 3 versions < 3.3 (#1627).

• Removed all py.test-X* entry points. The versioned, suffixed entry points were never documented and a leftover from a pre-virtualenv era. These entry points also created broken entry points in wheels, so removing them also removes a source of confusion for users (#1632). Thanks @obestwalter for the PR.

• pytest.skip() now raises an error when used to decorate a test function, as opposed to its original intent (to imperatively skip a test inside a test function). Previously this usage would cause the entire module to be skipped (#607). Thanks @omarkohl for the complete PR (#1519).

• Exit tests if a collection error occurs. A poll indicated most users will hit CTRL-C anyway as soon as they see collection errors, so pytest might as well make that the default behavior (#1421). A --continue-on-collection-errors option has been added to restore the previous behaviour. Thanks @olegpidsadnyi and @omarkohl for the complete PR (#1628).

• Renamed the pytest pdb module (plugin) into debugging to avoid clashes with the builtin pdb module.

• Raise a helpful failure message when requesting a parametrized fixture at runtime, e.g. with request.getfixturevalue. Previously these parameters were simply never defined, so a fixture decorated like @pytest.fixture(params=[0, 1, 2]) only ran once (#460). Thanks to @nikratio for the bug report, @RedBeardCode and @tomviner for the PR.

• _pytest.monkeypatch.monkeypatch class has been renamed to _pytest.monkeypatch.MonkeyPatch so it doesn’t conflict with the monkeypatch fixture.

• --exitfirst / -x can now be overridden by a following --maxfail=N and is just a synonym for --maxfail=1.

New Features

• Support nose-style __test__ attribute on methods of classes, including unittest-style Classes. If set to False, the test will not be collected.

• New doctest_namespace fixture for injecting names into the namespace in which doctests run. Thanks @milliams for the complete PR (#1428).

• New --doctest-report option available to change the output format of diffs when running (failing) doctests (implements #1749). Thanks @hartym for the PR.

• New name argument to pytest.fixture decorator which allows a custom name for a fixture (to solve the funcarg-shadowing-fixture problem). Thanks @novas0x2a for the complete PR (#1444).

• New approx() function for easily comparing floating-point numbers in tests. Thanks @kalekundert for the complete PR (#1441).

• Ability to add global properties in the final xunit output file by accessing the internal junitxml plugin (experimental). Thanks @tareqalayan for the complete PR #1454).

• New ExceptionInfo.match() method to match a regular expression on the string representation of an exception (#372). Thanks @omarkohl for the complete PR (#1502).

• __tracebackhide__ can now also be set to a callable which then can decide whether to filter the traceback based on the ExceptionInfo object passed to it. Thanks @The-Compiler for the complete PR (#1526).

• New pytest_make_parametrize_id(config, val) hook which can be used by plugins to provide friendly strings for custom types. Thanks @palaviv for the PR.

• capsys and capfd now have a disabled() context-manager method, which can be used to temporarily disable capture within a test. Thanks @nicoddemus for the PR.

• New cli flag --fixtures-per-test: shows which fixtures are being used for each selected test item. Features doc strings of fixtures by default. Can also show where fixtures are defined if combined with -v. Thanks @hackebrot for the PR.

• Introduce pytest command as recommended entry point. Note that py.test still works and is not scheduled for removal. Closes proposal #1629. Thanks @obestwalter and @davehunt for the complete PR (#1633).

• New cli flags:

  • --setup-plan: performs normal collection and reports the potential setup and teardown and does not execute any fixtures and tests;
  • --setup-only: performs normal collection, executes setup and teardown of fixtures and reports them;
  • --setup-show: performs normal test execution and additionally shows setup and teardown of fixtures;
  • --keep-duplicates: py.test now ignores duplicated paths given in the command line. To retain the previous behavior where the same test could be run multiple times by specifying it in the command-line multiple times, pass the --keep-duplicates argument (#1609);

  Thanks @d6e, @kvas-it, @sallner, @ioggstream and @omarkohl for the PRs.

• New CLI flag --override-ini/-o: overrides values from the ini file. For example: "-o xfail_strict=True"’. Thanks @blueyed and @fengxx for the PR.

• New hooks:

  • pytest_fixture_setup(fixturedef, request): executes fixture setup;
  • pytest_fixture_post_finalizer(fixturedef): called after the fixture’s finalizer and has access to the fixture’s result cache.

  Thanks @d6e, @sallner.

• Issue warnings for asserts whose test is a tuple literal. Such asserts will never fail because tuples are always truthy and are usually a mistake (see #1562). Thanks @kvas-it, for the PR.

• Allow passing a custom debugger class (e.g. --pdbcls=IPython.core.debugger:Pdb). Thanks to @anntzer for the PR.

Changes

• Plugins now benefit from assertion rewriting. Thanks @sober7, @nicoddemus and @flub for the PR.
• Change report.outcome for xpassed tests to "passed" in non-strict mode and "failed" in strict mode. Thanks to @hackebrot for the PR (#1795) and @gprasad84 for report (#1546).
• Tests marked with xfail(strict=False) (the default) now appear in JUnitXML reports as passing tests instead of skipped. Thanks to @hackebrot for the PR (#1795).
• Highlight path of the file location in the error report to make it easier to copy/paste. Thanks @suzaku for the PR (#1778).
• Fixtures marked with @pytest.fixture can now use yield statements exactly like those marked with the @pytest.yield_fixture decorator. This change renders @pytest.yield_fixture deprecated and makes @pytest.fixture with yield statements the preferred way to write teardown code (#1461). Thanks @csaftoiu for bringing this to attention and @nicoddemus for the PR.
• Explicitly passed parametrize ids do not get escaped to ascii (#1351). Thanks @ceridwen for the PR.
• Fixtures are now sorted in the error message displayed when an unknown fixture is declared in a test function. Thanks @nicoddemus for the PR.
• pytest_terminal_summary hook now receives the exitstatus of the test session as argument. Thanks @blueyed for the PR (#1809).
• Parametrize ids can accept None as specific test id, in which case the automatically generated id for that argument will be used. Thanks @palaviv for the complete PR (#1468).
• The parameter to xunit-style setup/teardown methods (setup_method, setup_module, etc.) is now optional and may be omitted. Thanks @okken for bringing this to attention and @nicoddemus for the PR.
• Improved automatic id generation selection in case of duplicate ids in parametrize. Thanks @palaviv for the complete PR (#1474).
• Now pytest warnings summary is shown up by default. Added a new flag --disable-pytest-warnings to explicitly disable the warnings summary (#1668).
• Make ImportError during collection more explicit by reminding the user to check the name of the test module/package(s) (#1426). Thanks @omarkohl for the complete PR (#1520).
• Add build/ and dist/ to the default --norecursedirs list. Thanks @mikofski for the report and @tomviner for the PR (#1544).
• pytest.raises in the context manager form accepts a custom message to raise when no exception occurred. Thanks @palaviv for the complete PR (#1616).
• conftest.py files now benefit from assertion rewriting; previously it was only available for test modules. Thanks @flub, @sober7 and @nicoddemus for the PR (#1619).
• Text documents without any doctests no longer appear as “skipped”. Thanks @graingert for reporting and providing a full PR (#1580).
• Ensure that a module within a namespace package can be found when it is specified on the command line together with the --pyargs option. Thanks to @taschini for the PR (#1597).
• Always include full assertion explanation during assertion rewriting. The previous behaviour was hiding sub-expressions that happened to be False, assuming this was redundant information. Thanks @bagerard for reporting (#1503). Thanks to @davehunt and @tomviner for the PR.
• OptionGroup.addoption() now checks if option names were already added before, to make it easier to track down issues like #1618. Before, you only got exceptions later from argparse library, giving no clue about the actual reason for double-added options.
• yield-based tests are considered deprecated and will be removed in pytest-4.0. Thanks @nicoddemus for the PR.
• [pytest] sections in setup.cfg files should now be named [tool:pytest] to avoid conflicts with other distutils commands (see #567). [pytest] sections in pytest.ini or tox.ini files are supported and unchanged. Thanks @nicoddemus for the PR.
• Using pytest_funcarg__ prefix to declare fixtures is considered deprecated and will be removed in pytest-4.0 (#1684). Thanks @nicoddemus for the PR.
• Passing a command-line string to pytest.main() is considered deprecated and scheduled for removal in pytest-4.0. It is recommended to pass a list of arguments instead (#1723).
• Rename getfuncargvalue to getfixturevalue. getfuncargvalue is still present but is now considered deprecated. Thanks to @RedBeardCode and @tomviner for the PR (#1626).
• optparse type usage now triggers DeprecationWarnings (#1740).
• optparse backward compatibility supports float/complex types (#457).
• Refined logic for determining the rootdir, considering only valid paths which fixes a number of issues: #1594, #1435 and #1471. Updated the documentation according to current behavior. Thanks to @blueyed, @davehunt and @matthiasha for the PR.
• Always include full assertion explanation. The previous behaviour was hiding sub-expressions that happened to be False, assuming this was redundant information. Thanks @bagerard for reporting (#1503). Thanks to @davehunt and @tomviner for PR.
• Better message in case of not using parametrized variable (see #1539). Thanks to @tramwaj29 for the PR.
• Updated docstrings with a more uniform style.
• Add stderr write for pytest.exit(msg) during startup. Previously the message was never shown. Thanks @BeyondEvil for reporting #1210. Thanks to @jgsonesen and @tomviner for the PR.
• No longer display the incorrect test deselection reason (#1372). Thanks @ronnypfannschmidt for the PR.
• The --resultlog command line option has been deprecated: it is little used and there are more modern and better alternatives (see #830). Thanks @nicoddemus for the PR.
• Improve error message with fixture lookup errors: add an ‘E’ to the first line and ‘>’ to the rest. Fixes #717. Thanks @blueyed for reporting and a PR, @eolo999 for the initial PR and @tomviner for his guidance during EuroPython2016 sprint.

Bug Fixes

• Parametrize now correctly handles duplicated test ids.
• Fix internal error issue when the method argument is missing for teardown_method() (#1605).
• Fix exception visualization in case the current working directory (CWD) gets deleted during testing (#1235). Thanks @bukzor for reporting. PR by @marscher.
• Improve test output for logical expression with brackets (#925). Thanks @DRMacIver for reporting and @RedBeardCode for the PR.
• Create correct diff for strings ending with newlines (#1553). Thanks @Vogtinator for reporting and @RedBeardCode and @tomviner for the PR.
• ConftestImportFailure now shows the traceback making it easier to identify bugs in conftest.py files (#1516). Thanks @txomon for the PR.
• Text documents without any doctests no longer appear as “skipped”. Thanks @graingert for reporting and providing a full PR (#1580).
• Fixed collection of classes with custom __new__ method. Fixes #1579. Thanks to @Stranger6667 for the PR.
• Fixed scope overriding inside metafunc.parametrize (#634). Thanks to @Stranger6667 for the PR.
• Fixed the total tests tally in junit xml output (#1798). Thanks to @cboelsen for the PR.
• Fixed off-by-one error with lines from request.node.warn. Thanks to @blueyed for the PR.

2.9.2 (2016-05-31)

Bug Fixes

• fix #510: skip tests where one parameterize dimension was empty thanks Alex Stapleton for the Report and @RonnyPfannschmidt for the PR
• Fix Xfail does not work with condition keyword argument. Thanks @astraw38 for reporting the issue (#1496) and @tomviner for PR the (#1524).
• Fix win32 path issue when putting custom config file with absolute path in pytest.main("-c your_absolute_path").
• Fix maximum recursion depth detection when raised error class is not aware of unicode/encoded bytes. Thanks @prusse-martin for the PR (#1506).
• Fix pytest.mark.skip mark when used in strict mode. Thanks @pquentin for the PR and @RonnyPfannschmidt for showing how to fix the bug.
• Minor improvements and fixes to the documentation. Thanks @omarkohl for the PR.
• Fix --fixtures to show all fixture definitions as opposed to just one per fixture name. Thanks to @hackebrot for the PR.

2.9.1 (2016-03-17)

Bug Fixes

• Improve error message when a plugin fails to load. Thanks @nicoddemus for the PR.
• Fix (#1178): pytest.fail with non-ascii characters raises an internal pytest error. Thanks @nicoddemus for the PR.
• Fix (#469): junit parses report.nodeid incorrectly, when params IDs contain ::. Thanks @tomviner for the PR (#1431).
• Fix (#578): SyntaxErrors containing non-ascii lines at the point of failure generated an internal py.test error. Thanks @asottile for the report and @nicoddemus for the PR.
• Fix (#1437): When passing in a bytestring regex pattern to parameterize attempt to decode it as utf-8 ignoring errors.
• Fix (#649): parametrized test nodes cannot be specified to run on the command line.
• Fix (#138): better reporting for python 3.3+ chained exceptions

2.9.0 (2016-02-29)

New Features

• New pytest.mark.skip mark, which unconditionally skips marked tests. Thanks @MichaelAquilina for the complete PR (#1040).
• --doctest-glob may now be passed multiple times in the command-line. Thanks @jab and @nicoddemus for the PR.
• New -rp and -rP reporting options give the summary and full output of passing tests, respectively. Thanks to @codewarrior0 for the PR.
• pytest.mark.xfail now has a strict option, which makes XPASS tests to fail the test suite (defaulting to False). There’s also a xfail_strict ini option that can be used to configure it project-wise. Thanks @rabbbit for the request and @nicoddemus for the PR (#1355).
• Parser.addini now supports options of type bool. Thanks @nicoddemus for the PR.
• New ALLOW_BYTES doctest option. This strips b prefixes from byte strings in doctest output (similar to ALLOW_UNICODE). Thanks @jaraco for the request and @nicoddemus for the PR (#1287).
• Give a hint on KeyboardInterrupt to use the --fulltrace option to show the errors. Fixes #1366. Thanks to @hpk42 for the report and @RonnyPfannschmidt for the PR.
• Catch IndexError exceptions when getting exception source location. Fixes a pytest internal error for dynamically generated code (fixtures and tests) where source lines are fake by intention.

Changes

• Important: py.code has been merged into the pytest repository as pytest._code. This decision was made because py.code had very few uses outside pytest and the fact that it was in a different repository made it difficult to fix bugs on its code in a timely manner. The team hopes with this to be able to better refactor out and improve that code. This change shouldn’t affect users, but it is useful to let users aware if they encounter any strange behavior.

  Keep in mind that the code for pytest._code is private and experimental, so you definitely should not import it explicitly!

  Please note that the original py.code is still available in pylib.

• pytest_enter_pdb now optionally receives the pytest config object. Thanks @nicoddemus for the PR.

• Removed code and documentation for Python 2.5 or lower versions, including removal of the obsolete _pytest.assertion.oldinterpret module. Thanks @nicoddemus for the PR (#1226).

• Comparisons now always show up in full when CI or BUILD_NUMBER is found in the environment, even when -vv isn’t used. Thanks @The-Compiler for the PR.

• --lf and --ff now support long names: --last-failed and --failed-first respectively. Thanks @MichaelAquilina for the PR.

• Added expected exceptions to pytest.raises fail message.

• Collection only displays progress (“collecting X items”) when in a terminal. This avoids cluttering the output when using --color=yes to obtain colors in CI integrations systems (#1397).

Bug Fixes

• The -s and -c options should now work under xdist; Config.fromdictargs now represents its input much more faithfully. Thanks to @bukzor for the complete PR (#680).
• Fix (#1290): support Python 3.5’s @ operator in assertion rewriting. Thanks @Shinkenjoe for report with test case and @tomviner for the PR.
• Fix formatting utf-8 explanation messages (#1379). Thanks @biern for the PR.
• Fix traceback style docs to describe all of the available options (auto/long/short/line/native/no), with auto being the default since v2.6. Thanks @hackebrot for the PR.
• Fix (#1422): junit record_xml_property doesn’t allow multiple records with same name.

2.8.7 (2016-01-24)

• fix #1338: use predictable object resolution for monkeypatch

2.8.6 (2016-01-21)

• fix #1259: allow for double nodeids in junitxml, this was a regression failing plugins combinations like pytest-pep8 + pytest-flakes
• Workaround for exception that occurs in pyreadline when using --pdb with standard I/O capture enabled. Thanks Erik M. Bray for the PR.
• fix #900: Better error message in case the target of a monkeypatch call raises an ImportError.
• fix #1292: monkeypatch calls (setattr, setenv, etc.) are now O(1). Thanks David R. MacIver for the report and Bruno Oliveira for the PR.
• fix #1223: captured stdout and stderr are now properly displayed before entering pdb when --pdb is used instead of being thrown away. Thanks Cal Leeming for the PR.
• fix #1305: pytest warnings emitted during pytest_terminal_summary are now properly displayed. Thanks Ionel Maries Cristian for the report and Bruno Oliveira for the PR.
• fix #628: fixed internal UnicodeDecodeError when doctests contain unicode. Thanks Jason R. Coombs for the report and Bruno Oliveira for the PR.
• fix #1334: Add captured stdout to jUnit XML report on setup error. Thanks Georgy Dyuldin for the PR.

2.8.5 (2015-12-11)

• fix #1243: fixed issue where class attributes injected during collection could break pytest. PR by Alexei Kozlenok, thanks Ronny Pfannschmidt and Bruno Oliveira for the review and help.
• fix #1074: precompute junitxml chunks instead of storing the whole tree in objects Thanks Bruno Oliveira for the report and Ronny Pfannschmidt for the PR
• fix #1238: fix pytest.deprecated_call() receiving multiple arguments (Regression introduced in 2.8.4). Thanks Alex Gaynor for the report and Bruno Oliveira for the PR.

2.8.4 (2015-12-06)

• fix #1190: deprecated_call() now works when the deprecated function has been already called by another test in the same module. Thanks Mikhail Chernykh for the report and Bruno Oliveira for the PR.
• fix #1198: --pastebin option now works on Python 3. Thanks Mehdy Khoshnoody for the PR.
• fix #1219: --pastebin now works correctly when captured output contains non-ascii characters. Thanks Bruno Oliveira for the PR.
• fix #1204: another error when collecting with a nasty __getattr__(). Thanks Florian Bruhin for the PR.
• fix the summary printed when no tests did run. Thanks Florian Bruhin for the PR.
• fix #1185 - ensure MANIFEST.in exactly matches what should go to a sdist
• a number of documentation modernizations wrt good practices. Thanks Bruno Oliveira for the PR.

2.8.3 (2015-11-18)

• fix #1169: add __name__ attribute to testcases in TestCaseFunction to support the @unittest.skip decorator on functions and methods. Thanks Lee Kamentsky for the PR.
• fix #1035: collecting tests if test module level obj has __getattr__(). Thanks Suor for the report and Bruno Oliveira / Tom Viner for the PR.
• fix #331: don’t collect tests if their failure cannot be reported correctly e.g. they are a callable instance of a class.
• fix #1133: fixed internal error when filtering tracebacks where one entry belongs to a file which is no longer available. Thanks Bruno Oliveira for the PR.
• enhancement made to highlight in red the name of the failing tests so they stand out in the output. Thanks Gabriel Reis for the PR.
• add more talks to the documentation
• extend documentation on the –ignore cli option
• use pytest-runner for setuptools integration
• minor fixes for interaction with OS X El Capitan system integrity protection (thanks Florian)

2.8.2 (2015-10-07)

• fix #1085: proper handling of encoding errors when passing encoded byte strings to pytest.parametrize in Python 2. Thanks Themanwithoutaplan for the report and Bruno Oliveira for the PR.
• fix #1087: handling SystemError when passing empty byte strings to pytest.parametrize in Python 3. Thanks Paul Kehrer for the report and Bruno Oliveira for the PR.
• fix #995: fixed internal error when filtering tracebacks where one entry was generated by an exec() statement. Thanks Daniel Hahler, Ashley C Straw, Philippe Gauthier and Pavel Savchenko for contributing and Bruno Oliveira for the PR.
• fix #1100 and #1057: errors when using autouse fixtures and doctest modules. Thanks Sergey B Kirpichev and Vital Kudzelka for contributing and Bruno Oliveira for the PR.

2.8.1 (2015-09-29)

• fix #1034: Add missing nodeid on pytest_logwarning call in addhook. Thanks Simon Gomizelj for the PR.
• ‘deprecated_call’ is now only satisfied with a DeprecationWarning or PendingDeprecationWarning. Before 2.8.0, it accepted any warning, and 2.8.0 made it accept only DeprecationWarning (but not PendingDeprecationWarning). Thanks Alex Gaynor for the issue and Eric Hunsberger for the PR.
• fix issue #1073: avoid calling __getattr__ on potential plugin objects. This fixes an incompatibility with pytest-django. Thanks Andreas Pelme, Bruno Oliveira and Ronny Pfannschmidt for contributing and Holger Krekel for the fix.
• Fix issue #704: handle versionconflict during plugin loading more gracefully. Thanks Bruno Oliveira for the PR.
• Fix issue #1064: “”–junitxml” regression when used with the “pytest-xdist” plugin, with test reports being assigned to the wrong tests. Thanks Daniel Grunwald for the report and Bruno Oliveira for the PR.
• (experimental) adapt more SEMVER style versioning and change meaning of master branch in git repo: “master” branch now keeps the bugfixes, changes aimed for micro releases. “features” branch will only be released with minor or major pytest releases.
• Fix issue #766 by removing documentation references to distutils. Thanks Russel Winder.
• Fix issue #1030: now byte-strings are escaped to produce item node ids to make them always serializable. Thanks Andy Freeland for the report and Bruno Oliveira for the PR.
• Python 2: if unicode parametrized values are convertible to ascii, their ascii representation is used for the node id.
• Fix issue #411: Add __eq__ method to assertion comparison example. Thanks Ben Webb.
• Fix issue #653: deprecated_call can be used as context manager.
• fix issue 877: properly handle assertion explanations with non-ascii repr Thanks Mathieu Agopian for the report and Ronny Pfannschmidt for the PR.
• fix issue 1029: transform errors when writing cache values into pytest-warnings

2.8.0 (2015-09-18)

• new --lf and -ff options to run only the last failing tests or “failing tests first” from the last run. This functionality is provided through porting the formerly external pytest-cache plugin into pytest core. BACKWARD INCOMPAT: if you used pytest-cache’s functionality to persist data between test runs be aware that we don’t serialize sets anymore. Thanks Ronny Pfannschmidt for most of the merging work.

• “-r” option now accepts “a” to include all possible reports, similar to passing “fEsxXw” explicitly (issue960). Thanks Abhijeet Kasurde for the PR.

• avoid python3.5 deprecation warnings by introducing version specific inspection helpers, thanks Michael Droettboom.

• fix issue562: @nose.tools.istest now fully respected.

• fix issue934: when string comparison fails and a diff is too large to display without passing -vv, still show a few lines of the diff. Thanks Florian Bruhin for the report and Bruno Oliveira for the PR.

• fix issue736: Fix a bug where fixture params would be discarded when combined with parametrization markers. Thanks to Markus Unterwaditzer for the PR.

• fix issue710: introduce ALLOW_UNICODE doctest option: when enabled, the u prefix is stripped from unicode strings in expected doctest output. This allows doctests which use unicode to run in Python 2 and 3 unchanged. Thanks Jason R. Coombs for the report and Bruno Oliveira for the PR.

• parametrize now also generates meaningful test IDs for enum, regex and class objects (as opposed to class instances). Thanks to Florian Bruhin for the PR.

• Add ‘warns’ to assert that warnings are thrown (like ‘raises’). Thanks to Eric Hunsberger for the PR.

• Fix issue683: Do not apply an already applied mark. Thanks ojake for the PR.

• Deal with capturing failures better so fewer exceptions get lost to /dev/null. Thanks David Szotten for the PR.

• fix issue730: deprecate and warn about the –genscript option. Thanks Ronny Pfannschmidt for the report and Christian Pommranz for the PR.

• fix issue751: multiple parametrize with ids bug if it parametrizes class with two or more test methods. Thanks Sergey Chipiga for reporting and Jan Bednarik for PR.

• fix issue82: avoid loading conftest files from setup.cfg/pytest.ini/tox.ini files and upwards by default (–confcutdir can still be set to override this). Thanks Bruno Oliveira for the PR.

• fix issue768: docstrings found in python modules were not setting up session fixtures. Thanks Jason R. Coombs for reporting and Bruno Oliveira for the PR.

• added tmpdir_factory, a session-scoped fixture that can be used to create directories under the base temporary directory. Previously this object was installed as a _tmpdirhandler attribute of the config object, but now it is part of the official API and using config._tmpdirhandler is deprecated. Thanks Bruno Oliveira for the PR.

• fix issue808: pytest’s internal assertion rewrite hook now implements the optional PEP302 get_data API so tests can access data files next to them. Thanks xmo-odoo for request and example and Bruno Oliveira for the PR.

• rootdir and inifile are now displayed during usage errors to help users diagnose problems such as unexpected ini files which add unknown options being picked up by pytest. Thanks to Pavel Savchenko for bringing the problem to attention in #821 and Bruno Oliveira for the PR.

• Summary bar now is colored yellow for warning situations such as: all tests either were skipped or xpass/xfailed, or no tests were run at all (this is a partial fix for issue500).

• fix issue812: pytest now exits with status code 5 in situations where no tests were run at all, such as the directory given in the command line does not contain any tests or as result of a command line option filters all out all tests (-k for example). Thanks Eric Siegerman (issue812) and Bruno Oliveira for the PR.

• Summary bar now is colored yellow for warning situations such as: all tests either were skipped or xpass/xfailed, or no tests were run at all (related to issue500). Thanks Eric Siegerman.

• New testpaths ini option: list of directories to search for tests when executing pytest from the root directory. This can be used to speed up test collection when a project has well specified directories for tests, being usually more practical than configuring norecursedirs for all directories that do not contain tests. Thanks to Adrian for idea (#694) and Bruno Oliveira for the PR.

• fix issue713: JUnit XML reports for doctest failures. Thanks Punyashloka Biswal.

• fix issue970: internal pytest warnings now appear as “pytest-warnings” in the terminal instead of “warnings”, so it is clear for users that those warnings are from pytest and not from the builtin “warnings” module. Thanks Bruno Oliveira.

• Include setup and teardown in junitxml test durations. Thanks Janne Vanhala.

• fix issue735: assertion failures on debug versions of Python 3.4+

• new option --import-mode to allow to change test module importing behaviour to append to sys.path instead of prepending. This better allows to run test modules against installed versions of a package even if the package under test has the same import root. In this example:

  testing/__init__.py
  testing/test_pkg_under_test.py
  pkg_under_test/
  

  the tests will run against the installed version of pkg_under_test when --import-mode=append is used whereas by default they would always pick up the local version. Thanks Holger Krekel.

• pytester: add method TmpTestdir.delete_loaded_modules(), and call it from inline_run() to allow temporary modules to be reloaded. Thanks Eduardo Schettino.

• internally refactor pluginmanager API and code so that there is a clear distinction between a pytest-agnostic rather simple pluginmanager and the PytestPluginManager which adds a lot of behaviour, among it handling of the local conftest files. In terms of documented methods this is a backward compatible change but it might still break 3rd party plugins which relied on details like especially the pluginmanager.add_shutdown() API. Thanks Holger Krekel.

• pluginmanagement: introduce pytest.hookimpl and pytest.hookspec decorators for setting impl/spec specific parameters. This substitutes the previous now deprecated use of pytest.mark which is meant to contain markers for test functions only.

• write/refine docs for “writing plugins” which now have their own page and are separate from the “using/installing plugins`` page.

• fix issue732: properly unregister plugins from any hook calling sites allowing to have temporary plugins during test execution.

• deprecate and warn about __multicall__ argument in hook implementations. Use the hookwrapper mechanism instead already introduced with pytest-2.7.

• speed up pytest’s own test suite considerably by using inprocess tests by default (testrun can be modified with –runpytest=subprocess to create subprocesses in many places instead). The main APIs to run pytest in a test is “runpytest()” or “runpytest_subprocess” and “runpytest_inprocess” if you need a particular way of running the test. In all cases you get back a RunResult but the inprocess one will also have a “reprec” attribute with the recorded events/reports.

• fix monkeypatch.setattr(“x.y”, raising=False) to actually not raise if “y” is not a pre-existing attribute. Thanks Florian Bruhin.

• fix issue741: make running output from testdir.run copy/pasteable Thanks Bruno Oliveira.

• add a new --noconftest argument which ignores all conftest.py files.

• add file and line attributes to JUnit-XML output.

• fix issue890: changed extension of all documentation files from txt to rst. Thanks to Abhijeet for the PR.

• fix issue714: add ability to apply indirect=True parameter on particular argnames. Thanks Elizaveta239.

• fix issue890: changed extension of all documentation files from txt to rst. Thanks to Abhijeet for the PR.

• fix issue957: “# doctest: SKIP” option will now register doctests as SKIPPED rather than PASSED. Thanks Thomas Grainger for the report and Bruno Oliveira for the PR.

• issue951: add new record_xml_property fixture, that supports logging additional information on xml output. Thanks David Diaz for the PR.

• issue949: paths after normal options (for example -s, -v, etc) are now properly used to discover rootdir and ini files. Thanks Peter Lauri for the report and Bruno Oliveira for the PR.

2.7.3 (2015-09-15)

• Allow ‘dev’, ‘rc’, or other non-integer version strings in importorskip. Thanks to Eric Hunsberger for the PR.
• fix issue856: consider –color parameter in all outputs (for example –fixtures). Thanks Barney Gale for the report and Bruno Oliveira for the PR.
• fix issue855: passing str objects as plugins argument to pytest.main is now interpreted as a module name to be imported and registered as a plugin, instead of silently having no effect. Thanks xmo-odoo for the report and Bruno Oliveira for the PR.
• fix issue744: fix for ast.Call changes in Python 3.5+. Thanks Guido van Rossum, Matthias Bussonnier, Stefan Zimmermann and Thomas Kluyver.
• fix issue842: applying markers in classes no longer propagate this markers to superclasses which also have markers. Thanks xmo-odoo for the report and Bruno Oliveira for the PR.
• preserve warning functions after call to pytest.deprecated_call. Thanks Pieter Mulder for PR.
• fix issue854: autouse yield_fixtures defined as class members of unittest.TestCase subclasses now work as expected. Thannks xmo-odoo for the report and Bruno Oliveira for the PR.
• fix issue833: –fixtures now shows all fixtures of collected test files, instead of just the fixtures declared on the first one. Thanks Florian Bruhin for reporting and Bruno Oliveira for the PR.
• fix issue863: skipped tests now report the correct reason when a skip/xfail condition is met when using multiple markers. Thanks Raphael Pierzina for reporting and Bruno Oliveira for the PR.
• optimized tmpdir fixture initialization, which should make test sessions faster (specially when using pytest-xdist). The only visible effect is that now pytest uses a subdirectory in the $TEMP directory for all directories created by this fixture (defaults to $TEMP/pytest-$USER). Thanks Bruno Oliveira for the PR.

2.7.2 (2015-06-23)

• fix issue767: pytest.raises value attribute does not contain the exception instance on Python 2.6. Thanks Eric Siegerman for providing the test case and Bruno Oliveira for PR.
• Automatically create directory for junitxml and results log. Thanks Aron Curzon.
• fix issue713: JUnit XML reports for doctest failures. Thanks Punyashloka Biswal.
• fix issue735: assertion failures on debug versions of Python 3.4+ Thanks Benjamin Peterson.
• fix issue114: skipif marker reports to internal skipping plugin; Thanks Floris Bruynooghe for reporting and Bruno Oliveira for the PR.
• fix issue748: unittest.SkipTest reports to internal pytest unittest plugin. Thanks Thomas De Schampheleire for reporting and Bruno Oliveira for the PR.
• fix issue718: failed to create representation of sets containing unsortable elements in python 2. Thanks Edison Gustavo Muenz.
• fix issue756, fix issue752 (and similar issues): depend on py-1.4.29 which has a refined algorithm for traceback generation.

2.7.1 (2015-05-19)

• fix issue731: do not get confused by the braces which may be present and unbalanced in an object’s repr while collapsing False explanations. Thanks Carl Meyer for the report and test case.
• fix issue553: properly handling inspect.getsourcelines failures in FixtureLookupError which would lead to an internal error, obfuscating the original problem. Thanks talljosh for initial diagnose/patch and Bruno Oliveira for final patch.
• fix issue660: properly report scope-mismatch-access errors independently from ordering of fixture arguments. Also avoid the pytest internal traceback which does not provide information to the user. Thanks Holger Krekel.
• streamlined and documented release process. Also all versions (in setup.py and documentation generation) are now read from _pytest/__init__.py. Thanks Holger Krekel.
• fixed docs to remove the notion that yield-fixtures are experimental. They are here to stay :) Thanks Bruno Oliveira.
• Support building wheels by using environment markers for the requirements. Thanks Ionel Maries Cristian.
• fixed regression to 2.6.4 which surfaced e.g. in lost stdout capture printing when tests raised SystemExit. Thanks Holger Krekel.
• reintroduced _pytest fixture of the pytester plugin which is used at least by pytest-xdist.

2.7.0 (2015-03-26)

• fix issue435: make reload() work when assert rewriting is active. Thanks Daniel Hahler.
• fix issue616: conftest.py files and their contained fixtures are now properly considered for visibility, independently from the exact current working directory and test arguments that are used. Many thanks to Eric Siegerman and his PR235 which contains systematic tests for conftest visibility and now passes. This change also introduces the concept of a rootdir which is printed as a new pytest header and documented in the pytest customize web page.
• change reporting of “diverted” tests, i.e. tests that are collected in one file but actually come from another (e.g. when tests in a test class come from a base class in a different file). We now show the nodeid and indicate via a postfix the other file.
• add ability to set command line options by environment variable PYTEST_ADDOPTS.
• added documentation on the new pytest-dev teams on bitbucket and github. See https://pytest.org/en/latest/contributing.html . Thanks to Anatoly for pushing and initial work on this.
• fix issue650: new option --docttest-ignore-import-errors which will turn import errors in doctests into skips. Thanks Charles Cloud for the complete PR.
• fix issue655: work around different ways that cause python2/3 to leak sys.exc_info into fixtures/tests causing failures in 3rd party code
• fix issue615: assertion rewriting did not correctly escape % signs when formatting boolean operations, which tripped over mixing booleans with modulo operators. Thanks to Tom Viner for the report, triaging and fix.
• implement issue351: add ability to specify parametrize ids as a callable to generate custom test ids. Thanks Brianna Laugher for the idea and implementation.
• introduce and document new hookwrapper mechanism useful for plugins which want to wrap the execution of certain hooks for their purposes. This supersedes the undocumented __multicall__ protocol which pytest itself and some external plugins use. Note that pytest-2.8 is scheduled to drop supporting the old __multicall__ and only support the hookwrapper protocol.
• majorly speed up invocation of plugin hooks
• use hookwrapper mechanism in builtin pytest plugins.
• add a doctest ini option for doctest flags, thanks Holger Peters.
• add note to docs that if you want to mark a parameter and the parameter is a callable, you also need to pass in a reason to disambiguate it from the “decorator” case. Thanks Tom Viner.
• “python_classes” and “python_functions” options now support glob-patterns for test discovery, as discussed in issue600. Thanks Ldiary Translations.
• allow to override parametrized fixtures with non-parametrized ones and vice versa (bubenkoff).
• fix issue463: raise specific error for ‘parameterize’ misspelling (pfctdayelise).
• On failure, the sys.last_value, sys.last_type and sys.last_traceback are set, so that a user can inspect the error via postmortem debugging (almarklein).

2.6.4 (2014-10-24)

• Improve assertion failure reporting on iterables, by using ndiff and pprint.
• removed outdated japanese docs from source tree.
• docs for “pytest_addhooks” hook. Thanks Bruno Oliveira.
• updated plugin index docs. Thanks Bruno Oliveira.
• fix issue557: with “-k” we only allow the old style “-” for negation at the beginning of strings and even that is deprecated. Use “not” instead. This should allow to pick parametrized tests where “-” appeared in the parameter.
• fix issue604: Escape % character in the assertion message.
• fix issue620: add explanation in the –genscript target about what the binary blob means. Thanks Dinu Gherman.
• fix issue614: fixed pastebin support.
• fix issue620: add explanation in the –genscript target about what the binary blob means. Thanks Dinu Gherman.
• fix issue614: fixed pastebin support.

2.6.3 (2014-09-24)

• fix issue575: xunit-xml was reporting collection errors as failures instead of errors, thanks Oleg Sinyavskiy.
• fix issue582: fix setuptools example, thanks Laszlo Papp and Ronny Pfannschmidt.
• Fix infinite recursion bug when pickling capture.EncodedFile, thanks Uwe Schmitt.
• fix issue589: fix bad interaction with numpy and others when showing exceptions. Check for precise “maximum recursion depth exceed” exception instead of presuming any RuntimeError is that one (implemented in py dep). Thanks Charles Cloud for analysing the issue.
• fix conftest related fixture visibility issue: when running with a CWD outside of a test package pytest would get fixture discovery wrong. Thanks to Wolfgang Schnerring for figuring out a reproducible example.
• Introduce pytest_enter_pdb hook (needed e.g. by pytest_timeout to cancel the timeout when interactively entering pdb). Thanks Wolfgang Schnerring.
• check xfail/skip also with non-python function test items. Thanks Floris Bruynooghe.

2.6.2 (2014-09-05)

• Added function pytest.freeze_includes(), which makes it easy to embed pytest into executables using tools like cx_freeze. See docs for examples and rationale. Thanks Bruno Oliveira.
• Improve assertion rewriting cache invalidation precision.
• fixed issue561: adapt autouse fixture example for python3.
• fixed issue453: assertion rewriting issue with __repr__ containing “n{“, “n}” and “n~”.
• fix issue560: correctly display code if an “else:” or “finally:” is followed by statements on the same line.
• Fix example in monkeypatch documentation, thanks t-8ch.
• fix issue572: correct tmpdir doc example for python3.
• Do not mark as universal wheel because Python 2.6 is different from other builds due to the extra argparse dependency. Fixes issue566. Thanks sontek.
• Implement issue549: user-provided assertion messages now no longer replace the py.test introspection message but are shown in addition to them.

2.6.1 (2014-08-07)

• No longer show line numbers in the –verbose output, the output is now purely the nodeid. The line number is still shown in failure reports. Thanks Floris Bruynooghe.
• fix issue437 where assertion rewriting could cause pytest-xdist slaves to collect different tests. Thanks Bruno Oliveira.
• fix issue555: add “errors” attribute to capture-streams to satisfy some distutils and possibly other code accessing sys.stdout.errors.
• fix issue547 capsys/capfd also work when output capturing (“-s”) is disabled.
• address issue170: allow pytest.mark.xfail(…) to specify expected exceptions via an optional “raises=EXC” argument where EXC can be a single exception or a tuple of exception classes. Thanks David Mohr for the complete PR.
• fix integration of pytest with unittest.mock.patch decorator when it uses the “new” argument. Thanks Nicolas Delaby for test and PR.
• fix issue with detecting conftest files if the arguments contain “::” node id specifications (copy pasted from “-v” output)
• fix issue544 by only removing “@NUM” at the end of “::” separated parts and if the part has a “.py” extension
• don’t use py.std import helper, rather import things directly. Thanks Bruno Oliveira.

2.6

• Cache exceptions from fixtures according to their scope (issue 467).
• fix issue537: Avoid importing old assertion reinterpretation code by default.
• fix issue364: shorten and enhance tracebacks representation by default. The new “–tb=auto” option (default) will only display long tracebacks for the first and last entry. You can get the old behaviour of printing all entries as long entries with “–tb=long”. Also short entries by default are now printed very similarly to “–tb=native” ones.
• fix issue514: teach assertion reinterpretation about private class attributes
• change -v output to include full node IDs of tests. Users can copy a node ID from a test run, including line number, and use it as a positional argument in order to run only a single test.
• fix issue 475: fail early and comprehensible if calling pytest.raises with wrong exception type.
• fix issue516: tell in getting-started about current dependencies.
• cleanup setup.py a bit and specify supported versions. Thanks Jurko Gospodnetic for the PR.
• change XPASS colour to yellow rather then red when tests are run with -v.
• fix issue473: work around mock putting an unbound method into a class dict when double-patching.
• fix issue498: if a fixture finalizer fails, make sure that the fixture is still invalidated.
• fix issue453: the result of the pytest_assertrepr_compare hook now gets it’s newlines escaped so that format_exception does not blow up.
• internal new warning system: pytest will now produce warnings when it detects oddities in your test collection or execution. Warnings are ultimately sent to a new pytest_logwarning hook which is currently only implemented by the terminal plugin which displays warnings in the summary line and shows more details when -rw (report on warnings) is specified.
• change skips into warnings for test classes with an __init__ and callables in test modules which look like a test but are not functions.
• fix issue436: improved finding of initial conftest files from command line arguments by using the result of parse_known_args rather than the previous flaky heuristics. Thanks Marc Abramowitz for tests and initial fixing approaches in this area.
• fix issue #479: properly handle nose/unittest(2) SkipTest exceptions during collection/loading of test modules. Thanks to Marc Schlaich for the complete PR.
• fix issue490: include pytest_load_initial_conftests in documentation and improve docstring.
• fix issue472: clarify that pytest.config.getvalue() cannot work if it’s triggered ahead of command line parsing.
• merge PR123: improved integration with mock.patch decorator on tests.
• fix issue412: messing with stdout/stderr FD-level streams is now captured without crashes.
• fix issue483: trial/py33 works now properly. Thanks Daniel Grana for PR.
• improve example for pytest integration with “python setup.py test” which now has a generic “-a” or “–pytest-args” option where you can pass additional options as a quoted string. Thanks Trevor Bekolay.
• simplified internal capturing mechanism and made it more robust against tests or setups changing FD1/FD2, also better integrated now with pytest.pdb() in single tests.
• improvements to pytest’s own test-suite leakage detection, courtesy of PRs from Marc Abramowitz
• fix issue492: avoid leak in test_writeorg. Thanks Marc Abramowitz.
• fix issue493: don’t run tests in doc directory with python setup.py test (use tox -e doctesting for that)
• fix issue486: better reporting and handling of early conftest loading failures
• some cleanup and simplification of internal conftest handling.
• work a bit harder to break reference cycles when catching exceptions. Thanks Jurko Gospodnetic.
• fix issue443: fix skip examples to use proper comparison. Thanks Alex Groenholm.
• support nose-style __test__ attribute on modules, classes and functions, including unittest-style Classes. If set to False, the test will not be collected.
• fix issue512: show “<notset>” for arguments which might not be set in monkeypatch plugin. Improves output in documentation.

2.5.2 (2014-01-29)

• fix issue409 – better interoperate with cx_freeze by not trying to import from collections.abc which causes problems for py27/cx_freeze. Thanks Wolfgang L. for reporting and tracking it down.
• fixed docs and code to use “pytest” instead of “py.test” almost everywhere. Thanks Jurko Gospodnetic for the complete PR.
• fix issue425: mention at end of “py.test -h” that –markers and –fixtures work according to specified test path (or current dir)
• fix issue413: exceptions with unicode attributes are now printed correctly also on python2 and with pytest-xdist runs. (the fix requires py-1.4.20)
• copy, cleanup and integrate py.io capture from pylib 1.4.20.dev2 (rev 13d9af95547e)
• address issue416: clarify docs as to conftest.py loading semantics
• fix issue429: comparing byte strings with non-ascii chars in assert expressions now work better. Thanks Floris Bruynooghe.
• make capfd/capsys.capture private, its unused and shouldn’t be exposed

2.5.1 (2013-12-17)

• merge new documentation styling PR from Tobias Bieniek.
• fix issue403: allow parametrize of multiple same-name functions within a collection node. Thanks Andreas Kloeckner and Alex Gaynor for reporting and analysis.
• Allow parameterized fixtures to specify the ID of the parameters by adding an ids argument to pytest.fixture() and pytest.yield_fixture(). Thanks Floris Bruynooghe.
• fix issue404 by always using the binary xml escape in the junitxml plugin. Thanks Ronny Pfannschmidt.
• fix issue407: fix addoption docstring to point to argparse instead of optparse. Thanks Daniel D. Wright.

2.5.0 (2013-12-12)

• dropped python2.5 from automated release testing of pytest itself which means it’s probably going to break soon (but still works with this release we believe).

• simplified and fixed implementation for calling finalizers when parametrized fixtures or function arguments are involved. finalization is now performed lazily at setup time instead of in the “teardown phase”. While this might sound odd at first, it helps to ensure that we are correctly handling setup/teardown even in complex code. User-level code should not be affected unless it’s implementing the pytest_runtest_teardown hook and expecting certain fixture instances are torn down within (very unlikely and would have been unreliable anyway).

• PR90: add –color=yes|no|auto option to force terminal coloring mode (“auto” is default). Thanks Marc Abramowitz.

• fix issue319 - correctly show unicode in assertion errors. Many thanks to Floris Bruynooghe for the complete PR. Also means we depend on py>=1.4.19 now.

• fix issue396 - correctly sort and finalize class-scoped parametrized tests independently from number of methods on the class.

• refix issue323 in a better way – parametrization should now never cause Runtime Recursion errors because the underlying algorithm for re-ordering tests per-scope/per-fixture is not recursive anymore (it was tail-call recursive before which could lead to problems for more than >966 non-function scoped parameters).

• fix issue290 - there is preliminary support now for parametrizing with repeated same values (sometimes useful to test if calling a second time works as with the first time).

• close issue240 - document precisely how pytest module importing works, discuss the two common test directory layouts, and how it interacts with PEP420-namespace packages.

• fix issue246 fix finalizer order to be LIFO on independent fixtures depending on a parametrized higher-than-function scoped fixture. (was quite some effort so please bear with the complexity of this sentence :) Thanks Ralph Schmitt for the precise failure example.

• fix issue244 by implementing special index for parameters to only use indices for paramentrized test ids

• fix issue287 by running all finalizers but saving the exception from the first failing finalizer and re-raising it so teardown will still have failed. We reraise the first failing exception because it might be the cause for other finalizers to fail.

• fix ordering when mock.patch or other standard decorator-wrappings are used with test methods. This fixues issue346 and should help with random “xdist” collection failures. Thanks to Ronny Pfannschmidt and Donald Stufft for helping to isolate it.

• fix issue357 - special case “-k” expressions to allow for filtering with simple strings that are not valid python expressions. Examples: “-k 1.3” matches all tests parametrized with 1.3. “-k None” filters all tests that have “None” in their name and conversely “-k ‘not None’”. Previously these examples would raise syntax errors.

• fix issue384 by removing the trial support code since the unittest compat enhancements allow trial to handle it on its own

• don’t hide an ImportError when importing a plugin produces one. fixes issue375.

• fix issue275 - allow usefixtures and autouse fixtures for running doctest text files.

• fix issue380 by making –resultlog only rely on longrepr instead of the “reprcrash” attribute which only exists sometimes.

• address issue122: allow @pytest.fixture(params=iterator) by exploding into a list early on.

• fix pexpect-3.0 compatibility for pytest’s own tests. (fixes issue386)

• allow nested parametrize-value markers, thanks James Lan for the PR.

• fix unicode handling with new monkeypatch.setattr(import_path, value) API. Thanks Rob Dennis. Fixes issue371.

• fix unicode handling with junitxml, fixes issue368.

• In assertion rewriting mode on Python 2, fix the detection of coding cookies. See issue #330.

• make “–runxfail” turn imperative pytest.xfail calls into no ops (it already did neutralize pytest.mark.xfail markers)

• refine pytest / pkg_resources interactions: The AssertionRewritingHook PEP302 compliant loader now registers itself with setuptools/pkg_resources properly so that the pkg_resources.resource_stream method works properly. Fixes issue366. Thanks for the investigations and full PR to Jason R. Coombs.

• pytestconfig fixture is now session-scoped as it is the same object during the whole test run. Fixes issue370.

• avoid one surprising case of marker malfunction/confusion:

  @pytest.mark.some(lambda arg: ...)
  def test_function():
  

  would not work correctly because pytest assumes @pytest.mark.some gets a function to be decorated already. We now at least detect if this arg is a lambda and thus the example will work. Thanks Alex Gaynor for bringing it up.

• xfail a test on pypy that checks wrong encoding/ascii (pypy does not error out). fixes issue385.

• internally make varnames() deal with classes’s __init__, although it’s not needed by pytest itself atm. Also fix caching. Fixes issue376.

• fix issue221 - handle importing of namespace-package with no __init__.py properly.

• refactor internal FixtureRequest handling to avoid monkeypatching. One of the positive user-facing effects is that the “request” object can now be used in closures.

• fixed version comparison in pytest.importskip(modname, minverstring)

• fix issue377 by clarifying in the nose-compat docs that pytest does not duplicate the unittest-API into the “plain” namespace.

• fix verbose reporting for @mock’d test functions

2.4.2 (2013-10-04)

• on Windows require colorama and a newer py lib so that py.io.TerminalWriter() now uses colorama instead of its own ctypes hacks. (fixes issue365) thanks Paul Moore for bringing it up.
• fix “-k” matching of tests where “repr” and “attr” and other names would cause wrong matches because of an internal implementation quirk (don’t ask) which is now properly implemented. fixes issue345.
• avoid tmpdir fixture to create too long filenames especially when parametrization is used (issue354)
• fix pytest-pep8 and pytest-flakes / pytest interactions (collection names in mark plugin was assuming an item always has a function which is not true for those plugins etc.) Thanks Andi Zeidler.
• introduce node.get_marker/node.add_marker API for plugins like pytest-pep8 and pytest-flakes to avoid the messy details of the node.keywords pseudo-dicts. Adapted docs.
• remove attempt to “dup” stdout at startup as it’s icky. the normal capturing should catch enough possibilities of tests messing up standard FDs.
• add pluginmanager.do_configure(config) as a link to config.do_configure() for plugin-compatibility

2.4.1 (2013-10-02)

• When using parser.addoption() unicode arguments to the “type” keyword should also be converted to the respective types. thanks Floris Bruynooghe, @dnozay. (fixes issue360 and issue362)
• fix dotted filename completion when using argcomplete thanks Anthon van der Neuth. (fixes issue361)
• fix regression when a 1-tuple (“arg”,) is used for specifying parametrization (the values of the parametrization were passed nested in a tuple). Thanks Donald Stufft.
• merge doc typo fixes, thanks Andy Dirnberger

2.4

known incompatibilities:

• if calling –genscript from python2.7 or above, you only get a standalone script which works on python2.7 or above. Use Python2.6 to also get a python2.5 compatible version.
• all xunit-style teardown methods (nose-style, pytest-style, unittest-style) will not be called if the corresponding setup method failed, see issue322 below.
• the pytest_plugin_unregister hook wasn’t ever properly called and there is no known implementation of the hook - so it got removed.
• pytest.fixture-decorated functions cannot be generators (i.e. use yield) anymore. This change might be reversed in 2.4.1 if it causes unforeseen real-life issues. However, you can always write and return an inner function/generator and change the fixture consumer to iterate over the returned generator. This change was done in lieu of the new pytest.yield_fixture decorator, see below.

new features:

• experimentally introduce a new pytest.yield_fixture decorator which accepts exactly the same parameters as pytest.fixture but mandates a yield statement instead of a return statement from fixture functions. This allows direct integration with “with-style” context managers in fixture functions and generally avoids registering of finalization callbacks in favour of treating the “after-yield” as teardown code. Thanks Andreas Pelme, Vladimir Keleshev, Floris Bruynooghe, Ronny Pfannschmidt and many others for discussions.

• allow boolean expression directly with skipif/xfail if a “reason” is also specified. Rework skipping documentation to recommend “condition as booleans” because it prevents surprises when importing markers between modules. Specifying conditions as strings will remain fully supported.

• reporting: color the last line red or green depending if failures/errors occurred or everything passed. thanks Christian Theunert.

• make “import pdb ; pdb.set_trace()” work natively wrt capturing (no “-s” needed anymore), making pytest.set_trace() a mere shortcut.

• fix issue181: –pdb now also works on collect errors (and on internal errors) . This was implemented by a slight internal refactoring and the introduction of a new hook pytest_exception_interact hook (see next item).

• fix issue341: introduce new experimental hook for IDEs/terminals to intercept debugging: pytest_exception_interact(node, call, report).

• new monkeypatch.setattr() variant to provide a shorter invocation for patching out classes/functions from modules:

  monkeypatch.setattr(“requests.get”, myfunc)

  will replace the “get” function of the “requests” module with myfunc.

• fix issue322: tearDownClass is not run if setUpClass failed. Thanks Mathieu Agopian for the initial fix. Also make all of pytest/nose finalizer mimic the same generic behaviour: if a setupX exists and fails, don’t run teardownX. This internally introduces a new method “node.addfinalizer()” helper which can only be called during the setup phase of a node.

• simplify pytest.mark.parametrize() signature: allow to pass a CSV-separated string to specify argnames. For example: pytest.mark.parametrize("input,expected",  [(1,2), (2,3)]) works as well as the previous: pytest.mark.parametrize(("input", "expected"), ...).

• add support for setUpModule/tearDownModule detection, thanks Brian Okken.

• integrate tab-completion on options through use of “argcomplete”. Thanks Anthon van der Neut for the PR.

• change option names to be hyphen-separated long options but keep the old spelling backward compatible. py.test -h will only show the hyphenated version, for example “–collect-only” but “–collectonly” will remain valid as well (for backward-compat reasons). Many thanks to Anthon van der Neut for the implementation and to Hynek Schlawack for pushing us.

• fix issue 308 - allow to mark/xfail/skip individual parameter sets when parametrizing. Thanks Brianna Laugher.

• call new experimental pytest_load_initial_conftests hook to allow 3rd party plugins to do something before a conftest is loaded.

Bug fixes:

• fix issue358 - capturing options are now parsed more properly by using a new parser.parse_known_args method.
• pytest now uses argparse instead of optparse (thanks Anthon) which means that “argparse” is added as a dependency if installing into python2.6 environments or below.
• fix issue333: fix a case of bad unittest/pytest hook interaction.
• PR27: correctly handle nose.SkipTest during collection. Thanks Antonio Cuni, Ronny Pfannschmidt.
• fix issue355: junitxml puts name=”pytest” attribute to testsuite tag.
• fix issue336: autouse fixture in plugins should work again.
• fix issue279: improve object comparisons on assertion failure for standard datatypes and recognise collections.abc. Thanks to Brianna Laugher and Mathieu Agopian.
• fix issue317: assertion rewriter support for the is_package method
• fix issue335: document py.code.ExceptionInfo() object returned from pytest.raises(), thanks Mathieu Agopian.
• remove implicit distribute_setup support from setup.py.
• fix issue305: ignore any problems when writing pyc files.
• SO-17664702: call fixture finalizers even if the fixture function partially failed (finalizers would not always be called before)
• fix issue320 - fix class scope for fixtures when mixed with module-level functions. Thanks Anatloy Bubenkoff.
• you can specify “-q” or “-qq” to get different levels of “quieter” reporting (thanks Katarzyna Jachim)
• fix issue300 - Fix order of conftest loading when starting py.test in a subdirectory.
• fix issue323 - sorting of many module-scoped arg parametrizations
• make sessionfinish hooks execute with the same cwd-context as at session start (helps fix plugin behaviour which write output files with relative path such as pytest-cov)
• fix issue316 - properly reference collection hooks in docs
• fix issue 306 - cleanup of -k/-m options to only match markers/test names/keywords respectively. Thanks Wouter van Ackooy.
• improved doctest counting for doctests in python modules – files without any doctest items will not show up anymore and doctest examples are counted as separate test items. thanks Danilo Bellini.
• fix issue245 by depending on the released py-1.4.14 which fixes py.io.dupfile to work with files with no mode. Thanks Jason R. Coombs.
• fix junitxml generation when test output contains control characters, addressing issue267, thanks Jaap Broekhuizen
• fix issue338: honor –tb style for setup/teardown errors as well. Thanks Maho.
• fix issue307 - use yaml.safe_load in example, thanks Mark Eichin.
• better parametrize error messages, thanks Brianna Laugher
• pytest_terminal_summary(terminalreporter) hooks can now use “.section(title)” and “.line(msg)” methods to print extra information at the end of a test run.

2.3.5 (2013-04-30)

• fix issue169: respect –tb=style with setup/teardown errors as well.
• never consider a fixture function for test function collection
• allow re-running of test items / helps to fix pytest-reruntests plugin and also help to keep less fixture/resource references alive
• put captured stdout/stderr into junitxml output even for passing tests (thanks Adam Goucher)
• Issue 265 - integrate nose setup/teardown with setupstate so it doesn’t try to teardown if it did not setup
• issue 271 - don’t write junitxml on slave nodes
• Issue 274 - don’t try to show full doctest example when doctest does not know the example location
• issue 280 - disable assertion rewriting on buggy CPython 2.6.0
• inject “getfixture()” helper to retrieve fixtures from doctests, thanks Andreas Zeidler
• issue 259 - when assertion rewriting, be consistent with the default source encoding of ASCII on Python 2
• issue 251 - report a skip instead of ignoring classes with init
• issue250 unicode/str mixes in parametrization names and values now works
• issue257, assertion-triggered compilation of source ending in a comment line doesn’t blow up in python2.5 (fixed through py>=1.4.13.dev6)
• fix –genscript option to generate standalone scripts that also work with python3.3 (importer ordering)
• issue171 - in assertion rewriting, show the repr of some global variables
• fix option help for “-k”
• move long description of distribution into README.rst
• improve docstring for metafunc.parametrize()
• fix bug where using capsys with pytest.set_trace() in a test function would break when looking at capsys.readouterr()
• allow to specify prefixes starting with “_” when customizing python_functions test discovery. (thanks Graham Horler)
• improve PYTEST_DEBUG tracing output by putting extra data on a new lines with additional indent
• ensure OutcomeExceptions like skip/fail have initialized exception attributes
• issue 260 - don’t use nose special setup on plain unittest cases
• fix issue134 - print the collect errors that prevent running specified test items
• fix issue266 - accept unicode in MarkEvaluator expressions

2.3.4 (2012-11-20)

• yielded test functions will now have autouse-fixtures active but cannot accept fixtures as funcargs - it’s anyway recommended to rather use the post-2.0 parametrize features instead of yield, see: http://pytest.org/en/latest/example/parametrize.html
• fix autouse-issue where autouse-fixtures would not be discovered if defined in an a/conftest.py file and tests in a/tests/test_some.py
• fix issue226 - LIFO ordering for fixture teardowns
• fix issue224 - invocations with >256 char arguments now work
• fix issue91 - add/discuss package/directory level setups in example
• allow to dynamically define markers via item.keywords[…]=assignment integrating with “-m” option
• make “-k” accept an expressions the same as with “-m” so that one can write: -k “name1 or name2” etc. This is a slight incompatibility if you used special syntax like “TestClass.test_method” which you now need to write as -k “TestClass and test_method” to match a certain method in a certain test class.

2.3.3 (2012-11-06)

• fix issue214 - parse modules that contain special objects like e. g. flask’s request object which blows up on getattr access if no request is active. thanks Thomas Waldmann.
• fix issue213 - allow to parametrize with values like numpy arrays that do not support an __eq__ operator
• fix issue215 - split test_python.org into multiple files
• fix issue148 - @unittest.skip on classes is now recognized and avoids calling setUpClass/tearDownClass, thanks Pavel Repin
• fix issue209 - reintroduce python2.4 support by depending on newer pylib which re-introduced statement-finding for pre-AST interpreters
• nose support: only call setup if it’s a callable, thanks Andrew Taumoefolau
• fix issue219 - add py2.4-3.3 classifiers to TROVE list
• in tracebacks ,* arg values are now shown next to normal arguments (thanks Manuel Jacob)
• fix issue217 - support mock.patch with pytest’s fixtures - note that you need either mock-1.0.1 or the python3.3 builtin unittest.mock.
• fix issue127 - improve documentation for pytest_addoption() and add a config.getoption(name) helper function for consistency.

2.3.2 (2012-10-25)

• fix issue208 and fix issue29 use new py version to avoid long pauses when printing tracebacks in long modules
• fix issue205 - conftests in subdirs customizing pytest_pycollect_makemodule and pytest_pycollect_makeitem now work properly
• fix teardown-ordering for parametrized setups
• fix issue127 - better documentation for pytest_addoption and related objects.
• fix unittest behaviour: TestCase.runtest only called if there are test methods defined
• improve trial support: don’t collect its empty unittest.TestCase.runTest() method
• “python setup.py test” now works with pytest itself
• fix/improve internal/packaging related bits:
  • exception message check of test_nose.py now passes on python33 as well
  • issue206 - fix test_assertrewrite.py to work when a global PYTHONDONTWRITEBYTECODE=1 is present
  • add tox.ini to pytest distribution so that ignore-dirs and others config bits are properly distributed for maintainers who run pytest-own tests

2.3.1 (2012-10-20)

• fix issue202 - fix regression: using “self” from fixture functions now works as expected (it’s the same “self” instance that a test method which uses the fixture sees)
• skip pexpect using tests (test_pdb.py mostly) on freebsd* systems due to pexpect not supporting it properly (hanging)
• link to web pages from –markers output which provides help for pytest.mark.* usage.

2.3.0 (2012-10-19)

• fix issue202 - better automatic names for parametrized test functions
• fix issue139 - introduce @pytest.fixture which allows direct scoping and parametrization of funcarg factories.
• fix issue198 - conftest fixtures were not found on windows32 in some circumstances with nested directory structures due to path manipulation issues
• fix issue193 skip test functions with were parametrized with empty parameter sets
• fix python3.3 compat, mostly reporting bits that previously depended on dict ordering
• introduce re-ordering of tests by resource and parametrization setup which takes precedence to the usual file-ordering
• fix issue185 monkeypatching time.time does not cause pytest to fail
• fix issue172 duplicate call of pytest.fixture decoratored setup_module functions
• fix junitxml=path construction so that if tests change the current working directory and the path is a relative path it is constructed correctly from the original current working dir.
• fix “python setup.py test” example to cause a proper “errno” return
• fix issue165 - fix broken doc links and mention stackoverflow for FAQ
• catch unicode-issues when writing failure representations to terminal to prevent the whole session from crashing
• fix xfail/skip confusion: a skip-mark or an imperative pytest.skip will now take precedence before xfail-markers because we can’t determine xfail/xpass status in case of a skip. see also: http://stackoverflow.com/questions/11105828/in-py-test-when-i-explicitly-skip-a-test-that-is-marked-as-xfail-how-can-i-get
• always report installed 3rd party plugins in the header of a test run
• fix issue160: a failing setup of an xfail-marked tests should be reported as xfail (not xpass)
• fix issue128: show captured output when capsys/capfd are used
• fix issue179: properly show the dependency chain of factories
• pluginmanager.register(…) now raises ValueError if the plugin has been already registered or the name is taken
• fix issue159: improve http://pytest.org/en/latest/faq.html especially with respect to the “magic” history, also mention pytest-django, trial and unittest integration.
• make request.keywords and node.keywords writable. All descendant collection nodes will see keyword values. Keywords are dictionaries containing markers and other info.
• fix issue 178: xml binary escapes are now wrapped in py.xml.raw
• fix issue 176: correctly catch the builtin AssertionError even when we replaced AssertionError with a subclass on the python level
• factory discovery no longer fails with magic global callables that provide no sane __code__ object (mock.call for example)
• fix issue 182: testdir.inprocess_run now considers passed plugins

• fix issue 188: ensure sys.exc_info is clear on python2

  before calling into a test

• fix issue 191: add unittest TestCase runTest method support
• fix issue 156: monkeypatch correctly handles class level descriptors
• reporting refinements:
  • pytest_report_header now receives a “startdir” so that you can use startdir.bestrelpath(yourpath) to show nice relative path
  • allow plugins to implement both pytest_report_header and pytest_sessionstart (sessionstart is invoked first).
  • don’t show deselected reason line if there is none
  • py.test -vv will show all of assert comparisons instead of truncating

2.2.4 (2012-05-22)

• fix error message for rewritten assertions involving the % operator
• fix issue 126: correctly match all invalid xml characters for junitxml binary escape
• fix issue with unittest: now @unittest.expectedFailure markers should be processed correctly (you can also use @pytest.mark markers)
• document integration with the extended distribute/setuptools test commands
• fix issue 140: properly get the real functions of bound classmethods for setup/teardown_class
• fix issue #141: switch from the deceased paste.pocoo.org to bpaste.net
• fix issue #143: call unconfigure/sessionfinish always when configure/sessionstart where called
• fix issue #144: better mangle test ids to junitxml classnames
• upgrade distribute_setup.py to 0.6.27

2.2.3 (2012-02-05)

• fix uploaded package to only include necessary files

2.2.2 (2012-02-05)

• fix issue101: wrong args to unittest.TestCase test function now produce better output
• fix issue102: report more useful errors and hints for when a test directory was renamed and some pyc/__pycache__ remain
• fix issue106: allow parametrize to be applied multiple times e.g. from module, class and at function level.
• fix issue107: actually perform session scope finalization
• don’t check in parametrize if indirect parameters are funcarg names
• add chdir method to monkeypatch funcarg
• fix crash resulting from calling monkeypatch undo a second time
• fix issue115: make –collectonly robust against early failure (missing files/directories)
• “-qq –collectonly” now shows only files and the number of tests in them
• “-q –collectonly” now shows test ids
• allow adding of attributes to test reports such that it also works with distributed testing (no upgrade of pytest-xdist needed)

2.2.1 (2011-12-16)

• fix issue99 (in pytest and py) internallerrors with resultlog now produce better output - fixed by normalizing pytest_internalerror input arguments.
• fix issue97 / traceback issues (in pytest and py) improve traceback output in conjunction with jinja2 and cython which hack tracebacks
• fix issue93 (in pytest and pytest-xdist) avoid “delayed teardowns”: the final test in a test node will now run its teardown directly instead of waiting for the end of the session. Thanks Dave Hunt for the good reporting and feedback. The pytest_runtest_protocol as well as the pytest_runtest_teardown hooks now have “nextitem” available which will be None indicating the end of the test run.
• fix collection crash due to unknown-source collected items, thanks to Ralf Schmitt (fixed by depending on a more recent pylib)

2.2.0 (2011-11-18)

• fix issue90: introduce eager tearing down of test items so that teardown function are called earlier.
• add an all-powerful metafunc.parametrize function which allows to parametrize test function arguments in multiple steps and therefore from independent plugins and places.
• add a @pytest.mark.parametrize helper which allows to easily call a test function with different argument values
• Add examples to the “parametrize” example page, including a quick port of Test scenarios and the new parametrize function and decorator.
• introduce registration for “pytest.mark.*” helpers via ini-files or through plugin hooks. Also introduce a “–strict” option which will treat unregistered markers as errors allowing to avoid typos and maintain a well described set of markers for your test suite. See exaples at http://pytest.org/en/latest/mark.html and its links.
• issue50: introduce “-m marker” option to select tests based on markers (this is a stricter and more predictable version of ‘-k’ in that “-m” only matches complete markers and has more obvious rules for and/or semantics.
• new feature to help optimizing the speed of your tests: –durations=N option for displaying N slowest test calls and setup/teardown methods.
• fix issue87: –pastebin now works with python3
• fix issue89: –pdb with unexpected exceptions in doctest work more sensibly
• fix and cleanup pytest’s own test suite to not leak FDs
• fix issue83: link to generated funcarg list
• fix issue74: pyarg module names are now checked against imp.find_module false positives
• fix compatibility with twisted/trial-11.1.0 use cases
• simplify Node.listchain
• simplify junitxml output code by relying on py.xml
• add support for skip properties on unittest classes and functions

2.1.3 (2011-10-18)

• fix issue79: assertion rewriting failed on some comparisons in boolops
• correctly handle zero length arguments (a la pytest ‘’)
• fix issue67 / junitxml now contains correct test durations, thanks ronny
• fix issue75 / skipping test failure on jython
• fix issue77 / Allow assertrepr_compare hook to apply to a subset of tests

2.1.2 (2011-09-24)

• fix assertion rewriting on files with windows newlines on some Python versions
• refine test discovery by package/module name (–pyargs), thanks Florian Mayer
• fix issue69 / assertion rewriting fixed on some boolean operations
• fix issue68 / packages now work with assertion rewriting
• fix issue66: use different assertion rewriting caches when the -O option is passed
• don’t try assertion rewriting on Jython, use reinterp

2.1.1

• fix issue64 / pytest.set_trace now works within pytest_generate_tests hooks
• fix issue60 / fix error conditions involving the creation of __pycache__
• fix issue63 / assertion rewriting on inserts involving strings containing ‘%’
• fix assertion rewriting on calls with a ** arg
• don’t cache rewritten modules if bytecode generation is disabled
• fix assertion rewriting in read-only directories
• fix issue59: provide system-out/err tags for junitxml output
• fix issue61: assertion rewriting on boolean operations with 3 or more operands
• you can now build a man page with “cd doc ; make man”

2.1.0 (2011-07-09)

• fix issue53 call nosestyle setup functions with correct ordering
• fix issue58 and issue59: new assertion code fixes
• merge Benjamin’s assertionrewrite branch: now assertions for test modules on python 2.6 and above are done by rewriting the AST and saving the pyc file before the test module is imported. see doc/assert.txt for more info.
• fix issue43: improve doctests with better traceback reporting on unexpected exceptions
• fix issue47: timing output in junitxml for test cases is now correct
• fix issue48: typo in MarkInfo repr leading to exception
• fix issue49: avoid confusing error when initizaliation partially fails
• fix issue44: env/username expansion for junitxml file path
• show releaselevel information in test runs for pypy
• reworked doc pages for better navigation and PDF generation
• report KeyboardInterrupt even if interrupted during session startup
• fix issue 35 - provide PDF doc version and download link from index page

2.0.3 (2011-05-11)

• fix issue38: nicer tracebacks on calls to hooks, particularly early configure/sessionstart ones
• fix missing skip reason/meta information in junitxml files, reported via http://lists.idyll.org/pipermail/testing-in-python/2011-March/003928.html
• fix issue34: avoid collection failure with “test” prefixed classes deriving from object.
• don’t require zlib (and other libs) for genscript plugin without –genscript actually being used.
• speed up skips (by not doing a full traceback representation internally)
• fix issue37: avoid invalid characters in junitxml’s output

2.0.2 (2011-03-09)

• tackle issue32 - speed up test runs of very quick test functions by reducing the relative overhead

• fix issue30 - extended xfail/skipif handling and improved reporting. If you have a syntax error in your skip/xfail expressions you now get nice error reports.

  Also you can now access module globals from xfail/skipif expressions so that this for example works now:

  import pytest
  import mymodule
  @pytest.mark.skipif("mymodule.__version__[0] == "1")
  def test_function():
      pass
  

  This will not run the test function if the module’s version string does not start with a “1”. Note that specifying a string instead of a boolean expressions allows py.test to report meaningful information when summarizing a test run as to what conditions lead to skipping (or xfail-ing) tests.

• fix issue28 - setup_method and pytest_generate_tests work together The setup_method fixture method now gets called also for test function invocations generated from the pytest_generate_tests hook.

• fix issue27 - collectonly and keyword-selection (-k) now work together Also, if you do “py.test –collectonly -q” you now get a flat list of test ids that you can use to paste to the py.test commandline in order to execute a particular test.

• fix issue25 avoid reported problems with –pdb and python3.2/encodings output

• fix issue23 - tmpdir argument now works on Python3.2 and WindowsXP Starting with Python3.2 os.symlink may be supported. By requiring a newer py lib version the py.path.local() implementation acknowledges this.

• fixed typos in the docs (thanks Victor Garcia, Brianna Laugher) and particular thanks to Laura Creighton who also reviewed parts of the documentation.

• fix slightly wrong output of verbose progress reporting for classes (thanks Amaury)

• more precise (avoiding of) deprecation warnings for node.Class|Function accesses

• avoid std unittest assertion helper code in tracebacks (thanks Ronny)

2.0.1 (2011-02-07)

• refine and unify initial capturing so that it works nicely even if the logging module is used on an early-loaded conftest.py file or plugin.
• allow to omit “()” in test ids to allow for uniform test ids as produced by Alfredo’s nice pytest.vim plugin.
• fix issue12 - show plugin versions with “–version” and “–traceconfig” and also document how to add extra information to reporting test header
• fix issue17 (import-* reporting issue on python3) by requiring py>1.4.0 (1.4.1 is going to include it)
• fix issue10 (numpy arrays truth checking) by refining assertion interpretation in py lib
• fix issue15: make nose compatibility tests compatible with python3 (now that nose-1.0 supports python3)
• remove somewhat surprising “same-conftest” detection because it ignores conftest.py when they appear in several subdirs.
• improve assertions (“not in”), thanks Floris Bruynooghe
• improve behaviour/warnings when running on top of “python -OO” (assertions and docstrings are turned off, leading to potential false positives)
• introduce a pytest_cmdline_processargs(args) hook to allow dynamic computation of command line arguments. This fixes a regression because py.test prior to 2.0 allowed to set command line options from conftest.py files which so far pytest-2.0 only allowed from ini-files now.
• fix issue7: assert failures in doctest modules. unexpected failures in doctests will not generally show nicer, i.e. within the doctest failing context.
• fix issue9: setup/teardown functions for an xfail-marked test will report as xfail if they fail but report as normally passing (not xpassing) if they succeed. This only is true for “direct” setup/teardown invocations because teardown_class/ teardown_module cannot closely relate to a single test.
• fix issue14: no logging errors at process exit
• refinements to “collecting” output on non-ttys
• refine internal plugin registration and –traceconfig output
• introduce a mechanism to prevent/unregister plugins from the command line, see http://pytest.org/en/latest/plugins.html#cmdunregister
• activate resultlog plugin by default
• fix regression wrt yielded tests which due to the collection-before-running semantics were not setup as with pytest 1.3.4. Note, however, that the recommended and much cleaner way to do test parametraization remains the “pytest_generate_tests” mechanism, see the docs.

2.0.0 (2010-11-25)

• pytest-2.0 is now its own package and depends on pylib-2.0
• new ability: python -m pytest / python -m pytest.main ability
• new python invocation: pytest.main(args, plugins) to load some custom plugins early.
• try harder to run unittest test suites in a more compatible manner by deferring setup/teardown semantics to the unittest package. also work harder to run twisted/trial and Django tests which should now basically work by default.
• introduce a new way to set config options via ini-style files, by default setup.cfg and tox.ini files are searched. The old ways (certain environment variables, dynamic conftest.py reading is removed).
• add a new “-q” option which decreases verbosity and prints a more nose/unittest-style “dot” output.
• fix issue135 - marks now work with unittest test cases as well
• fix issue126 - introduce py.test.set_trace() to trace execution via PDB during the running of tests even if capturing is ongoing.
• fix issue123 - new “python -m py.test” invocation for py.test (requires Python 2.5 or above)
• fix issue124 - make reporting more resilient against tests opening files on filedescriptor 1 (stdout).
• fix issue109 - sibling conftest.py files will not be loaded. (and Directory collectors cannot be customized anymore from a Directory’s conftest.py - this needs to happen at least one level up).
• introduce (customizable) assertion failure representations and enhance output on assertion failures for comparisons and other cases (Floris Bruynooghe)
• nose-plugin: pass through type-signature failures in setup/teardown functions instead of not calling them (Ed Singleton)
• remove py.test.collect.Directory (follows from a major refactoring and simplification of the collection process)
• majorly reduce py.test core code, shift function/python testing to own plugin
• fix issue88 (finding custom test nodes from command line arg)
• refine ‘tmpdir’ creation, will now create basenames better associated with test names (thanks Ronny)
• “xpass” (unexpected pass) tests don’t cause exitcode!=0
• fix issue131 / issue60 - importing doctests in __init__ files used as namespace packages
• fix issue93 stdout/stderr is captured while importing conftest.py
• fix bug: unittest collected functions now also can have “pytestmark” applied at class/module level
• add ability to use “class” level for cached_setup helper
• fix strangeness: mark.* objects are now immutable, create new instances

1.3.4 (2010-09-14)

• fix issue111: improve install documentation for windows
• fix issue119: fix custom collectability of __init__.py as a module
• fix issue116: –doctestmodules work with __init__.py files as well
• fix issue115: unify internal exception passthrough/catching/GeneratorExit
• fix issue118: new –tb=native for presenting cpython-standard exceptions

1.3.3 (2010-07-30)

• fix issue113: assertion representation problem with triple-quoted strings (and possibly other cases)
• make conftest loading detect that a conftest file with the same content was already loaded, avoids surprises in nested directory structures which can be produced e.g. by Hudson. It probably removes the need to use –confcutdir in most cases.
• fix terminal coloring for win32 (thanks Michael Foord for reporting)
• fix weirdness: make terminal width detection work on stdout instead of stdin (thanks Armin Ronacher for reporting)
• remove trailing whitespace in all py/text distribution files

1.3.2 (2010-07-08)

New features

• fix issue103: introduce py.test.raises as context manager, examples:

  with py.test.raises(ZeroDivisionError):
      x = 0
      1 / x
  
  with py.test.raises(RuntimeError) as excinfo:
      call_something()
  
  # you may do extra checks on excinfo.value|type|traceback here
  

  (thanks Ronny Pfannschmidt)

• Funcarg factories can now dynamically apply a marker to a test invocation. This is for example useful if a factory provides parameters to a test which are expected-to-fail:

  def pytest_funcarg__arg(request):
      request.applymarker(py.test.mark.xfail(reason="flaky config"))
      ...
  
  def test_function(arg):
      ...
  

• improved error reporting on collection and import errors. This makes use of a more general mechanism, namely that for custom test item/collect nodes node.repr_failure(excinfo) is now uniformly called so that you can override it to return a string error representation of your choice which is going to be reported as a (red) string.

• introduce ‘–junitprefix=STR’ option to prepend a prefix to all reports in the junitxml file.

Bug fixes

• make tests and the pytest_recwarn plugin in particular fully compatible to Python2.7 (if you use the recwarn funcarg warnings will be enabled so that you can properly check for their existence in a cross-python manner).
• refine –pdb: ignore xfailed tests, unify its TB-reporting and don’t display failures again at the end.
• fix assertion interpretation with the ** operator (thanks Benjamin Peterson)
• fix issue105 assignment on the same line as a failing assertion (thanks Benjamin Peterson)
• fix issue104 proper escaping for test names in junitxml plugin (thanks anonymous)
• fix issue57 -f|–looponfail to work with xpassing tests (thanks Ronny)
• fix issue92 collectonly reporter and –pastebin (thanks Benjamin Peterson)
• fix py.code.compile(source) to generate unique filenames
• fix assertion re-interp problems on PyPy, by defering code compilation to the (overridable) Frame.eval class. (thanks Amaury Forgeot)
• fix py.path.local.pyimport() to work with directories
• streamline py.path.local.mkdtemp implementation and usage
• don’t print empty lines when showing junitxml-filename
• add optional boolean ignore_errors parameter to py.path.local.remove
• fix terminal writing on win32/python2.4
• py.process.cmdexec() now tries harder to return properly encoded unicode objects on all python versions
• install plain py.test/py.which scripts also for Jython, this helps to get canonical script paths in virtualenv situations
• make path.bestrelpath(path) return “.”, note that when calling X.bestrelpath the assumption is that X is a directory.
• make initial conftest discovery ignore “–” prefixed arguments
• fix resultlog plugin when used in a multicpu/multihost xdist situation (thanks Jakub Gustak)
• perform distributed testing related reporting in the xdist-plugin rather than having dist-related code in the generic py.test distribution
• fix homedir detection on Windows
• ship distribute_setup.py version 0.6.13

1.3.1 (2010-05-25)

New features

• issue91: introduce new py.test.xfail(reason) helper to imperatively mark a test as expected to fail. Can be used from within setup and test functions. This is useful especially for parametrized tests when certain configurations are expected-to-fail. In this case the declarative approach with the @py.test.mark.xfail cannot be used as it would mark all configurations as xfail.

• issue102: introduce new –maxfail=NUM option to stop test runs after NUM failures. This is a generalization of the ‘-x’ or ‘–exitfirst’ option which is now equivalent to ‘–maxfail=1’. Both ‘-x’ and ‘–maxfail’ will now also print a line near the end indicating the Interruption.

• issue89: allow py.test.mark decorators to be used on classes (class decorators were introduced with python2.6) and also allow to have multiple markers applied at class/module level by specifying a list.

• improve and refine letter reporting in the progress bar: . pass f failed test s skipped tests (reminder: use for dependency/platform mismatch only) x xfailed test (test that was expected to fail) X xpassed test (test that was expected to fail but passed)

  You can use any combination of ‘fsxX’ with the ‘-r’ extended reporting option. The xfail/xpass results will show up as skipped tests in the junitxml output - which also fixes issue99.

• make py.test.cmdline.main() return the exitstatus instead of raising SystemExit and also allow it to be called multiple times. This of course requires that your application and tests are properly teared down and don’t have global state.

Bug Fixes

• improved traceback presentation: - improved and unified reporting for “–tb=short” option - Errors during test module imports are much shorter, (using –tb=short style) - raises shows shorter more relevant tracebacks - –fulltrace now more systematically makes traces longer / inhibits cutting
• improve support for raises and other dynamically compiled code by manipulating python’s linecache.cache instead of the previous rather hacky way of creating custom code objects. This makes it seemlessly work on Jython and PyPy where it previously didn’t.
• fix issue96: make capturing more resilient against Control-C interruptions (involved somewhat substantial refactoring to the underlying capturing functionality to avoid race conditions).
• fix chaining of conditional skipif/xfail decorators - so it works now as expected to use multiple @py.test.mark.skipif(condition) decorators, including specific reporting which of the conditions lead to skipping.
• fix issue95: late-import zlib so that it’s not required for general py.test startup.
• fix issue94: make reporting more robust against bogus source code (and internally be more careful when presenting unexpected byte sequences)

1.3.0 (2010-05-05)

• deprecate –report option in favour of a new shorter and easier to remember -r option: it takes a string argument consisting of any combination of ‘xfsX’ characters. They relate to the single chars you see during the dotted progress printing and will print an extra line per test at the end of the test run. This extra line indicates the exact position or test ID that you directly paste to the py.test cmdline in order to re-run a particular test.

• allow external plugins to register new hooks via the new pytest_addhooks(pluginmanager) hook. The new release of the pytest-xdist plugin for distributed and looponfailing testing requires this feature.

• add a new pytest_ignore_collect(path, config) hook to allow projects and plugins to define exclusion behaviour for their directory structure - for example you may define in a conftest.py this method:

  def pytest_ignore_collect(path):
      return path.check(link=1)
  

  to prevent even a collection try of any tests in symlinked dirs.

• new pytest_pycollect_makemodule(path, parent) hook for allowing customization of the Module collection object for a matching test module.

• extend and refine xfail mechanism: @py.test.mark.xfail(run=False) do not run the decorated test @py.test.mark.xfail(reason="...") prints the reason string in xfail summaries specifying --runxfail on command line virtually ignores xfail markers

• expose (previously internal) commonly useful methods: py.io.get_terminal_with() -> return terminal width py.io.ansi_print(…) -> print colored/bold text on linux/win32 py.io.saferepr(obj) -> return limited representation string

• expose test outcome related exceptions as py.test.skip.Exception, py.test.raises.Exception etc., useful mostly for plugins doing special outcome interpretation/tweaking

• (issue85) fix junitxml plugin to handle tests with non-ascii output

• fix/refine python3 compatibility (thanks Benjamin Peterson)

• fixes for making the jython/win32 combination work, note however: jython2.5.1/win32 does not provide a command line launcher, see http://bugs.jython.org/issue1491 . See pylib install documentation for how to work around.

• fixes for handling of unicode exception values and unprintable objects

• (issue87) fix unboundlocal error in assertionold code

• (issue86) improve documentation for looponfailing

• refine IO capturing: stdin-redirect pseudo-file now has a NOP close() method

• ship distribute_setup.py version 0.6.10

• added links to the new capturelog and coverage plugins

1.2.0 (2010-01-18)

• refined usage and options for “py.cleanup”:

  py.cleanup     # remove "*.pyc" and "*$py.class" (jython) files
  py.cleanup -e .swp -e .cache # also remove files with these extensions
  py.cleanup -s  # remove "build" and "dist" directory next to setup.py files
  py.cleanup -d  # also remove empty directories
  py.cleanup -a  # synonym for "-s -d -e 'pip-log.txt'"
  py.cleanup -n  # dry run, only show what would be removed
  

• add a new option “py.test –funcargs” which shows available funcargs and their help strings (docstrings on their respective factory function) for a given test path

• display a short and concise traceback if a funcarg lookup fails

• early-load “conftest.py” files in non-dot first-level sub directories. allows to conveniently keep and access test-related options in a test subdir and still add command line options.

• fix issue67: new super-short traceback-printing option: “–tb=line” will print a single line for each failing (python) test indicating its filename, lineno and the failure value

• fix issue78: always call python-level teardown functions even if the according setup failed. This includes refinements for calling setup_module/class functions which will now only be called once instead of the previous behaviour where they’d be called multiple times if they raise an exception (including a Skipped exception). Any exception will be re-corded and associated with all tests in the according module/class scope.

• fix issue63: assume <40 columns to be a bogus terminal width, default to 80

• fix pdb debugging to be in the correct frame on raises-related errors

• update apipkg.py to fix an issue where recursive imports might unnecessarily break importing

• fix plugin links

1.1.1 (2009-11-24)

• moved dist/looponfailing from py.test core into a new separately released pytest-xdist plugin.
• new junitxml plugin: –junitxml=path will generate a junit style xml file which is processable e.g. by the Hudson CI system.
• new option: –genscript=path will generate a standalone py.test script which will not need any libraries installed. thanks to Ralf Schmitt.
• new option: –ignore will prevent specified path from collection. Can be specified multiple times.
• new option: –confcutdir=dir will make py.test only consider conftest files that are relative to the specified dir.
• new funcarg: “pytestconfig” is the pytest config object for access to command line args and can now be easily used in a test.
• install py.test and py.which with a -$VERSION suffix to disambiguate between Python3, python2.X, Jython and PyPy installed versions.
• new “pytestconfig” funcarg allows access to test config object
• new “pytest_report_header” hook can return additional lines to be displayed at the header of a test run.
• (experimental) allow “py.test path::name1::name2::…” for pointing to a test within a test collection directly. This might eventually evolve as a full substitute to “-k” specifications.
• streamlined plugin loading: order is now as documented in customize.html: setuptools, ENV, commandline, conftest. also setuptools entry point names are turned to canonical names (“pytest_*”)
• automatically skip tests that need ‘capfd’ but have no os.dup
• allow pytest_generate_tests to be defined in classes as well
• deprecate usage of ‘disabled’ attribute in favour of pytestmark
• deprecate definition of Directory, Module, Class and Function nodes in conftest.py files. Use pytest collect hooks instead.
• collection/item node specific runtest/collect hooks are only called exactly on matching conftest.py files, i.e. ones which are exactly below the filesystem path of an item
• change: the first pytest_collect_directory hook to return something will now prevent further hooks to be called.
• change: figleaf plugin now requires –figleaf to run. Also change its long command line options to be a bit shorter (see py.test -h).
• change: pytest doctest plugin is now enabled by default and has a new option –doctest-glob to set a pattern for file matches.
• change: remove internal py._* helper vars, only keep py._pydir
• robustify capturing to survive if custom pytest_runtest_setup code failed and prevented the capturing setup code from running.
• make py.test.* helpers provided by default plugins visible early - works transparently both for pydoc and for interactive sessions which will regularly see e.g. py.test.mark and py.test.importorskip.
• simplify internal plugin manager machinery
• simplify internal collection tree by introducing a RootCollector node
• fix assert reinterpreation that sees a call containing “keyword=…”
• fix issue66: invoke pytest_sessionstart and pytest_sessionfinish hooks on slaves during dist-testing, report module/session teardown hooks correctly.
• fix issue65: properly handle dist-testing if no execnet/py lib installed remotely.
• skip some install-tests if no execnet is available
• fix docs, fix internal bin/ script generation

1.1.0 (2009-11-05)

• introduce automatic plugin registration via ‘pytest11’ entrypoints via setuptools’ pkg_resources.iter_entry_points
• fix py.test dist-testing to work with execnet >= 1.0.0b4
• re-introduce py.test.cmdline.main() for better backward compatibility
• svn paths: fix a bug with path.check(versioned=True) for svn paths, allow ‘%’ in svn paths, make svnwc.update() default to interactive mode like in 1.0.x and add svnwc.update(interactive=False) to inhibit interaction.
• refine distributed tarball to contain test and no pyc files
• try harder to have deprecation warnings for py.compat.* accesses report a correct location

1.0.3

• adjust and improve docs
• remove py.rest tool and internal namespace - it was never really advertised and can still be used with the old release if needed. If there is interest it could be revived into its own tool i guess.
• fix issue48 and issue59: raise an Error if the module from an imported test file does not seem to come from the filepath - avoids “same-name” confusion that has been reported repeatedly
• merged Ronny’s nose-compatibility hacks: now nose-style setup_module() and setup() functions are supported
• introduce generalized py.test.mark function marking
• reshuffle / refine command line grouping
• deprecate parser.addgroup in favour of getgroup which creates option group
• add –report command line option that allows to control showing of skipped/xfailed sections
• generalized skipping: a new way to mark python functions with skipif or xfail at function, class and modules level based on platform or sys-module attributes.
• extend py.test.mark decorator to allow for positional args
• introduce and test “py.cleanup -d” to remove empty directories
• fix issue #59 - robustify unittest test collection
• make bpython/help interaction work by adding an __all__ attribute to ApiModule, cleanup initpkg
• use MIT license for pylib, add some contributors
• remove py.execnet code and substitute all usages with ‘execnet’ proper
• fix issue50 - cached_setup now caches more to expectations for test functions with multiple arguments.
• merge Jarko’s fixes, issue #45 and #46
• add the ability to specify a path for py.lookup to search in
• fix a funcarg cached_setup bug probably only occurring in distributed testing and “module” scope with teardown.
• many fixes and changes for making the code base python3 compatible, many thanks to Benjamin Peterson for helping with this.
• consolidate builtins implementation to be compatible with >=2.3, add helpers to ease keeping 2 and 3k compatible code
• deprecate py.compat.doctest|subprocess|textwrap|optparse
• deprecate py.magic.autopath, remove py/magic directory
• move pytest assertion handling to py/code and a pytest_assertion plugin, add “–no-assert” option, deprecate py.magic namespaces in favour of (less) py.code ones.
• consolidate and cleanup py/code classes and files
• cleanup py/misc, move tests to bin-for-dist
• introduce delattr/delitem/delenv methods to py.test’s monkeypatch funcarg
• consolidate py.log implementation, remove old approach.
• introduce py.io.TextIO and py.io.BytesIO for distinguishing between text/unicode and byte-streams (uses underlying standard lib io.* if available)
• make py.unittest_convert helper script available which converts “unittest.py” style files into the simpler assert/direct-test-classes py.test/nosetests style. The script was written by Laura Creighton.
• simplified internal localpath implementation

1.0.2 (2009-08-27)

• fixing packaging issues, triggered by fedora redhat packaging, also added doc, examples and contrib dirs to the tarball.
• added a documentation link to the new django plugin.

1.0.1 (2009-08-19)

• added a ‘pytest_nose’ plugin which handles nose.SkipTest, nose-style function/method/generator setup/teardown and tries to report functions correctly.
• capturing of unicode writes or encoded strings to sys.stdout/err work better, also terminalwriting was adapted and somewhat unified between windows and linux.
• improved documentation layout and content a lot
• added a “–help-config” option to show conftest.py / ENV-var names for all longopt cmdline options, and some special conftest.py variables. renamed ‘conf_capture’ conftest setting to ‘option_capture’ accordingly.
• fix issue #27: better reporting on non-collectable items given on commandline (e.g. pyc files)
• fix issue #33: added –version flag (thanks Benjamin Peterson)
• fix issue #32: adding support for “incomplete” paths to wcpath.status()
• “Test” prefixed classes are not collected by default anymore if they have an __init__ method
• monkeypatch setenv() now accepts a “prepend” parameter
• improved reporting of collection error tracebacks
• simplified multicall mechanism and plugin architecture, renamed some internal methods and argnames

1.0.0 (2009-08-04)

• more terse reporting try to show filesystem path relatively to current dir
• improve xfail output a bit

1.0.0b9 (2009-07-31)

• cleanly handle and report final teardown of test setup
• fix svn-1.6 compat issue with py.path.svnwc().versioned() (thanks Wouter Vanden Hove)
• setup/teardown or collection problems now show as ERRORs or with big “E“‘s in the progress lines. they are reported and counted separately.
• dist-testing: properly handle test items that get locally collected but cannot be collected on the remote side - often due to platform/dependency reasons
• simplified py.test.mark API - see keyword plugin documentation
• integrate better with logging: capturing now by default captures test functions and their immediate setup/teardown in a single stream
• capsys and capfd funcargs now have a readouterr() and a close() method (underlyingly py.io.StdCapture/FD objects are used which grew a readouterr() method as well to return snapshots of captured out/err)
• make assert-reinterpretation work better with comparisons not returning bools (reported with numpy from thanks maciej fijalkowski)
• reworked per-test output capturing into the pytest_iocapture.py plugin and thus removed capturing code from config object
• item.repr_failure(excinfo) instead of item.repr_failure(excinfo, outerr)

1.0.0b8 (2009-07-22)

• pytest_unittest-plugin is now enabled by default
• introduced pytest_keyboardinterrupt hook and refined pytest_sessionfinish hooked, added tests.
• workaround a buggy logging module interaction (“closing already closed files”). Thanks to Sridhar Ratnakumar for triggering.
• if plugins use “py.test.importorskip” for importing a dependency only a warning will be issued instead of exiting the testing process.
• many improvements to docs: - refined funcargs doc , use the term “factory” instead of “provider” - added a new talk/tutorial doc page - better download page - better plugin docstrings - added new plugins page and automatic doc generation script
• fixed teardown problem related to partially failing funcarg setups (thanks MrTopf for reporting), “pytest_runtest_teardown” is now always invoked even if the “pytest_runtest_setup” failed.
• tweaked doctest output for docstrings in py modules, thanks Radomir.

1.0.0b7

• renamed py.test.xfail back to py.test.mark.xfail to avoid two ways to decorate for xfail
• re-added py.test.mark decorator for setting keywords on functions (it was actually documented so removing it was not nice)
• remove scope-argument from request.addfinalizer() because request.cached_setup has the scope arg. TOOWTDI.
• perform setup finalization before reporting failures
• apply modified patches from Andreas Kloeckner to allow test functions to have no func_code (#22) and to make “-k” and function keywords work (#20)
• apply patch from Daniel Peolzleithner (issue #23)
• resolve issue #18, multiprocessing.Manager() and redirection clash
• make __name__ == “__channelexec__” for remote_exec code

1.0.0b3 (2009-06-19)

• plugin classes are removed: one now defines hooks directly in conftest.py or global pytest_*.py files.
• added new pytest_namespace(config) hook that allows to inject helpers directly to the py.test.* namespace.
• documented and refined many hooks
• added new style of generative tests via pytest_generate_tests hook that integrates well with function arguments.

1.0.0b1

• introduced new “funcarg” setup method, see doc/test/funcarg.txt
• introduced plugin architecture and many new py.test plugins, see doc/test/plugins.txt
• teardown_method is now guaranteed to get called after a test method has run.
• new method: py.test.importorskip(mod,minversion) will either import or call py.test.skip()
• completely revised internal py.test architecture
• new py.process.ForkedFunc object allowing to fork execution of a function to a sub process and getting a result back.

XXX lots of things missing here XXX

0.9.2

• refined installation and metadata, created new setup.py, now based on setuptools/ez_setup (thanks to Ralf Schmitt for his support).
• improved the way of making py.* scripts available in windows environments, they are now added to the Scripts directory as “.cmd” files.
• py.path.svnwc.status() now is more complete and uses xml output from the ‘svn’ command if available (Guido Wesdorp)
• fix for py.path.svn* to work with svn 1.5 (Chris Lamb)
• fix path.relto(otherpath) method on windows to use normcase for checking if a path is relative.
• py.test’s traceback is better parseable from editors (follows the filenames:LINENO: MSG convention) (thanks to Osmo Salomaa)
• fix to javascript-generation, “py.test –runbrowser” should work more reliably now
• removed previously accidentally added py.test.broken and py.test.notimplemented helpers.
• there now is a py.__version__ attribute

0.9.1

This is a fairly complete list of v0.9.1, which can serve as a reference for developers.

• allowing + signs in py.path.svn urls [39106]
• fixed support for Failed exceptions without excinfo in py.test [39340]
• added support for killing processes for Windows (as well as platforms that support os.kill) in py.misc.killproc [39655]
• added setup/teardown for generative tests to py.test [40702]
• added detection of FAILED TO LOAD MODULE to py.test [40703, 40738, 40739]
• fixed problem with calling .remove() on wcpaths of non-versioned files in py.path [44248]
• fixed some import and inheritance issues in py.test [41480, 44648, 44655]
• fail to run greenlet tests when pypy is available, but without stackless [45294]
• small fixes in rsession tests [45295]
• fixed issue with 2.5 type representations in py.test [45483, 45484]
• made that internal reporting issues displaying is done atomically in py.test [45518]
• made that non-existing files are ignored by the py.lookup script [45519]
• improved exception name creation in py.test [45535]
• made that less threads are used in execnet [merge in 45539]
• removed lock required for atomic reporting issue displaying in py.test [45545]
• removed globals from execnet [45541, 45547]
• refactored cleanup mechanics, made that setDaemon is set to 1 to make atexit get called in 2.5 (py.execnet) [45548]
• fixed bug in joining threads in py.execnet’s servemain [45549]
• refactored py.test.rsession tests to not rely on exact output format anymore [45646]
• using repr() on test outcome [45647]
• added ‘Reason’ classes for py.test.skip() [45648, 45649]
• killed some unnecessary sanity check in py.test.collect [45655]
• avoid using os.tmpfile() in py.io.fdcapture because on Windows it’s only usable by Administrators [45901]
• added support for locking and non-recursive commits to py.path.svnwc [45994]
• locking files in py.execnet to prevent CPython from segfaulting [46010]
• added export() method to py.path.svnurl
• fixed -d -x in py.test [47277]
• fixed argument concatenation problem in py.path.svnwc [49423]
• restore py.test behaviour that it exits with code 1 when there are failures [49974]
• don’t fail on html files that don’t have an accompanying .txt file [50606]
• fixed ‘utestconvert.py < input’ [50645]
• small fix for code indentation in py.code.source [50755]
• fix _docgen.py documentation building [51285]
• improved checks for source representation of code blocks in py.test [51292]
• added support for passing authentication to py.path.svn* objects [52000, 52001]
• removed sorted() call for py.apigen tests in favour of [].sort() to support Python 2.3 [52481]