Python Testing 101: pytest

Overview

pytest is an awesome Python test framework. According to its homepage:

pytest is a mature full-featured Python testing tool that helps you write better programs.

Pytests may be written either as functions or as methods in classes – unlike unittest, which forces tests to be inside classes. Test classes must be named “Test*”, and test functions/methods must be named “test_*”. Test classes also need not inherit from unittest.TestCase or any other base class. Thus, pytests tend to be more concise and more Pythonic. pytest can also run unittest and nose tests.

pytest provides many advanced test framework features:

pytest is actively supported for both Python 2 and 3.

Installation

Use pip to install the pytest module. Optionally, install other plugins as well.

pip install pytest
pip install pytest-cov
pip install pytest-xdist
pip install pytest-bdd

Project Structure

The modules containing pytests should be named “test_*.py” or “*_test.py”. While the pytest discovery mechanism can find tests anywhere, pytests must be placed into separate directories from the product code packages. These directories may either be under the project root or under the Python package. However, the pytest directories must not be Python packages themselves, meaning that they should not have “__init__.py” files. (My recommendation is to put all pytests under “[project root]/tests”.) Test configuration may be added to configuration files, which may go by the names “pytest.ini”, “tox.ini”, or “setup.cfg”.

[project root directory]
|‐‐ [product code packages]
|-- [test directories]
|   |-- test_*.py
|   `-- *_test.py
`-- [pytest.ini|tox.ini|setup.cfg]

Example Code

An example project named example-py-pytest is located in my GitHub python-testing-101 repository. The project has the following structure:

example-py-pytest
|-- com.automationpanda.example
|   |-- __init__.py
|   |-- calc_class.py
|   `-- calc_func.py
|-- tests
|   |-- test_calc_class.py
|   `-- test_calc_func.py
|-- README.md
`-- pytest.ini

The pytest.ini file is simply a configuration file stub. Feel free to add contents for local testing needs.


# Add pytest options here
[pytest]

view raw

pytest.ini

hosted with ❤ by GitHub

The com.automationpanda.example.calc_func module contains basic math functions.


def add(a, b):
return a + b
def subtract(a, b):
return a – b
def multiply(a, b):
return a * b
def divide(a, b):
return a * 1.0 / b
def maximum(a, b):
return a if a >= b else b
def minimum(a, b):
return a if a <= b else b

view raw

calc_func.py

hosted with ❤ by GitHub

The calc_func tests located in tests/test_calc_func.py are written as functions. Test functions are preferable to test classes when testing functions without side effects.


import pytest
from com.automationpanda.example.calc_func import *
NUMBER_1 = 3.0
NUMBER_2 = 2.0
def test_add():
value = add(NUMBER_1, NUMBER_2)
assert value == 5.0
def test_subtract():
value = subtract(NUMBER_1, NUMBER_2)
assert value == 1.0
def test_subtract_negative():
value = subtract(NUMBER_2, NUMBER_1)
assert value == -1.0
def test_multiply():
value = multiply(NUMBER_1, NUMBER_2)
assert value == 6.0
def test_divide():
value = divide(NUMBER_1, NUMBER_2)
assert value == 1.5

view raw

test_calc_func.py

hosted with ❤ by GitHub

The divide-by-zero test uses pytest.raises:


def test_divide_by_zero():
with pytest.raises(ZeroDivisionError) as e:
divide(NUMBER_1, 0)
assert "division by zero" in str(e.value)

view raw

test_calc_func.py

hosted with ❤ by GitHub

And the min/max tests use parameterization:


@pytest.mark.parametrize("a,b,expected", [
(NUMBER_1, NUMBER_2, NUMBER_1),
(NUMBER_2, NUMBER_1, NUMBER_1),
(NUMBER_1, NUMBER_1, NUMBER_1),
])
def test_maximum(a, b, expected):
assert maximum(a, b) == expected
@pytest.mark.parametrize("a,b,expected", [
(NUMBER_1, NUMBER_2, NUMBER_2),
(NUMBER_2, NUMBER_1, NUMBER_2),
(NUMBER_2, NUMBER_2, NUMBER_2),
])
def test_minimum(a, b, expected):
assert minimum(a, b) == expected

view raw

test_calc_func.py

hosted with ❤ by GitHub

The com.automationpanda.example.calc_class module contains the Calculator class, which uses the math functions from calc_func. Keeping the functional spirit, the private _do_math method takes in a reference to the math function for greater code reusability.


from com.automationpanda.example.calc_func import *
class Calculator(object):
def __init__(self):
self._last_answer = 0.0
@property
def last_answer(self):
return self._last_answer
def _do_math(self, a, b, func):
self._last_answer = func(a, b)
return self.last_answer
def add(self, a, b):
return self._do_math(a, b, add)
def subtract(self, a, b):
return self._do_math(a, b, subtract)
def multiply(self, a, b):
return self._do_math(a, b, multiply)
def divide(self, a, b):
return self._do_math(a, b, divide)
def maximum(self, a, b):
return self._do_math(a, b, maximum)
def minimum(self, a, b):
return self._do_math(a, b, minimum)

view raw

calc_class.py

hosted with ❤ by GitHub

While tests for the Calculator class could be written using a test class, pytest test functions are just as capable. Fixtures enable a more fine-tuned setup/cleanup mechanism than the typical xUnit-like methods found in test classes. Fixtures can also be used in conjunction with parameterized methods. The tests/test_calc_class.py module is very similar to tests/test_calc_func.py and shows how to use fixtures for testing a class.


import pytest
from com.automationpanda.example.calc_class import Calculator
# "Constants"
NUMBER_1 = 3.0
NUMBER_2 = 2.0
# Fixtures
@pytest.fixture
def calculator():
return Calculator()
# Helpers
def verify_answer(expected, answer, last_answer):
assert expected == answer
assert expected == last_answer
# Test Cases
def test_last_answer_init(calculator):
assert calculator.last_answer == 0.0
def test_add(calculator):
answer = calculator.add(NUMBER_1, NUMBER_2)
verify_answer(5.0, answer, calculator.last_answer)
def test_subtract(calculator):
answer = calculator.subtract(NUMBER_1, NUMBER_2)
verify_answer(1.0, answer, calculator.last_answer)
def test_subtract_negative(calculator):
answer = calculator.subtract(NUMBER_2, NUMBER_1)
verify_answer(–1.0, answer, calculator.last_answer)
def test_multiply(calculator):
answer = calculator.multiply(NUMBER_1, NUMBER_2)
verify_answer(6.0, answer, calculator.last_answer)
def test_divide(calculator):
answer = calculator.divide(NUMBER_1, NUMBER_2)
verify_answer(1.5, answer, calculator.last_answer)
def test_divide_by_zero(calculator):
with pytest.raises(ZeroDivisionError) as e:
calculator.divide(NUMBER_1, 0)
assert "division by zero" in str(e.value)
@pytest.mark.parametrize("a,b,expected", [
(NUMBER_1, NUMBER_2, NUMBER_1),
(NUMBER_2, NUMBER_1, NUMBER_1),
(NUMBER_1, NUMBER_1, NUMBER_1),
])
def test_maximum(calculator, a, b, expected):
answer = calculator.maximum(a, b)
verify_answer(expected, answer, calculator.last_answer)
@pytest.mark.parametrize("a,b,expected", [
(NUMBER_1, NUMBER_2, NUMBER_2),
(NUMBER_2, NUMBER_1, NUMBER_2),
(NUMBER_2, NUMBER_2, NUMBER_2),
])
def test_minimum(calculator, a, b, expected):
answer = calculator.minimum(a, b)
verify_answer(expected, answer, calculator.last_answer)

view raw

test_calc_class.py

hosted with ❤ by GitHub

Personally, I prefer to write pytests as functions because they are usually cleaner and more flexible than classes. Plus, test functions appeal to my affinity for functional programming.

Test Launch

Basic Test Execution

pytest has a very powerful command line for launching tests. Simply run the pytest module from within the project root directory, and pytest will automatically discover tests.

# Find and run all pytests from the current directory
python -m pytest

# Run pytests under a given path
python -m pytest 

# Run pytests in a specific module
python -m pytest tests/test_calc_func.py

# Generate JUnit-style XML test reports
python -m pytest --junitxml=[path-to-file]

# Get command help
python -m pytest -h

The terminal output looks like this:

python -m pytest
=============================== test session starts ===============================
platform darwin -- Python 2.7.13, pytest-3.0.6, py-1.4.32, pluggy-0.4.0
rootdir: /Users/andylpk247/Programming/automation-panda/python-testing-101/example-py-pytest, inifile: pytest.ini
plugins: cov-2.4.0
collected 25 items

tests/test_calc_class.py .............
tests/test_calc_func.py ............

============================ 25 passed in 0.11 seconds ============================

pytest also provides shorter “pytest” and “py.test” command that may be run instead of the longer “python -m pytest” module form. However, the shorter commands do not append the current path to PYTHONPATH, meaning modules under test may not be importable. Make sure to update PYTHONPATH before using the shorter commands.

# Update the Python path
PYTHONPATH=$PYTHONPATH:.

# Discover and run tests using the shorter command
pytest

Code Coverage

To run code coverage with the pytest-cov plugin module, use the following command. The report types are optional, but all four types are show below. Specific paths for each report may be appended using “:”.

# Run tests with code coverage
python -m pytest [test-path] [other-options] \
      --cov= \
      --cov-report=annotate \
      --cov-report=html \
      --cov-report=term \
      --cov-report=xml

Code coverage output on the terminal (“term” cov-report) looks like this:

python -m pytest --cov=com --cov-report=term
============================= test session starts ==============================
platform darwin -- Python 3.6.5, pytest-3.0.6, py-1.4.32, pluggy-0.4.0
rootdir: /Users/andylpk247/Programming/automation-panda/python-testing-101/example-py-pytest, inifile: pytest.ini
plugins: cov-2.4.0
collected 25 items 

tests/test_calc_class.py .............
tests/test_calc_func.py ............

---------- coverage: platform darwin, python 3.6.5-final-0 -----------
Name                                        Stmts   Miss  Cover
---------------------------------------------------------------
com/__init__.py                                 0      0   100%
com/automationpanda/__init__.py                 0      0   100%
com/automationpanda/example/__init__.py         0      0   100%
com/automationpanda/example/calc_class.py      21      0   100%
com/automationpanda/example/calc_func.py       12      0   100%
---------------------------------------------------------------
TOTAL                                          33      0   100%

========================== 25 passed in 0.11 seconds ===========================<span id="mce_SELREST_start" style="overflow:hidden;line-height:0;"></span>

Parallel Testing

Parallel testing is vital for more intense testing, such as web testing. The pytest-xdist plugin makes it possible both to scale-up tests by running more than one test process and to scale-out by running tests on other machines. (As a prerequisite, machines need rsync and SSH.) The command below shows how to run multiple test sub-processes; refer to official documentation for multi-machine setup

python -m pytest -n 4
============================= test session starts ==============================
platform darwin -- Python 3.6.5, pytest-3.0.6, py-1.4.32, pluggy-0.4.0
rootdir: /Users/andylpk247/Programming/automation-panda/python-testing-101/example-py-pytest, inifile: pytest.ini
plugins: xdist-1.22.2, forked-0.2, cov-2.4.0
gw0 [25] / gw1 [25] / gw2 [25] / gw3 [25]
scheduling tests via LoadScheduling
.........................
========================== 25 passed in 1.30 seconds ===========================

Pros and Cons

I’ll say it again: pytest is awesome. It is a powerful test framework with many features, yet its tests are concise and readable. It is very popular and actively supported for both versions of Python. It can handle testing at the unit, integration, and end-to-end levels. It can also be extended with plugins, notably ones for code coverage, parallel execution, and BDD. The only challenge with pytest is that advanced features (namely fixtures) have a learning curve.

My recommendation is to use pytest for standard functional testing in Python. It is one of the best and most popular test frameworks available, and it beats the pants off of alternatives like unittest and nose. pytest is my go-to Python framework, period.

This article is meant to be an introduction. Check out Python Testing with pytest by Brian Okken for deeper study.

 

Update: On 4/21/2018, I added pytest-xdist and pytest-bdd plugins, and I made some cosmetic changes.

Update: On 7/29/2018, I added the book recommendation for “Python Testing with pytest.”