Overview
doctest is a rather unique Python test framework: it turns documented Python statements into test cases. Doctests may be written in two places:
- Directly in the docstrings of the module under test
- In separate text files (potentially for better organization)
The doctest module searches for examples (lines starting with “>>>”) and runs them as if they were interactive sessions entered into a Python shell. The subsequent lines, until the next “>>>” or a blank line, contain the expected output. A doctest will fail if the actual output does not match the expected output verbatim (e.g., string equality).
The doctest module is included out-of-the-box with Python 2 and 3. Like unittest, it can generate XML reports using unittest-xml-reporting.
Installation
doctest does not need any special installation because it comes with Python. However, the unittest-xml-reporting module may be installed with pip if needed:
> pip install unittest-xml-reporting
Project Structure
When doctests are embedded into docstrings, no structural differences are needed. However, if doctests are written as separate text files, then text files should be put under a directory named “doctests”. It may be prudent to create subdirectories that align with the Python package names. Doctest text files should be named after the modules they cover.
[project root directory]
|‐‐ [product code packages]
`-- doctests (?)
`-- test_*.txt (?)
Example Code
An example project named example-py-doctest is located in my GitHub python-testing-101 repository. The project has the following structure:
example-py-doctest |-- com.automationpanda.example | |-- __init__.py | |-- calc_class.py | `-- calc_func.py |-- doctests | `-- test_calc_class.txt `-- README.md
The com.automationpanda.example.calc_func module contains doctests embedded in the docstrings of math functions, alongside other comments:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| def add(a, b): | |
| """ | |
| Adds two numbers. | |
| >>> add(3, 2) | |
| 5 | |
| """ | |
| return a + b | |
| def subtract(a, b): | |
| """ | |
| Subtracts two numbers. | |
| >>> subtract(3, 2) | |
| 1 | |
| >>> subtract(2, 3) | |
| -1 | |
| """ | |
| return a – b | |
| def multiply(a, b): | |
| """ | |
| Multiplies two numbers. | |
| >>> multiply(3, 2) | |
| 6 | |
| """ | |
| return a * b | |
| def divide(a, b): | |
| """ | |
| Divides two numbers. | |
| Automatically raises ZeroDivisionError. | |
| >>> divide(3.0, 2.0) | |
| 1.5 | |
| >>> divide(1.0, 0) | |
| Traceback (most recent call last): | |
| … | |
| ZeroDivisionError: float division by zero | |
| """ | |
| return a * 1.0 / b | |
| def maximum(a, b): | |
| """ | |
| Finds the maximum of two numbers. | |
| >>> maximum(3, 2) | |
| 3 | |
| >>> maximum(2, 3) | |
| 3 | |
| >>> maximum(3, 3) | |
| 3 | |
| """ | |
| return a if a >= b else b | |
| def minimum(a, b): | |
| """ | |
| Finds the minimum of two numbers. | |
| >>> minimum(3, 2) | |
| 2 | |
| >>> minimum(2, 3) | |
| 2 | |
| >>> minimum(2, 2) | |
| 2 | |
| """ | |
| return a if a <= b else b |
On the other hand, the com.automationpanda.example.calc_class module contains a Calculator class without doctests in docstrings:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| class Calculator(object): | |
| def __init__(self): | |
| self._last_answer = 0.0 | |
| @property | |
| def last_answer(self): | |
| return self._last_answer | |
| def add(self, a, b): | |
| self._last_answer = a + b | |
| return self.last_answer | |
| def subtract(self, a, b): | |
| self._last_answer = a – b | |
| return self.last_answer | |
| def multiply(self, a, b): | |
| self._last_answer = a * b | |
| return self.last_answer | |
| def divide(self, a, b): | |
| # automatically raises ZeroDivisionError | |
| self._last_answer = a * 1.0 / b | |
| return self.last_answer | |
| def maximum(self, a, b): | |
| self._last_answer = a if a >= b else b | |
| return self.last_answer | |
| def minimum(self, a, b): | |
| self._last_answer = a if a <= b else b | |
| return self.last_answer |
Its doctests are located in a separate text file at doctests/test_calc_class.txt:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| The “com.automationpanda.example.calc_class“ module | |
| ===================================================== | |
| >>> from com.automationpanda.example.calc_class import Calculator | |
| >>> calc = Calculator() | |
| >>> calc.add(3, 2) | |
| 5 | |
| >>> calc.subtract(3, 2) | |
| 1 | |
| >>> calc.subtract(2, 3) | |
| -1 | |
| >>> calc.multiply(3, 2) | |
| 6 | |
| >>> calc.divide(3.0, 2.0) | |
| 1.5 | |
| >>> calc.divide(1.0, 0) | |
| Traceback (most recent call last): | |
| … | |
| ZeroDivisionError: float division by zero | |
| >>> calc.maximum(3, 2) | |
| 3 | |
| >>> calc.maximum(2, 3) | |
| 3 | |
| >>> calc.maximum(3, 3) | |
| 3 | |
| >>> calc.minimum(3, 2) | |
| 2 | |
| >>> calc.minimum(2, 3) | |
| 2 | |
| >>> calc.minimum(2, 2) | |
| 2 |
Doctests are run in the order in which they are written. The examples above align functions with docstrings and classes with text files, but this is not required. Functions may have doctests in separate text files, and classes may have doctests embedded in method docstrings. Additional tricks are documented online.
Test Launch
To launch tests from the command line, change directory to the project root directory and run the doctest module directly from the python command. Note that doctests use file paths, not module names.
# Run doctests embedded as docstrings > python -m doctest com/automationpanda/example/calc_func.py # Run doctests written in separate text files > python -m doctest doctests/test_calc_class.txt
When doctests run successfully, they don’t print any output! This may be surprising to a first-time user, but no news is good news. However, to force output, include the “-v” option:
# Run doctests with verbose output to print successes as well as failures > python -m doctest -v com/automationpanda/example/calc_func.py > python -m doctest -v doctests/test_calc_class.txt
Output should look something like this:
> python -m doctest -v doctests/test_calc_class.txt
Trying:
from com.automationpanda.example.calc_class import Calculator
Expecting nothing
ok
Trying:
calc = Calculator()
Expecting nothing
ok
Trying:
calc.add(3, 2)
Expecting:
5
ok
...
1 items passed all tests:
14 tests in test_calc_class.txt
14 tests in 1 items.
14 passed and 0 failed.
Test passed.
Doctests can also generate XML reports using unittest-xml-reporting. Follow the same instructions given for unittest. Furthermore, doctests can integrate with unittest discovery, so that test suites can run together.
Pros and Cons
doctest has many positive aspects. It is very simple yet powerful, and it has practically no learning curve. Since the doctest module comes with Python out of the box, no extra dependencies are required. It integrates nicely with unittest. Tests can be written in-line with the code, providing not only verification tests but also examples for the reader. And if in-line tests are deemed too messy, they can be moved to separate text files.
However, doctest has limitations. First of all, doctests are not independent: Python commands run sequentially and build upon each other. Thus, doctests may not be run individually, and side effects from one example may affect another. doctest also lacks many features of advanced frameworks, including hooks, assertions, tracing, discovery, replay, and advanced reporting. Theoretically, many of these things could be put into doctests, but they would be inelegantly jury-rigged. Long doctests become cumbersome. Furthermore, console output string-matching is not a robust assertion method. Silent Python statements that do not return a value or print output cannot be legitimately tested. Programmers can easily mistype expected output. Output format might also change in future development, or it may be nondeterministic (like for timestamps).
My main recommendation is this: use doctest for small needs but not big needs. doctest would be a good option for small tools and scripts that need spot checks instead of intense testing. It is also well suited for functional programming testing, in which expressions do not have side effects. Doctests should also be used to provide standard examples in docstrings wherever possible, in conjunction with other tests. Rich documentation is wonderful, and working examples can be a godsend. However, serious testing needs a serious framework, such as pytest or behave.
Automation Panda 
Many thanks for the nice post, it was very interesting and informative.
LikeLike