Python Testing 101: pytest-bdd

Warning: If you are new to BDD, then I strongly recommend reading the BDD 101 series before trying to use pytest-bdd. Also, make sure that you are already familiar with the pytest framework.

Overview

pytest-bdd is a behavior-driven (BDD) test framework that is very similar to behave, Cucumber and SpecFlow. BDD frameworks are very different from more traditional frameworks like unittest and pytest. Test scenarios are written in Gherkin “.feature” files using plain language. Each Given, When, and Then step is “glued” to a step definition – a Python function decorated by a matching string in a step definition module. This means that there is a separation of concerns between test cases and test code. Gherkin steps may also be reused by multiple scenarios.

pytest-bdd is very similar to other Python BDD frameworks like behave, radish, and lettuce. However, unlike the others, pytest-bdd is not a standalone framework: it is a plugin for pytest. Thus, all of pytest‘s features and plugins can be used with pytest-bdd. This is a huge advantage!

Installation

Use pip to install both pytest and pytest-bdd.

pip install pytest
pip install pytest-bdd

Project Structure

Project structure for pytest-bdd is actually pretty flexible (since it is based on pytest), but the following conventions are recommended:

  • All test code should appear under a test directory named “tests”.
  • Feature files should be placed in a test subdirectory named “features”.
  • Step definition modules should be placed in a test subdirectory named “step_defs”.
  • conftest.py files should be located together with step definition modules.

Other names and hierarchies may be used. For example, large test suites can have feature-specific directories of features and step defs. pytest should be able to discover tests anywhere under the test directory.

[project root directory]
|‐‐ [product code packages]
|-- [test directories]
|   |-- features
|   |   `-- *.feature
|   `-- step_defs
|       |-- __init__.py
|       |-- conftest.py
|       `-- test_*.py
`-- [pytest.ini|tox.ini|setup.cfg]

Note: Step definition module names do not need to be the same as feature file names. Any step definition can be used by any feature file within the same project.

Example Code

An example project named behavior-driven-python located in GitHub shows how to write tests using pytest-bdd. This section will explain how the Web tests are designed.

The top layer for pytest-bdd tests is the set of Gherkin feature files. Notice how the scenario below is concise, focused, meaningful, and declarative:

@web @duckduckgo
Feature: DuckDuckGo Web Browsing
  As a web surfer,
  I want to find information online,
  So I can learn new things and get tasks done.

  # The "@" annotations are tags
  # One feature can have multiple scenarios
  # The lines immediately after the feature title are just comments

  Scenario: Basic DuckDuckGo Search
    Given the DuckDuckGo home page is displayed
    When the user searches for "panda"
    Then results are shown for "panda"

Each scenario step is “glued” to a decorated Python function called a step definition. Step definitions are written in Python test modules, as shown below:

import pytest

from pytest_bdd import scenarios, given, when, then, parsers
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

# Constants

DUCKDUCKGO_HOME = 'https://duckduckgo.com/'

# Scenarios

scenarios('../features/web.feature')

# Fixtures

@pytest.fixture
def browser():
    b = webdriver.Firefox()
    b.implicitly_wait(10)
    yield b
    b.quit()

# Given Steps

@given('the DuckDuckGo home page is displayed')
def ddg_home(browser):
    browser.get(DUCKDUCKGO_HOME)

# When Steps

@when(parsers.parse('the user searches for "{phrase}"'))
def search_phrase(browser, phrase):
    search_input = browser.find_element_by_id('search_form_input_homepage')
    search_input.send_keys(phrase + Keys.RETURN)

# Then Steps

@then(parsers.parse('results are shown for "{phrase}"'))
def search_results(browser, phrase):
    # Check search result list
    # (A more comprehensive test would check results for matching phrases)
    # (Check the list before the search phrase for correct implicit waiting)
    links_div = browser.find_element_by_id('links')
    assert len(links_div.find_elements_by_xpath('//div')) > 0
    # Check search phrase
    search_input = browser.find_element_by_id('search_form_input')
    assert search_input.get_attribute('value') == phrase

Notice how each Given/When/Then step has a function with an appropriate decorator. Arguments, such as the search “phrase,” may also be passed from step to function. pytest-bdd provides a few argument parsers out of the box and also lets programmers implement their own. (By default, strings are compared using equality.) One function can be decorated for many steps, too.

pytest fixtures may also be used by step functions. The code above uses a fixture to initialize the Firefox WebDriver before each scenario and then quit it after each scenario. Fixtures follow all the same rules, including scope. Any step function can use a fixture by declaring it as an argument. Furthermore, any “@given” step function that returns a value can also be used as a fixture. Please read the official docs for more info about fixtures with pytest-bdd.

One important, easily-overlooked detail is that scenarios must be explicitly declared in test modules. Unlike other BDD frameworks that treat feature files as the main scripts, pytest-bdd treats the “test_*.py” module as the main scripts (because that’s what pytest does). Scenarios may be specified explicitly using scenario decorators, or all scenarios in a list of feature files may be included implicitly using the “scenarios” shortcut function shown above.

To share steps across multiple feature files, add them to the “conftest.py” file instead of the test modules. Since scenarios must be declared within a test module, they can only use step functions available within the same module or in “conftest.py”. As a best practice, put commonly shared steps in “conftest.py” and feature-specific steps in the test module. The same recommendation also applies for hooks.

Scenario outlines require special implementation on the Python side to run successfully. Unfortunately, steps used by scenario outlines need unique step decorators and extra converting. Please read the official docs or the example project to see examples.

Test Launch

pytest-bdd can leverage the full power of pytest. Tests can be run in full or filtered by tag. Below are example commands using the example project:

# run all tests
pytest

# filter tests by test module
# note: feature files cannot be run directly
pytest tests/step_defs/test_unit_basic.py
pytest tests/step_defs/test_unit_outlines.py
pytest tests/step_defs/test_unit_service.py
pytest tests/step_defs/test_unit_web.py

# filter tests by tags
# running by tag is typically better than running by path
pytest -k "unit"
pytest -k "service"
pytest -k "web"
pytest -k "add or remove"
pytest -k "unit and not outline"

# print JUnit report
pytest -junitxml=/path/for/output

pytest-bdd tests can be executed and filtered together with regular pytest tests. Tests can all be located within the same directory. Tags work just like pytest.mark. As a warning, marks must be explicitly added to “pytest.ini” starting with pytest 5.0.

All other pytest plugins should work, too. For example:

Pros and Cons

Just like for other BDD frameworks, pytest-bdd is best suited for black-box testing because it forces the developer to write test cases in plain, descriptive language. In my opinion, it is arguably the best BDD framework currently available for Python because it rests on the strength and extendability of pytest. It also has PyCharm support (in the Professional Edition). However, it can be more cumbersome to use than behave due to the extra code needed for declaring scenarios, implementing scenario outlines, and sharing steps. Nevertheless, I would still recommend pytest-bdd over behave for most users because it is more powerful – pytest is just awesome!

52 comments

  1. Pingback: Python BDD Framework Comparison | Automation Panda
  2. Pingback: PyCon Canada 2018 Reflections | Automation Panda
  4. Pingback: Using Multiple Test Frameworks Simultaneously | Automation Panda

  5. Hi,

    I am new to coding and using Pytest BDD. Is it possible to use Pycharm community version for Pytest BDD ?

    Thanks,

    Like

    Reply

      1. Thanks for getting back. I am fine to ignore the automatic generation of step definitions.
        But will i be able to set up framework and execute tests in Community version.

        Like

      2. In theory, yes. Just make sure the right packages are installed and then run the tests using pytest. PyCharm Community Edition has Run Configuration for pytest.

        Like

  7. Hi Andy,
    Greetings,

    If possible to run all the tests cases by using pytest-bdd in community version. Obviously there is no auto generating step definition file.. As Per your comment we can create manunaly right?
    Suppose in future any dependencies need for my framework because once I have started with community version I don’t need stuck.

    Please suggest me.

    Like

    Reply

    1. Yes, you can create step definitions manually. I don’t think using PyCharm Community Edition will limit your future possibilities because you would write tests no differently inside or outside the IDE.

      Like

      Reply

  9. Hi Andy, I want to capture each execution, so I used this hook “def pytest_bdd_before_step(request, feature, scenario, step, step_func):
    print(f’Step Executed: {step}’)” like but it is not logging any thing but I use this hook “def pytest_bdd_step_error(request, feature, scenario, step, step_func, step_func_args, exception):
    print(f’Step failed: {step}’)” (created the error manually) and it is logging all the success and failed steps. How to get the successful steps alone in the log file?

    Like

    Reply

    1. Tried with this hook also”def pytest_bdd_before_step_call(request, feature, scenario, step, step_func, step_func_args):
      print(f’Step Executed: {step}’)” same result, with fail only all the steps are logged

      Like

      Reply
  10. Pingback: Test-Driving TestProject’s New Python SDK | Automation Panda

  11. Hi Andy,

    Thanks for the explanation.

    I need help with something, I’m trying to run a test (login_step_defs.py) who is pointing to a specific feature file using scenarios(‘../features/name_login.feature’) but that specific feature has 3 scenarios inside. When I run the test, it only executes a single scenario.

    I tried using commands like:

    * pytest -k “web” —-> With the same “web” tag in my scenarios inside the feature file.
    * python -m pytest ui_tests/web_test/test/step_definitions/login_step_defs.py

    Do you know any command or something to run all the scenarios? Should I separate those scenarios in different features files ?

    Thanks.

    Like

    Reply
  14. Pingback: Learning Python Test Automation | Automation Panda

  15. Hi Andy,
    How can we set the scope of a fixture to a “feature” level? I want to call the fixture after running all scenarios within a feature file. I have tried class, session, package and module but none of them run after all scenarios in a feature file. Class scope triggers the fixture after every scenario and the others trigger only once(before tests start) throughout the test run.

    Like

    Reply

    1. Unfortunately, I don’t know if there is a way to do that with pytest-bdd. To be honest, with other frameworks (like SpecFlow), I’ve never needed to handle before/after logic at the level of the feature. I stick to per-scenario or per-whole-suite-run.

      Like

      Reply

      1. Thanks Andy. The reason I asked is – I want to create a few resources only in the first scenario and just make some read calls to those resources in other scenrios. If I let every scenario create it’s own resource, it would be too much data. Any hack that you know of?

        Like

      2. You could create a pytest fixture for your setup and set its scope to “module” or “session”. Then, call it from desired step definition functions. Even if you set the scope to “session” the fixture will be called only by the steps that actually use it.

        Like

  16. Thanks, Andy. I did try session and module scopes. They both were triggered only once throughout the test run. But, I needed it to be triggered for every feature file. I guess, I will have to let it run with every scenario with “class” scope and keep cleaning up at the end of every successful run of the scenario.

    Like

    Reply

  18. Hi Andy , how to share a global variable between step definitions in conftest and test modules? .I have defined global variables using pytest.variablename = {} in conftest.py and trying to modify the dictionary value from the test_example.py like pytest.variablename[‘name’] =”qa1″ .Is there a betterway to do this.

    Like

    Reply

  19. Hi Andy,

    I found this post and your blog a few months ago when I decided to give this BDD thing a try. And it turned out, this post was all I needed to get started and have my own test for my own app up in minutes. Which was a great experience!
    So I wanted to say thank you for running this blog and sharing your knowledge. Definitely looking forward to your book!

    Like

    Reply

  20. Hi Andy

    Now is 2021, like to get your opinion that if pytest-bdd is still better than behave? I trying to decide which frame to use for integration testing between different applications

    Like

    Reply

  21. Hi Andy, is it good to use multiple conftest under tests dir, like the below format.
    tests
    project_A_tests
    conftest
    project_B_tests
    conftest

    Like

    Reply

    1. pytest lets you create a conftest.py file in any subdirectory to apply to all tests in that subdirectory. That’s a good way to control fixture scope. Just make sure to use the hierarchy wisely.

      Liked by 1 person

      Reply

  22. Hi Andy. I’m completely new to the BDD framework. I just wanted to ask, while using BDD do I still need to use the Page Object Model method when writing test cases? Thanks

    Like

    Reply

      1. Hi Andy, thanks for answering my above question. Do you have any material or course using POM within a BDD framework? I can’t find much information or example hence why I thought you didn’t need to.

        Like

      2. Unfortunately, I don’t have an example ready that shows pytest-bdd with page objects. However, my TAU course on Selenium WebDriver with Python shows page objects. You can use page objects in a similar way with pytest-bdd. Just create the classes and call them from step definition functions.

        Like

  25. Hi,
    I m using selenium pytest BDD. Now for every scenario there is a test in pytest. And every time browser will open and close for each scenario.
    I want to keep one browser open to run multiple test and thn it shud get close. Example: in single browser 5 TC run and thn browser is closing.

    Is there any way to handle it??

    Like

    Reply

  26. Hi, from what I understood from pytest-bdd documentation, it should be possible to use tags in examples tables, but it doesn’t work the same as is in the Behave. Do you use this function in your tests, or maybe know how that should be used? Something like this, based on the tag you can pick which table you want to use:
    @test
    Scenario Outline: Test
    Where Eat
    Then You are full

    @tag1
    Example:
    |meal|
    |Hot-dog|

    @tag2
    |meal|
    |pizza|

    Like

    Reply

  27. Hi there,
    pytest has this pytest-xdist module, which can have multiple workers running in parallel mode.

    In your web test case, if I was to simulate hundred person to enter different key-word on duckduckgo.com, and to search the results at the same time. should I have more than 1 copy of feature file, and more that 1 copy of test_web.py file ?

    In other way to ask my question, can I include more than one feature file, in this line:
    “scenarios(‘../features/web.feature’)” ?

    I have used behave BDD a lot and I am searching for some stability logic for my new project.

    Thanks,

    Jack

    Like

    Reply

Leave a comment