Author: Andy Knight

I'm a software engineer who specializes in test automation.

Frameworks: All-in-One or Piece-by-Piece?

Software frameworks are great because they apply the principle of Separation of Concerns. A framework’s tools and code handle a specific need in a standard way for developers to write other code more easily. For example:

  • Web frameworks support receiving requests and sending responses.
  • Test frameworks include test case structure, runners, and reporting mechanisms.
  • Logging frameworks control how messages are gathered and stored.
  • Dependency injection frameworks create and manage object instances.

Recently, a question hit me: How far should a framework go to separate concerns? Should a framework try to do everything all-in-one, or should it behave more like a library that focuses on doing one thing well?

Let’s look at Python Web frameworks as an example. Django, the “Web framework for perfectionists with deadlines,” provides everything a developer could want out of the box. Flask, on the other hand, is a “microframework” that prides itself on minimalism: any extras must be handled by extensions or other packages. The differences between the two become clear when comparing some of their features:

Feature Django Flask
HTTP Requests and Routing Included Werkzeug (bundled)
Templates Included Jinja2 (bundled)
Forms Included None (Flask-WTF)
Object-Relational Mapping (ORM) Included None (SQLAlchemy)
Security Included None (Flask-Security)
Localization Included None (Flask-Babel)
Admin Interface Included None

Clearly, Django is all-in-one, while Flask is piece-by-piece. To make a serious Flask app, developers must pull in many extra pieces. There are many other frameworks with similar competitions:

  • JavaScript testing: Jasmine vs. Mocha
  • JavaScript development: Angular vs. React
  • Java BDD testing: Serenity vs. Cucumber-JVM

I think each approach has its merits. All-in-one frameworks are more convenient to use, especially for beginners. For developers who are new to a domain or just need to get something up fast, all-in-ones are the better choice. They come with all units already integrated together, and they often have good documentation. However, developing an all-in-one framework takes much more work because it covers multiple concerns. Developers may also feel shoehorned into the framework’s way of doing things. All-in-ones typically dictate what they believe to be the “best” solution.

Piece-by-piece frameworks require more expertise but offer greater flexibility. Developers can pick and choose the pieces they need, and they can change the packages used by the solution more easily. Found a better ORM? Not a problem. Need to localize the site in Chinese? Add it! Solutions can avoid excess weight and stay nimble for the future. The big challenge is successful integration. Furthermore, a library or framework for a singular concern tends to solve the concern in better ways simply because project contributors give it exclusive focus. The more I learn about a space, the more I lean towards a piece-by-piece approach.

As always, pick frameworks based on the needs at hand. For example, I like to use Django to make websites for my wife’s small businesses because the admin interface is just so convenient for her, even though I could get away with Flask. However, I’ll probably pick Mocha (piece-by-piece) over Jasmine (all-in-one) whenever I return to JavaScript testing.

PyGotham 2018 Reflections

As only New Yorkers know, if you can get through the twilight, you’ll live through the night. (Dorothy Parker)

PyGotham 2018 was my first PyGotham conference and my third conference of the year. I first heard about PyGotham when Jon Banafato and Dan Crosta, two of the organizers, invited me to submit proposals when we met at the PyCon 2018 Instagram dinner. I enjoyed the conference very much, but my experience was much different than the previous two conferences.

A Tourist Day in New York

PyGotham is hosted in New York City. I booked my flight a day before the conference started to make sure that I’d be on time. My wife also came along for the trip. That meant Thursday, October 4 was a tourist day! Our flight arrived in EWR at about 10:30am. We took a commuter train to Penn Station in Manhattan and walked a block to our hotel, Hotel Pennsylvania. Unfortunately, we couldn’t check into our room until 3pm, so we dropped our bags (for $15) and went along our way.

Our first stop was lunch at Xi’an Famous Foods. XFF is a fast casual restaurant chain for authentic Chinese food local to NYC. We ordered two noodle dishes (one liangpi “cold skin” noodles and the other the wide hand-pulled “biang biang” noodles), a pulled pork “burger,” and hawberry tea. Our meal was absolutely delicious.

We spent our afternoon at the National September 11 Memorial & Museum. I was just starting 8th grade when the terrorist attacks happened, but I remembered all the details clearly. The outdoor memorial – a sunken fountain for each tower’s square base – showed both emptiness and resolve. The museum was built into the underground foundations for the towers, essentially as a cave. The starkness of the concrete, the use of light and shadow, and the pacing of exhibits all highlighted the tragedy of 9/11/2001. Every American should visit the site if they are able.

One World Trade Center
The slurry wall
Part of the antenna

We checked into our hotel before dinner that evening. And then we learned why Hotel Pennsylvania has only 2.7 stars on Google: it’s nasty. The hotel is 99 years old and looks like it. The carpets are filthy, the lights are dim, and most of the building smells damp. I chose to stay there because the conference was held there, but I don’t think I will be returning.

One plus side of Hotel PA is that it is right next to Koreatown. My wife and I ate dinner at Five Senses, a trendy Korean restaurant with a line nearly out the door. We then walked around a bit to see the hustle and bustle: we tried Gong Cha bubble tea, got breakfast for the next morning at H-Mart, and bought some face masks at Kosette Beauty Market. A Korean friend of ours was jealous!

Dinner at the Five Senses
A baby octopus!

The Conference

The conference itself lasted two days: Friday, October 5 and Saturday, October 6. About 600 people attended over the two days. By this time, I was well acquainted with Python conference formats. Each day opened with a keynote, followed by several talks in a few rooms with breaks in between. I reconnected with old friends like Trey Hunner and Gabriel Boorse and also made a few new ones.

Trey (left) and me (right)
Gabriel (middle), his friend (left), and me (right)

The opening keynote was “Software Freedom and Ethical Technology” by Karen M. Sandler. Karen explained why free, open source software is foundational for the ethical use and development of technology. She used the example of her Implantable Cardioverter Defibrillator (ICD). Nearly all modern ICDs have wireless connectivity, which renders them vulnerable to hacking. Since the source code in this device and so many others in the Internet of things is closed, individuals are unable to make changes for customization or protection. Karen’s talk was definitely thought-provoking.

My favorite talk, by far, was Trey Hunner’s “Python Oddities Explained”. Trey dove deep into the way variables, data structures, and scope work in Python 3. This talk was particularly helpful for me because I work in several languages and sometimes assume how small details work.

My talk, “Egad! How Do We Start Writing (Better) Tests?“, explained how to build a functional test automation solution in Python from the ground up. It is the same talk I delivered at PyOhio 2018, though I hope it turned out better the second time!

Attendance was pretty good, too.

IMG_3638

Conference room panorama for my talk

Video recordings for all talks can be found on the PyGotham 2018 YouTube channel.

PyGotham had a stronger business vibe than the other Python conferences I attended. Many attendees came as part of company groups. (I heard rumors that about a hundred came from Bloomberg.) Speakers more openly talked about their companies and specific projects. This all made sense considering the location being NYC.

A number of vendors also had tables with goodies to give away. Bank of America, Buzzfeed, Venmo, Linode, Clover, and Django Girls all showed up, along with others I cannot remember off the top of my head. I netted two t-shirts (in addition to two PyGotham t-shirts), a notebook, some silly putty from Linode, and a handful of stickers.

PyGotham 2018!
Sticker addition complete!
The sticker table

After Hours

On Friday night after the conference, my wife and I took the Subway to Brooklyn and walked the Brooklyn Bridge back to Manhattan. The view of the city skyline was spectacular. We then hopped the Subway up to the Upper West Side for drinks with my friend Evan. On the way back to the hotel, we stopped at a dim sum restaurant for a late night dinner.

We didn’t have time to explore New York on Saturday night because we had an overnight flight to catch: we were going to London, England for the week after the conference! We checked out of the hotel, caught the Long Island Rail Road to JFK, and then flew Norwegian Air across the pond.

IMG_3644

Dim sum? Dim yum!

Takeaways

During PyGotham 2018, I spent a lot of time pondering software testing and automation. I like to use conferences as my time to disconnect from immediacy and think big thoughts.

One thought I formed was the nuance between testing and change detection. A friend of Gabriel’s posed an idea for REST API service testing. Rather than write a bunch of test cases that check responses, why not simply gather responses and diff them against previous responses? Request automation would still be needed, but response assertions could basically be skipped. Any changes would be identified and reported. This approach struck me as novel and useful but not comprehensive. Change detection will reveal when a code change results in a feature change, but it lacks the moral calculus of true testing because it cannot determine if the change is good or bad. It must yield the judgment to a higher authority, like a developer, a tester, or even an AI agent (which arguably further begs the question of goodness). Nevertheless, I think change detection can be very useful in conjunction with testing, either as a tool to assist manual testing or as an extension to an automation framework.

Another thought I had was to notice the widening gap between companies over testing maturity. Some companies are on the bleeding edge with the latest technologies, tools, and frameworks. They have multi-stage pipelines with countless automated tests and several daily deployments. Other companies are stuck in the past, still suffering through “Agile transformations” and failing tests. There are now several vendors pitching advanced testing solutions, like Sauce Labs for cross-browser testing and Applitools for visual testing. I wonder how many of these vendors are too far ahead for companies with less mature testing to catch up.

For the first time at a conference, I felt serious self doubt. Am I behind the times as a Software Engineer in Test? Is my method of software testing already antiquated? Am I truly a developer, or do I just pretend? Is there a future for the SET role? Will my ideas add value to the field? Will I have time to pursue all of my professional and technical desires? Am I good enough to be here at PyGotham? I felt a bit lonely and withdrawn. My malaise did not sour my conference experience, but it did come unexpectedly. Nevertheless, I must use it as impetus to become a better me.

Final Thoughts

I’m thankful for the opportunity I had to speak at PyGotham 2018. I thank the conference organizers for accepting my talk proposal, as well as PrecisionLender for sponsoring my travel costs. I’d love to return for PyGotham 2019 if possible!

Python Testing 101: pytest-bdd

Warning: If you are new to BDD, then I strongly recommend reading the BDD 101 series before trying to use pytest-bdd. Also, make sure that you are already familiar with the pytest framework.

Overview

pytest-bdd is a behavior-driven (BDD) test framework that is very similar to behave, Cucumber and SpecFlow. BDD frameworks are very different from more traditional frameworks like unittest and pytest. Test scenarios are written in Gherkin “.feature” files using plain language. Each Given, When, and Then step is “glued” to a step definition – a Python function decorated by a matching string in a step definition module. This means that there is a separation of concerns between test cases and test code. Gherkin steps may also be reused by multiple scenarios.

pytest-bdd is very similar to other Python BDD frameworks like behave, radish, and lettuce. However, unlike the others, pytest-bdd is not a standalone framework: it is a plugin for pytest. Thus, all of pytest‘s features and plugins can be used with pytest-bdd. This is a huge advantage!

Installation

Use pip to install both pytest and pytest-bdd.

pip install pytest
pip install pytest-bdd

Project Structure

Project structure for pytest-bdd is actually pretty flexible (since it is based on pytest), but the following conventions are recommended:

  • All test code should appear under a test directory named “tests”.
  • Feature files should be placed in a test subdirectory named “features”.
  • Step definition modules should be placed in a test subdirectory named “step_defs”.
  • conftest.py files should be located together with step definition modules.

Other names and hierarchies may be used. For example, large test suites can have feature-specific directories of features and step defs. pytest should be able to discover tests anywhere under the test directory.

[project root directory]
|‐‐ [product code packages]
|-- [test directories]
|   |-- features
|   |   `-- *.feature
|   `-- step_defs
|       |-- __init__.py
|       |-- conftest.py
|       `-- test_*.py
`-- [pytest.ini|tox.ini|setup.cfg]

Note: Step definition module names do not need to be the same as feature file names. Any step definition can be used by any feature file within the same project.

Example Code

An example project named behavior-driven-python located in GitHub shows how to write tests using pytest-bdd. This section will explain how the Web tests are designed.

The top layer for pytest-bdd tests is the set of Gherkin feature files. Notice how the scenario below is concise, focused, meaningful, and declarative:

@web @duckduckgo
Feature: DuckDuckGo Web Browsing
  As a web surfer,
  I want to find information online,
  So I can learn new things and get tasks done.

  # The "@" annotations are tags
  # One feature can have multiple scenarios
  # The lines immediately after the feature title are just comments

  Scenario: Basic DuckDuckGo Search
    Given the DuckDuckGo home page is displayed
    When the user searches for "panda"
    Then results are shown for "panda"

Each scenario step is “glued” to a decorated Python function called a step definition. Step definitions are written in Python test modules, as shown below:

import pytest

from pytest_bdd import scenarios, given, when, then, parsers
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

# Constants

DUCKDUCKGO_HOME = 'https://duckduckgo.com/'

# Scenarios

scenarios('../features/web.feature')

# Fixtures

@pytest.fixture
def browser():
    b = webdriver.Firefox()
    b.implicitly_wait(10)
    yield b
    b.quit()

# Given Steps

@given('the DuckDuckGo home page is displayed')
def ddg_home(browser):
    browser.get(DUCKDUCKGO_HOME)

# When Steps

@when(parsers.parse('the user searches for "{phrase}"'))
def search_phrase(browser, phrase):
    search_input = browser.find_element_by_id('search_form_input_homepage')
    search_input.send_keys(phrase + Keys.RETURN)

# Then Steps

@then(parsers.parse('results are shown for "{phrase}"'))
def search_results(browser, phrase):
    # Check search result list
    # (A more comprehensive test would check results for matching phrases)
    # (Check the list before the search phrase for correct implicit waiting)
    links_div = browser.find_element_by_id('links')
    assert len(links_div.find_elements_by_xpath('//div')) > 0
    # Check search phrase
    search_input = browser.find_element_by_id('search_form_input')
    assert search_input.get_attribute('value') == phrase

Notice how each Given/When/Then step has a function with an appropriate decorator. Arguments, such as the search “phrase,” may also be passed from step to function. pytest-bdd provides a few argument parsers out of the box and also lets programmers implement their own. (By default, strings are compared using equality.) One function can be decorated for many steps, too.

pytest fixtures may also be used by step functions. The code above uses a fixture to initialize the Firefox WebDriver before each scenario and then quit it after each scenario. Fixtures follow all the same rules, including scope. Any step function can use a fixture by declaring it as an argument. Furthermore, any “@given” step function that returns a value can also be used as a fixture. Please read the official docs for more info about fixtures with pytest-bdd.

One important, easily-overlooked detail is that scenarios must be explicitly declared in test modules. Unlike other BDD frameworks that treat feature files as the main scripts, pytest-bdd treats the “test_*.py” module as the main scripts (because that’s what pytest does). Scenarios may be specified explicitly using scenario decorators, or all scenarios in a list of feature files may be included implicitly using the “scenarios” shortcut function shown above.

To share steps across multiple feature files, add them to the “conftest.py” file instead of the test modules. Since scenarios must be declared within a test module, they can only use step functions available within the same module or in “conftest.py”. As a best practice, put commonly shared steps in “conftest.py” and feature-specific steps in the test module. The same recommendation also applies for hooks.

Scenario outlines require special implementation on the Python side to run successfully. Unfortunately, steps used by scenario outlines need unique step decorators and extra converting. Please read the official docs or the example project to see examples.

Test Launch

pytest-bdd can leverage the full power of pytest. Tests can be run in full or filtered by tag. Below are example commands using the example project:

# run all tests
pytest

# filter tests by test module
# note: feature files cannot be run directly
pytest tests/step_defs/test_unit_basic.py
pytest tests/step_defs/test_unit_outlines.py
pytest tests/step_defs/test_unit_service.py
pytest tests/step_defs/test_unit_web.py

# filter tests by tags
# running by tag is typically better than running by path
pytest -k "unit"
pytest -k "service"
pytest -k "web"
pytest -k "add or remove"
pytest -k "unit and not outline"

# print JUnit report
pytest -junitxml=/path/for/output

pytest-bdd tests can be executed and filtered together with regular pytest tests. Tests can all be located within the same directory. Tags work just like pytest.mark. As a warning, marks must be explicitly added to “pytest.ini” starting with pytest 5.0.

All other pytest plugins should work, too. For example:

Pros and Cons

Just like for other BDD frameworks, pytest-bdd is best suited for black-box testing because it forces the developer to write test cases in plain, descriptive language. In my opinion, it is arguably the best BDD framework currently available for Python because it rests on the strength and extendability of pytest. It also has PyCharm support (in the Professional Edition). However, it can be more cumbersome to use than behave due to the extra code needed for declaring scenarios, implementing scenario outlines, and sharing steps. Nevertheless, I would still recommend pytest-bdd over behave for most users because it is more powerful – pytest is just awesome!

Speaking Pythonese

The Python community, like many groups, has its own language – and I don’t mean just Python. There are many words and phrases thrown around that may confuse people new to Python. I originally shared some terms in my article, Which Version of Python Should I Use?, but below are some more of those colloquialisms for quick reference:

Word or Phrase Meaning
Anaconda
  • Open-source implementation of Python (and R)
  • Meant for data scientists
  • Uses the conda package manager
Benevolent Dictator for Life (BDFL)
  • Guido van Rossum
  • The inventor of Python
  • Resigned in July 2018 but remains BDFL Emeritus
The CheeseShop
  • a fun code name for the Python Package Index
Class
  • A programming definition for creating objects
  • Combines attributes (variables) and behaviors (methods)
  • Useful for reusing code
Conda
  • Package manager for Python and other languages
  • Part of the Anaconda project
Core Developer
  • A developer who has commit privilege to the CPython codebase
  • Very few Pythonistas are core developers
CPython
  • The default and most widely used implementation of the Python language
  • Implemented in C
Django
  • A batteries-included Python Web framework for perfectionists with deadlines
  • Offers many features out of the box
  • The most popular Python framework in 2017
  • Size: Flask < Pyramid < Django
Flask
  • A microframework for Python Web development
  • Uses Werkzeug and Jinja2
  • Super-minimalist
  • Size: Flask < Pyramid < Django
Function
  • A definition for a callable subroutine
  • May take inputs
  • May return outputs
  • Great for code reuse
  • Functions are first-order values
The Hitchhiker’s Guide to Python
  • A popular online guide to Python
  • Opinionated
  • Shares many good best practices
Jinja2
  • A Python template engine
  • Inspired by Django’s templates
Jupyter
  • A project for interactive Python development
  • “Jupyter Notebooks” allow programmers to dynamically rewrite and rerun Python code, and then share code easily with others
  • Popular with data scientists
Module
  • A Python source code file containing definitions and statements
  • Every .py file is a module
  • May be imported by other modules to reuse code
NumPy
  • A popular Python package for scientific computing
Pandas
  • A popular Python package for data analysis
pip
  • The PyPA-recommended tool for installing Python packages
  • Command: “pip install “
  • Recursive acronym for “pip installs packages”
PyBites
  • A community of Pythoneers who improve their skills through code challenges
PyCharm
  • A popular Python IDE developed by JetBrains
  • Offers great development features
  • Has a free Community Edition and a paid Professional Edition
PyCon
  • The annual Python conference held in North America
  • GO – it will change your life!
  • Several other conferences are held worldwide
PyPy
  • An alternative Python implementation
  • Known for speed, memory usage, and compatibility
  • Good alternative to CPython for high performance workloads
Pyramid
  • A Python Web framework
  • Start small, finish big, stay finished
  • Provides many parts but not everything (such as ORM)
  • Size: Flask < Pyramid < Django
pytest
  • A lightweight-yet-powerful Python test framework (and arguably the best)
Python 2
  • The old version of Python
  • Will reach end-of-life in 2020
  • Final version will be 2.7.x
  • Please upgrade to version 3
Python 3
  • The current version of Python
  • Most packages now support Python 3
  • Has incompatibilities with Python 2
  • Please don’t use Python 2
Python Enhancement Proposal (PEP)
  • Official proposals for enhancing the Python language
Python Package Index (PyPI)
The Python Software Foundation (PSF)
  • Non-profit organization
  • Keeps Python going strong
  • Support them!
Pythoneer
  • A programmer who uses Python to solve problems
  • Styled off the word “engineer”
Pythonic
  • Describes idiomatic code for Python
  • Closely related to conciseness, readability, and elegance
  • Highly recommended
  • Follow style guidelines to learn how to be Pythonic
Pythonista
  • Someone who loves the Python language
  • Often an advanced Python programmer
Sphinx
  • A popular Python tool to generate documentation
virtualenv
  • Tool to create isolated Python environments
  • Enables programmers to use different versions of Python and packages for different projects
  • Also see venv and pipenv
Web Server Gateway Interface (WSGI)
  • A specification for how Web servers forward requests to Web applications and frameworks
  • A core piece of Python Web development
  • See PEP-333 and PEP-3333
The Zen of Python
  • The list of guiding principles for Python’s design
  • Run “import this” to see them
  • See PEP-20

The Software Engineer in Test

I am a Software Engineer in Test (SET). Many people don’t know quite what that means, though. Developers frequently refer to me as a “tester” or “QA,” and a former director once thought I did DevOps. While my work covers these areas, they aren’t the main focus. Let’s clarify what it means to be a SET.

What is a Software Engineer in Test?

A “Software Engineer in Test” (SET) builds solutions to testing problems. Those problems could include slow feedback, flaky tests, costly runs, poor product quality, lack of communication during development, and more. In some circles, the role is called “Software Development Engineer in Test” (SDET).

Frequently, SETs focus on developing test automation solutions for running tests quickly and repeatedly. Test automation is a software product, after all. Just as front-end developers write web pages and back-end developers write microservices, SETs write automated tests. The same practices and coding skills apply. I frequently say that a Software Engineer in Test must have the heart of a developer.

However, a SET is more than just an automation engineer. SETs also look holistically at quality through the software development lifecycle. They build systems for fast feedback, such as Continuous Integration pipelines and reports. They work with developers and product owners to develop good product designs. They provide tools and frameworks to empower others to cover any testing needs. At times, they may also jump in to help a team with manual and exploratory testing.

So they just write test scripts?

No. Never say this to a SET. Test automation involves much more than just “writing test scripts.” A serious testing solution requires serious design and effort. The top-level automation of a test case is usually just a small piece. SETs are responsible for:

  • Collaborating with developers and product owners
    • Contributing to planning and design
    • Reviewing product code
    • Formulating test scenarios
  • Developing test automation frameworks
  • Automating test scenarios using the frameworks
  • Knowing and using design patterns where appropriate
  • Setting up the infrastructure to run tests
    • Running tests in CI/CD
    • Running tests in parallel
    • Running tests with the appropriate test data
  • Setting up dashboards for reporting test results in real time
  • Teaching others good quality and testing practices
  • Developing tools to assist manual and exploratory testing

Saying that testing is “just scripting” belittles the role of the SET and underestimates the workload it requires.

How did this role start?

Software “tester” and “QA” roles have existed for decades, but the SET role first became distinct in the 2000s when large-scale test automation became both feasible and necessary. According to Wikipedia, Microsoft coined the title “Software Developer Engineer in Test” (SDET) in 2005, and others like Amazon and Apple quickly adopted it. Google coined the name “Software Engineer in Test” for the same type of role. I personally prefer the SET title over the SDET title simply because it is more concise.

How is it different from “QA” or “testing” roles?

To me, the SET role is distinct from the traditional “QA” or “testing” role. Software testers historically focused on manual testing and thus didn’t need strong programming skills. Their fortes were product domain knowledge, intuition, out-of-the-box thinking, test planning, and test system setup. And these certainly are important, indispensable skills! SETs, however, live in both the development and testing worlds. They use developer skills to provide software solutions for testing problems. Automation is usually much more central to the SET role. Nevertheless, there will always be a need for manual testing because there are some problems a human can catch much more easily than a script (or an AI agent). A traditional tester will be responsible for doing test cases for a team, while a SET builds the solutions to empower other testers and developers to do the testing.

Personally, I avoid using the titles “QA” and “software tester” for myself because they don’t accurately describe all that I do. I also avoid the title “automation engineer” because, again, it is reductionist. I tackle software testing with the heart of a developer, and I set up test automation solutions from the ground up. I’m proud to be a software engineer who specializes in testing.

As a bonus, check out Test and Code episode 47, in which Brian Okken and I discuss what it means to be a Software Engineer in Test (among other topics).

Book Review: pytest Quick Start Guide

tl;dr

Title pytest Quick Start Guide
(available from Packt and Amazon)
Author Bruno Oliveira (@nicoddemus)
Publication 2018 Packt Publishing
Summary “Write better Python code with simple and maintainable tests” – a very readable guide for pytest’s main features.
Prerequisites Intermediate-level Python programming.

Summary

pytest Quick Start Guide is a new book about using pytest for Python test automation. Bruno Oliveira explains not only how to use pytest but also why its features are useful. Even though this book is written as a “quick start” introduction, it nevertheless dives deep into pytest’s major features. It covers:

  1. An introduction to pytest
  2. Why pytest is superior to unittest
  3. Setting up pytest with pip and virtualenv
  4. Writing basic tests
  5. How assertions really work
  6. Common command line arguments
  7. Marks and parametrization
  8. Using and writing fixtures
  9. Popular plugins
  10. Converting unittest suites to pytest

Example code is hosted on GitHub at PacktPublishing/pytest-Quick-Start-Guide.

Praises

This book is an easy introduction to test automation in Python with pytest. Readers should have intermediate-level Python skills, but they do not need previous testing or automation skills. The progression of chapters makes it easy to start quick and then go deeper. Oliveira has a very accessible writing style, too.

The unittest refactoring guide is a hidden gem. The unittest module is great because it comes bundled with Python, but it is also clunky and not very Pythonic. Not all teams know the best way to modernize their test suites. Oliveira provides many pieces of practical advice for making the change, at varying degrees of conversion. The big changes use pytest’s fixtures and assertions.

A Comparison to Python Testing with pytest

Python Testing with pytest by Brian Okken is another popular pytest book. Both books are great resources for learning pytest, but each approaches the framework from a unique perspective. How do they compare? Here’s what I saw:

Oliveira’s book Okken’s book
Oliveira’s book is a great introductory guide for pytest. Oliveira’s writing style makes the reader feel like the author is almost teaching in person. The book’s main theme is getting the reader to use the main features of pytest pragmatically for general testing needs. The main differences in content are the unittest refactoring guide and some of the plugins and fixtures covered. This book is probably the best choice for beginners who want to learn the basics of pytest quickly. Okken’s book is introductory but also a great manual for future reference. Okken’s writing style is direct and concise, which covers more material in fewer pages. The format for each chapter is consistent: for each idea: idea → code → output → explanation; exercises at the end. Okken also covers how to create and share custom pytest plugins. This book is probably the best choice for people who want to master the ins and outs of pytest.

Ultimately, I recommend both books because they are both excellent.

Takeaways

Reading books on the same subject by different authors helps the reader learn the subject better. I’ve used pytest quite a lot myself, but I was able to learn new things from both pytest Quick Start Guide and Python Testing with pytest. Reading how experts use and think about the framework makes me a better engineer. Different writing styles and different opinions also challenge my own understandings. (It’s also funny that the authors of both pytest books have the same initials – “B.O.”)

pytest is really popular. There are now multiple good books on the subject. It’s becoming the de facto test automation framework for Python, outpacing unittest, nose, and others. These days, it seems more popular to write a pytest plugin than to create a new framework.

Ignoring Files with Git

Git is one of the most popular version control systems (VCS) available, especially thanks to hosting vendors like GitHub. It keeps code safe and shareable. Sometimes, however, certain files should not be shared, like local settings or temporary configs. Git provides a few ways to make sure those files are ignored.

.gitignore

The easiest and most common way to ignore files is to use a gitignore file. Simply create a file named .gitignore in the repository’s root directory. Then, add names and patterns for any files and directories that should not be added to the repository. Use the asterisk (“*”) as a wildcard. For example, “*.class” will ignore all files that have the “.class” extension. Remember to add the .gitignore file to the repository so that it can be shared. As a bonus, Git hosting vendors like GitHub usually provide standard .gitignore templates for popular languages.

Any files covered by the .gitignore file will not be added to the repository. This approach is ideal for local IDE settings like .idea or .vscode, compiler output files like *.class or *.pyc, and test reports. For example, here’s GitHub’s .gitignore template for Java:


# Compiled class file
*.class
# Log file
*.log
# BlueJ files
*.ctxt
# Mobile Tools for Java (J2ME)
.mtj.tmp/
# Package Files #
*.jar
*.war
*.nar
*.ear
*.zip
*.tar.gz
*.rar
# virtual machine crash logs, see http://www.java.com/en/download/help/error_hotspot.xml
hs_err_pid*

view raw

GitHub_Java.gitignore

hosted with ❤ by GitHub

.git/info/exclude

As a best practice, .gitignore should be committed to the repository, which means all team members will share the same set of ignored files. However, some files should be ignored locally and not globally. Those files could be added to .gitignore, but large .gitignore files become cryptic and more likely to break other people’s setup. Thankfully, git provides a local-only solution: the .git/info/exclude file (under the repository’s hidden .git directory). Simply open it with a text editor and add new entries using the same file pattern format as .gitignore.

# Append a new file to ignore locally
echo "my_private_file" >> .git/info/exclude

skip-worktree

A .gitignore file prevents a file from being added to a repository, but what about preventing changes from being committed to an existing file? For example, developers may want to safely override settings in a shared config file for local testing. That’s where skip-worktree comes in: it allows a developer to “skip” any local changes made to a given file. Changes will not appear under “git status” and thus will not be committed.

Use the following commands:

# Ignore local changes to an existing file
git update-index --skip-worktree path/to/file

# Stop ignoring local changes
git update-index --no-skip-worktree path/to/file

Warning: The skip-worktree setting applies only to the local repository. It is not applied globally! Each developer will need to run the skip-worktree command in their local repository.

assume-unchanged

Another option for ignoring files is assume-unchanged. Like skip-worktree, it makes Git ignore changes to files. However, whereas skip-worktree assumes that the user intends to change the file, assume-unchanged assumes that the user will not change the file. The intention is different. Large projects using slow file systems may gain significant performance optimizations by marking unused directories as assume-unchanged. This option also works with other update-index options like really-refresh.

Use the following commands:

# Assume a file will be unchanged
git update-index --assume-unchanged path/to/file

# Undo that assumption
git update-index --no-assume-unchanged path/to/file

Again, this setting applies only to the local repository – it is not applied globally.

Comparison Table

Which is the best way to ignore files?

Method Description Best Use Cases Scope
.gitignore file Prevents files from being added to the repository. Local settings, compiler output, test results, etc. Global
.git/info/exclude file Prevents local files from being added to the repository. Local settings, compiler output, test results, etc. Local
skip-worktree setting Prevents local changes from being committed to an existing file. Shared files that will have local overwrites, like config files. Local
assume-unchanged setting Allows Git to skip files that won’t be changed for performance optimization. Files and folders that a developer won’t touch. Local

Resources

Book Review: Python Testing with pytest

tl;dr

Title Python Testing with pytest
Author Brian Okken (@brianokken)
Publication 2017 (The Pragmatic Programmers)
Summary How to use all the features of pytest for Python test automation – “simple, rapid, effective, and scalable.”
Prerequisites Intermediate-level Python programming.

Summary

Python Testing with pytest is the book on pytest*. Brian Okken covers all the ins and outs of the framework. The book is useful both as tutorial for learning pytest as well as a reference for specific framework features. It covers:

  • Getting started with pytest
  • Writing simple tests as functions
  • Writing more interesting tests with assertions, exceptions, and parameters
  • Using all the different execution options
  • Writing fixtures to flexibly separate concerns and reuse code
  • Using built-in fixtures like tmpdir, pytestconfig, and monkeypatch
  • Using configuration files to control execution
  • Integrating pytest with other tools like pdb, tox, and Jenkins

Appendices also cover:

  • Using Python virtual environments
  • Installing packages with pip
  • An overview of popular plugins like pytest-xdist and pytest-cov
  • Packaging and distributing Python packages

[* Update on 9/27/2018: Check out my review on another excellent pytest book, pytest Quick Start Guide by Bruno Oliveira.]

Praises

This book is a comprehensive guide to pytest. It thoroughly covers the framework’s features and gives pointers to more info elsewhere. Even though pytest has excellent online documentation, I still recommend this book to anyone who wants to become a pytest master. Online docs tend to be fragmented with each piece limited in scope, whereas books like this one are designed to be read progressively and orderly for maximal understanding of the material.

I love how this book is example-driven. Each section follows a simple yet powerful outline: idea → code → output → explanation. Having real code with real output truly cements the point of each mini-lesson. New topics are carefully unfolded so that they build upon previous topics, making the book read like a collection of tutorials. Examples at the end of every chapter challenge the readers to practice what they learn. The formatting of each section also looks great.

The extra info on related topics like pip and virtualenv is also a nice touch. Python pros probably don’t need it, but beginners might get stuck without it.

The rocket ship logo on the cover is also really cool!

Takeaways

pytest is one of the best functional test frameworks currently available in any language. It breaks the clunky xUnit mold, in which class structures were awkwardly superimposed over test cases as if one size fits all. Instead, pytest is lightweight and minimalist because it relies on functions and fixtures. Scope is much easier to manage, code is more reusable, and side effects can more easily be avoided. pytest has taken over Python testing because it is so Pythonic.

Brian’s concise writing style has also inspired me to be more direct in my own writing. I tend to be rather verbose out of my desire to be descriptive. However, fewer words often leave a more powerful impression. They also make the message easier to comprehend. Python is beloved for its concise expressiveness, and as a Pythonista, it would be fitting for me to adopt that trait into my English.

If I had a wish list for a second edition, I’d love to see more info about assertions and other plugins (namely pytest-bdd). I think an appendix with comparisons to other Python test frameworks would also be valuable.

A Warning

I ordered a physical copy of this book directly from Amazon (not a third-party seller). Unfortunately, that copy was missing all the introductory content: the table of contents, the acknowledgements, and the preface. The first page after the front cover was Chapter 1. Befuddled, I reached out to Brian Okken (who I personally met at PyCon 2018). We suspected that it was either a misprint or a bootleg copy. Either way, we sent the evidence to the publisher, and Amazon graciously exchanged my defective copy for the real deal. Please look out for this sort of problem if you purchase a printed copy of this book for yourself!

If you want to learn more about pytest, please read my article Python Testing 101: pytest.

NuGet Quick Reference

What is NuGet?

NuGet is a package manager for Microsoft .NET. It installs packages and manages dependencies for .NET projects. It is like Maven (Java) or pip (Python). The NuGet Gallery hosts thousands of popular packages like Json.NET, NUnit, and jQuery. If you develop .NET applications (like in C#), then you probably need to use NuGet.

Installing Packages

The easiest way to use NuGet is through Visual Studio, which includes NuGet features by default. Packages are managed per project. Right-click on a project in Solution Explorer and select “Manage NuGet Packages…” to open the project’s package manager page.

  • The Browse tab lets you search and install new packages.
  • The Installed tab shows which packages are installed and can uninstall them.
  • The Updates tab lets you update packages to their latest versions.

Nuget Package Manager Page

The NuGet Package Manager page for a project in Visual Studio

When packages are installed and updated, NuGet also pulls any dependencies they require. Visual Studio also creates a packages.config file for all dependencies. Then, just build and run!


<?xml version="1.0" encoding="utf-8"?>
<packages>
<package id="FluentAssertions" version="5.4.1" targetFramework="net461" />
<package id="Newtonsoft.Json" version="11.0.2" targetFramework="net461" />
<package id="SpecFlow" version="2.4.0" targetFramework="net461" />
<package id="SpecRun.Runner" version="1.8.0" targetFramework="net461" />
<package id="SpecRun.SpecFlow" version="1.8.0" targetFramework="net461" />
<package id="SpecRun.SpecFlow.2-4-0" version="1.8.0" targetFramework="net461" />
<package id="System.ValueTuple" version="4.5.0" targetFramework="net461" />
</packages>

view raw

example-nuget-quick-ref-packages-config.xml

hosted with ❤ by GitHub

NuGet Configuration

NuGet can be configured using a NuGet.Config file. This file can be placed under a project directory, a solution directory, or a system-wide location. One of the most common settings is the package sources: NuGet uses the public nuget.org repository by default, but others (like private company repos) can also be added. Check the nuget.config reference online for docs on all options. (Package sources can also be configured through Visual Studio under Tools > NuGet Package Manager > Package Manager Settings.)


<?xml version="1.0" encoding="utf-8"?>
<configuration>
<packageSources>
<add key="nuget.org" value="https://www.nuget.org/api/v2/" />
<add key="my.company.com" value="https://my.company.com/httpAuth/app/nuget/v1/FeedService.svc/" />
</packageSources>
</configuration>

view raw

example-nuget-quick-ref-nuget-config.xml

hosted with ❤ by GitHub

NuGet Package Manager Console

Sometimes, it’s helpful to control NuGet directly through the Package Manager Console. From the menu bar: Tools > NuGet Package Manager > Package Manager Console. For example, when packages get messed up, I’ll run “Update-Package -Reinstall” to reinstall everything. (Right-clicking the solution and selecting “Restore NuGet Packages” never seems to work for me.) Check the help command or the official guide for more info.

Nuget Package Manager Console

The NuGet Package Manager Console in Visual Studio

NuGet CLI

The NuGet CLI nuget.exe provides the full extent of NuGet features, including the ability to make packages. It is more powerful than the Package Manager Console. It must be installed independently – it does not come with Visual Studio. Check the NuGet CLI reference online for full details. The .NET Core CLI dotnet.exe can also be used for managing packages. See the feature comparison for the differences.

nuget CLI

The NuGet CLI

Creating a NuGet Package

A NuGet package is basically a ZIP file with a .nupkg extension. It typically contains an assembly DLL and maybe other related files. Creating a NuGet package is pretty easy:

  1. Install the NuGet CLI.
  2. Create a .nuspec file for the project.
  3. Add appropriate settings to the .nuspec file.
  4. Run the “nuget pack” command to create the .nupkg file.
  5. Publish the .nupkg file to the desired destination.

The .nuspec file can be created by running the “nuget spec” command in the project’s directory. The generated <project-name>.nuspec file will contain replacement tokens that will be substituted with values from the project’s AssemblyInfo when the package is built. Make sure to set AssemblyInfo values appropriately for the substitution. The version is especially important, and the automatic version format may be useful for guaranteeing uniqueness. Be sure to add any packages upon which the project depends as dependencies, too. (The .nuspec file can also be created manually.) Refer to the .nuspec reference for full details.

The standard package creation command is “nuget pack <project-name>.nuspec”. However, if the .nuspec file contains replacement tokens, then use “nuget pack <project-name>.csproj” instead. Once the package is created, it can be published publicly to nuget.org or to a private NuGet feed.

Below is an example .nuspec file with replacement tokens:


<?xml version="1.0"?>
<package >
<metadata>
<id>$id$</id>
<version>$version$</version>
<title>$title$</title>
<authors>$author$</authors>
<owners>$author$</owners>
<requireLicenseAcceptance>false</requireLicenseAcceptance>
<description>$description$</description>
<copyright>$copyright$</copyright>
<tags>tag1 tag2</tags>
<dependencies>
<dependency id="Newtonsoft.Json" version="11.0.2" />
<dependency id="RestSharp" version="106.3.1" />
<dependency id="Selenium.Support" version="3.14.0" />
<dependency id="Selenium.WebDriver" version="3.14.0" />
</dependencies>
</metadata>
</package>

view raw

example-nuget-quick-ref-nuspec.xml

hosted with ❤ by GitHub

Resources

 

Behavior-Driven Blasphemy

This is my 100th post on Automation Panda! I’m thrilled to see how much this blog has grown and how many people it has helped. For such a monumental occasion, I have chosen to voice a rather controversial opinion about test automation.

Behavior-driven development seems to be the software testing buzzword of the decade. What started as a refinement of test-driven development by developers in Europe and the UK quickly became the big process fad of the 2010’s. The Cucumber project (now 10 years old) developed or inspired Gherkin-based test automation frameworks in all the major programming languages. Companies started requiring Given-When-Then format for acceptance criteria and test scenarios. Three Amigos meetings became standard calendar fixtures during sprints. Organizations that once undertook “Agile transformations” now have similar initiatives for BDD. For better or worse, BDD exists and cannot be ignored.

The dogmatic benefits of BDD are better collaboration and automation. However, leaders frequently insist that Gherkin-style test frameworks add value only when paired with practices like Example Mapping. “BDD is a process, not a tool,” is a common mantra. “Otherwise, the Gherkin just gets in the way.” Although I wholeheartedly agree that behavior-driven practices add significant value to the development process, I nevertheless espouse a rather blasphemous opinion:

BDD test automation frameworks are better than traditional frameworks for black box functional testing even when BDD processes are not followed.

What Exactly Are You Saying?

My claim is that behavior-driven test frameworks like Cucumber, SpecFlow, and behave are significantly better than traditional xUnit-style frameworks for testing live features. For example, I would rather use SpecFlow than NUnit for testing a Web app with Selenium WebDriver, whether or not the other two Amigos are with me. The resulting automation code has better structure, readability, and reusability.

I’m not saying that teams shouldn’t do BDD practices, and I’m not saying that the Three Amigos should be separated. Collaboration is key to success, and BDD really helps. Example Mapping is one of the most useful practices a development team can do. I’m also not saying that BDD frameworks should be used for all testing purposes – they are poorly suited for unit testing and for performance testing.

Objection!

I find myself very lonely in this opinion. BDD leaders repeatedly insist that BDD is not about testing and automation:

The most outspoken BDDers (mostly coalescing around the Cucumber community) have largely moved their focus to the collaboration benefits, almost forsaking the automation benefits. (This may not necessarily be true, but it appears that way based on the literature and materials floating on the Web.) That outlook is somewhat disingenuous because the main tools supporting BDD are, in fact, test frameworks.

BDD also has outspoken opponents – it’s love or hate. I’ve personally spoken with several engineers who despise Gherkin-based frameworks. “I can see how it would be valuable when a whole team embraces behavior-driven practices,” many have told me, “but otherwise, the Gherkin layer just gets in the way of automation.” I’ve heard it called “plaster” and “garbage.” Engineers just want to code their tests. And code should always be readable, right?

hqdefault

Testing is an inherently opinionated space. People can never seem to agree on things.

The Bigger Picture

Test automation must be developed regardless of any specific development practices, and its architecture must stand firmly in its own right. Unfortunately, both sides miss the bigger picture:

The best solution for test automation is a domain-specific language.

A domain-specific language (DSL) is a programming language with a purpose. It is designed to handle very specific needs, rather than general-purpose programming. For example:

  • SQL is a DSL for database queries.
  • XPath is a DSL for finding elements in an XML document.
  • YAML is a DSL for object serialization.

Gherkin is also a DSL – for behavior specification.

Domain-specific languages naturally suit test automation due to the clear difference between test cases and test code. Test cases are procedures that exercise product behavior. Anyone can write a test case. They are dictated or explained in plain language. Test code, however, is the software implementation of test cases. Test code handles function calls, logging, exceptions, and all those other little programming details that help run tests. A test automation DSL separates those concerns: test cases are written in a special language, and the interpreter handles repetitive, low-level details. Some type of extensions can handle product-specific interactions. The purpose of a language is to effectively express intention – and the intention is to test the product.

To truly achieve an optimal solution, however, the DSL and its interpreter must be treated as part of the automation software, just like the test cases and extensions. Remember, a language’s interpreter is just another piece of software. The interpreter is part of the separation of concerns and the single responsibility principle. Concerns that would typically be handled by classes and functions in traditional test code should be moved to the interpreter. For example, the interpreter should automatically log every test case step, rather that forcing the author to write explicit logging statements.

When I worked at NetApp years ago, I implemented a DSL to test platform-level features of our operating system. I called it DS – short for “Design Steps” (from HP ALM) (but also not without an affinity for the Nintendo DS). NetApp’s entire test automation code was developed in Perl at the time, so I implemented the DS interpreter in Perl to reuse existing libraries. DS test cases were typically only a dozen lines long each, and DS expressions could call specially-written Perl modules directly for complete extendability. During the first big release using DS, my team saved countless hours of automation development time as compared to the previous release while delivering a higher number of tests. I also did this before I had ever heard of BDD.

Unfortunately, most teams have neither the time to develop their own testing DSL nor the understanding of compiler theory to build it right. And if they were given such a language, they typically limit themselves to the provided implementation instead of taking ownership to extend the language for their needs.

nintendo-ds-1

The original Nintendo DS. Fun times!

Who Truly Misunderstands Gherkin?

Enter Gherkin: the world’s first major general-purpose, off-the-shelf language for test automation. It is general enough to cover any case through its plain language steps, yet specific enough to standardize tests. Users don’t need to be compiler theory experts – they just make up their own step names and provide the definition code to execute them. Early BDD projects like JBehave and Cucumber packaged an interpreter as a test framework and delivered it to a testing world still stuck on JUnit. The need for a testing DSL was there, whether or not the BDD folks meant to serve it.

Cucumber-ites frequently bemoan that their framework is misunderstood by the masses. They shudder to see teams using their framework purely for test automation. However, Cucumber effectively lowered the entry barrier for teams to make their own testing DSLs. Kodak did the same thing for film: they made it cheap and standard so anyone could be a photographer. Not everyone who uses a BDD framework misunderstands its purpose: some (like me) just see an alternative value proposition than what is preached by orthodox BDD practitioners. Gherkin fills a need that nobody knew. Its popularity validates that claim.

Benefits Apart from Process

Using a BDD framework adds much value to testing and development even without BDD processes. Below are just a handful of benefits:

  1. Focus first on good scenarios. Gherkin forces authors to think before they code.
  2. Faster automation development. Gherkin steps are reusable and parametrizable.
  3. Stronger structure. Engineers know where to put things in the framework.
  4. Test understandability. Anyone can read scenarios because they are written in plain language. Business people can help. New people can pick it up fast.
  5. Test sharing. Feature files can be shared apart from test code, which can be helpful for business partners.
  6. Test similarity. Tests all look the same. Team members can more easily help each other.
  7. Clearer failures. When a scenario fails, reports show exactly what step failed.
  8. Simpler bug reports. Use scenario steps as instructions to reproduce the failure.
  9. 2-phase test reviews. Review Gherkin first and then test code second to make sure the test cases are good before implementing the wrong things.
  10. BDD enablement. Using a BDD framework opens the door for a team to embrace better behavioral practices in the future.

I wrote about these advantages before:

Case Studies

I’m also not the only one who finds value in BDD test frameworks outside of the full BDD process. Below are five case studies.

radish

radish is a Python test framework inspired by Cucumber. Its DSL syntax is a superset of Gherkin that adds preconditions, loops, variables, and expressions. These language additions indicate a bias towards automation because they enable engineers to write tests more programmatically, albeit in a Gherkin-ese way.

Karate

Karate is a test framework with a full DSL based on Gherkin with steps specifically tailored to Web service calls. Although it is implemented in Java, testers do not need to do any Java programming to write complete tests cases from day one. Peter Thomas, the creator of Karate, unabashedly declares that Karate does not truly adhere to BDD but nevertheless uses Cucumber for its automation benefits. (Note: Karate is working to move completely off of Cucumber. See GitHub issue #444.)

REST Assured

REST Assured is a Java package for testing REST APIs. Unlike Karate, REST Assured provides a fluent syntax (and not a DSL) for writing service calls directly in Java code. The fluent syntax is based on Gherkin: given() a request spec is created, when() the call is made, then() verify the response. Although REST Assured is not a full testing framework, it nevertheless pulls inspiration from BDD frameworks for order and structure.

Cycle

Cycle is a BDD-focused test automation platform from Cycle Labs for testing Web, terminal, and desktop apps. Cycle is unique because it provides out-of-the-box steps for all types of supported testing so that no programming experience is required. Testers write feature files using Cycle 2.0’s slick new Electron app. Scenarios are written in CycleScript, a Gherkin-ese language with additions like variables and sub-scenario calls. Steps tend to be imperative, but that’s the tradeoff for not requiring lower-level programming.

Hexawise

Hexawise is a combinatorial testing tool designed to maximize coverage with minimal test counts by smartly joining feature variations. It helps testers write better tests with less redundancy and fewer gaps. Although Hexawise has historically assisted manual testers, it also can generate Gherkin feature files for test variations.

mexican-coast-dried-sea-cucumber

Not all cucumbers are the same. Above is a sea cucumber.

Good Enough?

Gherkin-based test frameworks are not perfect, but they do provide good structure. They gained popularity outside of the pure BDD movement because they genuinely added value to testing and automation. Like any other tool, teams will use them in both good and bad ways. (Trust me, I’ve seen scary Gherkin.)

It’s interesting to see how groups outside the Cucumber diaspora are attempting to solve the limitations of pure Gherkin. Each case study above showed a unique path. Clearly, the test automation problem has not yet been completely solved, but current BDD frameworks are the best off-the-shelf solutions we have until a new software testing movement comes along.