Testing Web Services with Karate

Karate is a relatively new open source framework for testing Web services. Even though Karate is written in Java, its main value proposition is that testers don’t need to do any Java programming in order to write fully automated tests. Instead, testers use a Gherkin-like language with steps for making requests and validating responses. It’s like Cucumber with out-of-the-box Web API steps! There are a bunch of other nifty features, too.

This article is my quick-start guide for Karate. As a prerequisite, make sure you understand how Web services work (like REST APIs). Knowing BDD will also help.

My System

Since Karate is an open-source Java project, it can run almost anywhere. Here’s my system config:

  • macOS 10.13.6 (High Sierra)
  • Java 1.8.0_191
  • Apache Maven 3.6.0
  • Karate 0.9.0

Warning: I initially attempted to run my Karate project using Java 11.0.1, but I repeatedly hit SSL handshake exceptions despite trying many fixes. Downgrading to Java 8 fixed the errors. See Issue #617.

Project Setup

I created my Karate project using the Maven archetype since I am quite familiar with Maven. I named my project “firstchop”:

mvn archetype:generate \
  -DarchetypeGroupId=com.intuit.karate \
  -DarchetypeArtifactId=karate-archetype \
  -DarchetypeVersion=0.9.0 \
  -DgroupId=com.automationpanda \
  -DartifactId=firstchop

The directory layout was standard for Java Maven / Cucumber-JVM projects. (In fact, Karate was based on Cucumber-JVM until version 0.8.0.) Interestingly, the Karate docs recommend placing feature files under src/test/java instead of src/test/resources.

IDE

The Karate docs recommend using Eclipse or IntelliJ IDEA for developing Karate tests. Both IDEs offer support for JUnit and Cucumber, which Karate can leverage not only for editing but also for running tests. I’d probably use IntelliJ IDEA for serious testing.

However, for my initial exploration, I chose to use Visual Studio Code. Why?

  • It’s fast and easy.
  • It has good support for Java and Gherkin.
  • It has a file explorer and an integrated terminal.
karate-ide

Here’s what my Karate project looked like inside Visual Studio Code. Nice!

The Examples

The archetype project included example tests in the users.feature file. The first scenario from the file, copied below, tests getting users from the JSONPlaceholder REST API:

Feature: sample karate test script

Background:
* url 'https://jsonplaceholder.typicode.com'

Scenario: get all users and then get the first user by id

Given path 'users'
When method get
Then status 200

* def first = response[0]

Given path 'users', first.id
When method get
Then status 200

Anyone familiar with Cucumber will immediately recognize the Given/When/Then format for scenarios. However, Karate’s standard steps make the language more powerful than raw Gherkin:

  • Given steps build requests
  • When steps make request calls
  • Then steps validate responses
  • Catch-all steps (*) provide additional directives, like setting variables

All steps are quite concise. The Java implementation is essentially hidden from the tester (unless, of course, they want to plunge into the framework’s lower levels). Furthermore, this scenario shows how to use data from one response as the input for a second request.

Running Tests

Since I chose to use Maven and Visual Studio Code, the easiest way to run tests without any additional configuration was through the command line using “mvn test”. The example tests came with an ExamplesTest.java file that will run all feature files in the package when it is discovered during Maven’s “test” phase. The project defaulted to using JUnit 4, but JUnit 5 is also supported.

Running the tests will print many lines to the console. Every request and response will be printed. Below is the tail end of a successful run for users.feature:

$ mvn test
.
.
.
---------------------------------------------------------
feature: classpath:examples/users/users.feature
scenarios:  2 | passed:  2 | failed:  0 | time: 1.5232
---------------------------------------------------------
HTML report: (paste into browser to view) | Karate version: 0.9.0
file:/path/to/firstchop/target/surefire-reports/examples.users.users.html
---------------------------------------------------------

Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.09 sec

Results :

Tests run: 2, Failures: 0, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  5.635 s
[INFO] Finished at: 2018-12-09T23:32:29-05:00
[INFO] ------------------------------------------------------------------------

Karate can generate helpful test reports, too. The project generates JUnit reports by default, but other report formats like Cucumber reports are also possible.

karate-report-1

The JUnit HTML report shows a step-by-step log for each scenario.

karate-report-2

Full requests and responses are automatically logged, which is great for debugging.

karate-report-3

Failures are visually easy to identify. Above, I hacked the example to deliberately fail.

My Scenarios

Seeing examples is great, but I wanted to write my own scenarios with a real Web service for some hands-on experience. So, I wrote a test for Recipe Puppy, a search engine that provides links to recipes for input ingredients:

Feature: Recipe Puppy

Background:
* url 'http://www.recipepuppy.com'

Scenario Outline: Get a recipe for <ingredient>

Given path 'api'
And params {i: '<ingredient>'}
When method get
Then status 200
And match response contains {results: '#array'}
And match response.results[*] contains
  """
  {
    title: '#string',
    href: '#string',
    ingredients: '#regex <ingredient>',
    thumbnail: '#string'
  }
  """

  Examples:
    | ingredient |
    | tomato     |
    | pepperoni  |
    | cheese     |

 

Here are a few things I learned while writing this new feature file:

  • Scenario Outlines are supported, but data-driven features are preferred.
  • JSON objects can be used as step arguments or as variables.
  • Matching syntax is simple but sophisticated.
  • Markers like “#array” support fuzzy matching when specific values are unknown.
  • Multi-line JSON expressions need block quotes.

This new scenario ran just as successfully as the examples!

Standalone Execution

Even though writing tests using Karate’s domain-specific language does not require Java development skills, setting up the full Karate project does. Thankfully, Karate provides a standalone JAR that can run feature files without any other dependencies or configuration. It simply takes in paths to feature files, runs them, and generates Cucumber reports. The standalone JAR would be a good option for testers who don’t have strong programming skills.

karate-cuke-report

This was the Cucumber report generated by running my Recipe Puppy test with the standalone JAR.

Other Features

Despite being a fairly young project (with a GitHub creation date of February 7, 2017), Karate is full of nifty features that I have yet to try:

  • Parallel test execution
  • A mock servlet
  • A UI for visually debugging scripts
  • Reading files of many types into variables
  • Calling a feature file from another feature file
  • Calling JavaScript code
  • Calling Java code
  • Using scenarios as Gatling performance tests

Project contributors are also experimenting to extend Karate to test browser, mobile, and desktop UIs using Selenium WebDriver.

Analysis

Overall, Karate is a great tool for testing Web services. It handles all of the programming implementation details so that testers can focus more on testing. Its syntax is concise, clear, and versatile. Its native support for JSON object makes request and response handling feel natural. The GitHub docs are on-point. Anyone testing REST APIs should give it serious consideration.

Even though Karate uses Gherkin-like syntax, it is not truly a behavior-driven test framework. Instead, it simply leverages the strengths of Cucumber – readability, reusability, and tooling – to make automated Web service testing easier. I have long stated that one of the best solutions to test automation challenges is to create a domain-specific testing language that automatically handles low-level details. (See Behavior-Driven Blasphemy.) The Karate project validates my claim. Karate’s DSL acts much more like a testing language than a business-oriented specification language. Its Gherkin keywords simply provide structure and familiarity to its steps. And it’s perfectly okay that Karate isn’t “pure BDD” – just ask the project’s creator.

With that said, I would argue that a tester probably needs basic programming skills to be successful with Karate. Execution needs Java setup no matter what, and feature files are really just fancy test scripts. And Web service APIs are inherently code-y.

I look forward to seeing how the Karate project grows, especially if/when WebDriver-based steps are added to the language.

Resources

 

Frameworks: All-in-One or Piece-by-Piece?

Software frameworks are great because they apply the principle of Separation of Concerns. A framework’s tools and code handle a specific need in a standard way for developers to write other code more easily. For example:

  • Web frameworks support receiving requests and sending responses.
  • Test frameworks include test case structure, runners, and reporting mechanisms.
  • Logging frameworks control how messages are gathered and stored.
  • Dependency injection frameworks create and manage object instances.

Recently, a question hit me: How far should a framework go to separate concerns? Should a framework try to do everything all-in-one, or should it behave more like a library that focuses on doing one thing well?

Let’s look at Python Web frameworks as an example. Django, the “Web framework for perfectionists with deadlines,” provides everything a developer could want out of the box. Flask, on the other hand, is a “microframework” that prides itself on minimalism: any extras must be handled by extensions or other packages. The differences between the two become clear when comparing some of their features:

Feature Django Flask
HTTP Requests and Routing Included Werkzeug (bundled)
Templates Included Jinja2 (bundled)
Forms Included None (Flask-WTF)
Object-Relational Mapping (ORM) Included None (SQLAlchemy)
Security Included None (Flask-Security)
Localization Included None (Flask-Babel)
Admin Interface Included None

Clearly, Django is all-in-one, while Flask is piece-by-piece. To make a serious Flask app, developers must pull in many extra pieces. There are many other frameworks with similar competitions:

  • JavaScript testing: Jasmine vs. Mocha
  • JavaScript development: Angular vs. React
  • Java BDD testing: Serenity vs. Cucumber-JVM

I think each approach has its merits. All-in-one frameworks are more convenient to use, especially for beginners. For developers who are new to a domain or just need to get something up fast, all-in-ones are the better choice. They come with all units already integrated together, and they often have good documentation. However, developing an all-in-one framework takes much more work because it covers multiple concerns. Developers may also feel shoehorned into the framework’s way of doing things. All-in-ones typically dictate what they believe to be the “best” solution.

Piece-by-piece frameworks require more expertise but offer greater flexibility. Developers can pick and choose the pieces they need, and they can change the packages used by the solution more easily. Found a better ORM? Not a problem. Need to localize the site in Chinese? Add it! Solutions can avoid excess weight and stay nimble for the future. The big challenge is successful integration. Furthermore, a library or framework for a singular concern tends to solve the concern in better ways simply because project contributors give it exclusive focus. The more I learn about a space, the more I lean towards a piece-by-piece approach.

As always, pick frameworks based on the needs at hand. For example, I like to use Django to make websites for my wife’s small businesses because the admin interface is just so convenient for her, even though I could get away with Flask. However, I’ll probably pick Mocha (piece-by-piece) over Jasmine (all-in-one) whenever I return to JavaScript testing.

PyGotham 2018 Reflections

As only New Yorkers know, if you can get through the twilight, you’ll live through the night. (Dorothy Parker)

PyGotham 2018 was my first PyGotham conference and my third conference of the year. I first heard about PyGotham when Jon Banafato and Dan Crosta, two of the organizers, invited me to submit proposals when we met at the PyCon 2018 Instagram dinner. I enjoyed the conference very much, but my experience was much different than the previous two conferences.

A Tourist Day in New York

PyGotham is hosted in New York City. I booked my flight a day before the conference started to make sure that I’d be on time. My wife also came along for the trip. That meant Thursday, October 4 was a tourist day! Our flight arrived in EWR at about 10:30am. We took a commuter train to Penn Station in Manhattan and walked a block to our hotel, Hotel Pennsylvania. Unfortunately, we couldn’t check into our room until 3pm, so we dropped our bags (for $15) and went along our way.

Our first stop was lunch at Xi’an Famous Foods. XFF is a fast casual restaurant chain for authentic Chinese food local to NYC. We ordered two noodle dishes (one liangpi “cold skin” noodles and the other the wide hand-pulled “biang biang” noodles), a pulled pork “burger,” and hawberry tea. Our meal was absolutely delicious.

We spent our afternoon at the National September 11 Memorial & Museum. I was just starting 8th grade when the terrorist attacks happened, but I remembered all the details clearly. The outdoor memorial – a sunken fountain for each tower’s square base – showed both emptiness and resolve. The museum was built into the underground foundations for the towers, essentially as a cave. The starkness of the concrete, the use of light and shadow, and the pacing of exhibits all highlighted the tragedy of 9/11/2001. Every American should visit the site if they are able.

We checked into our hotel before dinner that evening. And then we learned why Hotel Pennsylvania has only 2.7 stars on Google: it’s nasty. The hotel is 99 years old and looks like it. The carpets are filthy, the lights are dim, and most of the building smells damp. I chose to stay there because the conference was held there, but I don’t think I will be returning.

One plus side of Hotel PA is that it is right next to Koreatown. My wife and I ate dinner at Five Senses, a trendy Korean restaurant with a line nearly out the door. We then walked around a bit to see the hustle and bustle: we tried Gong Cha bubble tea, got breakfast for the next morning at H-Mart, and bought some face masks at Kosette Beauty Market. A Korean friend of ours was jealous!

The Conference

The conference itself lasted two days: Friday, October 5 and Saturday, October 6. About 600 people attended over the two days. By this time, I was well acquainted with Python conference formats. Each day opened with a keynote, followed by several talks in a few rooms with breaks in between. I reconnected with old friends like Trey Hunner and Gabriel Boorse and also made a few new ones.

The opening keynote was “Software Freedom and Ethical Technology” by Karen M. Sandler. Karen explained why free, open source software is foundational for the ethical use and development of technology. She used the example of her Implantable Cardioverter Defibrillator (ICD). Nearly all modern ICDs have wireless connectivity, which renders them vulnerable to hacking. Since the source code in this device and so many others in the Internet of things is closed, individuals are unable to make changes for customization or protection. Karen’s talk was definitely thought-provoking.

My favorite talk, by far, was Trey Hunner’s “Python Oddities Explained”. Trey dove deep into the way variables, data structures, and scope work in Python 3. This talk was particularly helpful for me because I work in several languages and sometimes assume how small details work.

My talk, “Egad! How Do We Start Writing (Better) Tests?“, explained how to build a functional test automation solution in Python from the ground up. It is the same talk I delivered at PyOhio 2018, though I hope it turned out better the second time!

Attendance was pretty good, too.

IMG_3638

Conference room panorama for my talk

Video recordings for all talks can be found on the PyGotham 2018 YouTube channel.

PyGotham had a stronger business vibe than the other Python conferences I attended. Many attendees came as part of company groups. (I heard rumors that about a hundred came from Bloomberg.) Speakers more openly talked about their companies and specific projects. This all made sense considering the location being NYC.

A number of vendors also had tables with goodies to give away. Bank of America, Buzzfeed, Venmo, Linode, Clover, and Django Girls all showed up, along with others I cannot remember off the top of my head. I netted two t-shirts (in addition to two PyGotham t-shirts), a notebook, some silly putty from Linode, and a handful of stickers.

After Hours

On Friday night after the conference, my wife and I took the Subway to Brooklyn and walked the Brooklyn Bridge back to Manhattan. The view of the city skyline was spectacular. We then hopped the Subway up to the Upper West Side for drinks with my friend Evan. On the way back to the hotel, we stopped at a dim sum restaurant for a late night dinner.

We didn’t have time to explore New York on Saturday night because we had an overnight flight to catch: we were going to London, England for the week after the conference! We checked out of the hotel, caught the Long Island Rail Road to JFK, and then flew Norwegian Air across the pond.

IMG_3644

Dim sum? Dim yum!

Takeaways

During PyGotham 2018, I spent a lot of time pondering software testing and automation. I like to use conferences as my time to disconnect from immediacy and think big thoughts.

One thought I formed was the nuance between testing and change detection. A friend of Gabriel’s posed an idea for REST API service testing. Rather than write a bunch of test cases that check responses, why not simply gather responses and diff them against previous responses? Request automation would still be needed, but response assertions could basically be skipped. Any changes would be identified and reported. This approach struck me as novel and useful but not comprehensive. Change detection will reveal when a code change results in a feature change, but it lacks the moral calculus of true testing because it cannot determine if the change is good or bad. It must yield the judgment to a higher authority, like a developer, a tester, or even an AI agent (which arguably further begs the question of goodness). Nevertheless, I think change detection can be very useful in conjunction with testing, either as a tool to assist manual testing or as an extension to an automation framework.

Another thought I had was to notice the widening gap between companies over testing maturity. Some companies are on the bleeding edge with the latest technologies, tools, and frameworks. They have multi-stage pipelines with countless automated tests and several daily deployments. Other companies are stuck in the past, still suffering through “Agile transformations” and failing tests. There are now several vendors pitching advanced testing solutions, like Sauce Labs for cross-browser testing and Applitools for visual testing. I wonder how many of these vendors are too far ahead for companies with less mature testing to catch up.

For the first time at a conference, I felt serious self doubt. Am I behind the times as a Software Engineer in Test? Is my method of software testing already antiquated? Am I truly a developer, or do I just pretend? Is there a future for the SET role? Will my ideas add value to the field? Will I have time to pursue all of my professional and technical desires? Am I good enough to be here at PyGotham? I felt a bit lonely and withdrawn. My malaise did not sour my conference experience, but it did come unexpectedly. Nevertheless, I must use it as impetus to become a better me.

Final Thoughts

I’m thankful for the opportunity I had to speak at PyGotham 2018. I thank the conference organizers for accepting my talk proposal, as well as PrecisionLender for sponsoring my travel costs. I’d love to return for PyGotham 2019 if possible!

Python Testing 101: pytest-bdd

Warning: If you are new to BDD, then I strongly recommend reading the BDD 101 series before trying to use pytest-bdd. Also, make sure that you are already familiar with the pytest framework.

Overview

pytest-bdd is a behavior-driven (BDD) test framework that is very similar to behaveCucumber and SpecFlow. BDD frameworks are very different from more traditional frameworks like unittest and pytest. Test scenarios are written in Gherkin “.feature” files using plain language. Each Given, When, and Then step is “glued” to a step definition – a Python function decorated by a matching string in a step definition module. This means that there is a separation of concerns between test cases and test code. Gherkin steps may also be reused by multiple scenarios.

pytest-bdd is very similar to other Python BDD frameworks like behave, radish, and lettuce. However, unlike the others, pytest-bdd is not a standalone framework: it is a plugin for pytest. Thus, all of pytest‘s features and plugins can be used with pytest-bdd. This is a huge advantage!

Installation

Use pip to install both pytest and pytest-bdd.

pip install pytest
pip install pytest-bdd

Project Structure

Project structure for pytest-bdd is actually pretty flexible (since it is based on pytest), but the following conventions are recommended:

  • All test code should appear under a test directory named “tests”.
  • Feature files should be placed in a test subdirectory named “features”.
  • Step definition modules should be placed in a test subdirectory named “step_defs”.
  • conftest.py files should be located together with step definition modules.

Other names and hierarchies may be used. For example, large test suites can have feature-specific directories of features and step defs. pytest should be able to discover tests anywhere under the test directory.

[project root directory]
|‐‐ [product code packages]
|-- [test directories]
|   |-- features
|   |   `-- *.feature
|   `-- step_defs
|       |-- conftest.py
|       `-- test_*.py
`-- [pytest.ini|tox.ini|setup.cfg]

Note: Step definition module names do not need to be the same as feature file names. Any step definition can be used by any feature file within the same project.

Example Code

An example project named behavior-driven-python located in GitHub shows how to write tests using pytest-bdd. This section will explain how the Web tests are designed.

The top layer for pytest-bdd tests is the set of Gherkin feature files. Notice how the scenario below is concise, focused, meaningful, and declarative:

@web @duckduckgo
Feature: DuckDuckGo Web Browsing
  As a web surfer,
  I want to find information online,
  So I can learn new things and get tasks done.

  # The "@" annotations are tags
  # One feature can have multiple scenarios
  # The lines immediately after the feature title are just comments

  Scenario: Basic DuckDuckGo Search
    Given the DuckDuckGo home page is displayed
    When the user searches for "panda"
    Then results are shown for "panda"

Each scenario step is “glued” to a decorated Python function called a step definition. Step definitions are written in Python test modules, as shown below:

import pytest
from pytest_bdd import scenarios, given, when, then, parsers
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

# Constants

DUCKDUCKGO_HOME = 'https://duckduckgo.com/'

# Scenarios

scenarios('../features/web.feature')

# Fixtures

@pytest.fixture
def browser():
    b = webdriver.Firefox()
    b.implicitly_wait(10)
    yield b
    b.quit()

# Given Steps

@given('the DuckDuckGo home page is displayed')
def ddg_home(browser):
    browser.get(DUCKDUCKGO_HOME)

# When Steps

@when(parsers.parse('the user searches for "{phrase}"'))
def search_phrase(browser, phrase):
    search_input = browser.find_element_by_name('q')
    search_input.send_keys(phrase + Keys.RETURN)

# Then Steps

@then(parsers.parse('results are shown for "{phrase}"'))
def search_results(browser, phrase):
    links_div = browser.find_element_by_id('links')
    assert len(links_div.find_elements_by_xpath('//div')) > 0
    search_input = browser.find_element_by_name('q')
    assert search_input.get_attribute('value') == phrase

Notice how each Given/When/Then step has a function with an appropriate decorator. Arguments, such as the search “phrase,” may also be passed from step to function. pytest-bdd provides a few argument parsers out of the box and also lets programmers implement their own. (By default, strings are compared using equality.) One function can be decorated for many steps, too.

pytest fixtures may also be used by step functions. The code above uses a fixture to initialize the Firefox WebDriver before each scenario and then quit it after each scenario. Fixtures follow all the same rules, including scope. Any step function can use a fixture by declaring it as an argument. Furthermore, any “@given” step function that returns a value can also be used as a fixture. Please read the official docs for more info about fixtures with pytest-bdd.

One important, easily-overlooked detail is that scenarios must be explicitly declared in test modules. Unlike other BDD frameworks that treat feature files as the main scripts, pytest-bdd treats the “test_*.py” module as the main scripts (because that’s what pytest does). Scenarios may be specified explicitly using scenario decorators, or all scenarios in a list of feature files may be included implicitly using the “scenarios” shortcut function shown above.

To share steps across multiple feature files, add them to the “conftest.py” file instead of the test modules. Since scenarios must be declared within a test module, they can only use step functions available within the same module or in “conftest.py”. As a best practice, put commonly shared steps in “conftest.py” and feature-specific steps in the test module. The same recommendation also applies for hooks.

Scenario outlines require special implementation on the Python side to run successfully. Unfortunately, steps used by scenario outlines need unique step decorators and extra converting. Please read the official docs or the example project to see examples.

Test Launch

pytest-bdd can leverage the full power of pytest. Tests can be run in full or filtered by tag. Below are example commands using the example project:

# run all tests
pytest

# filter tests by test module
# note: feature files cannot be run directly
pytest tests/step_defs/test_unit_basic.py
pytest tests/step_defs/test_unit_outlines.py
pytest tests/step_defs/test_unit_service.py
pytest tests/step_defs/test_unit_web.py

# filter tests by tags
# running by tag is typically better than running by path
pytest -k "unit"
pytest -k "service"
pytest -k "web"
pytest -k "add or remove"
pytest -k "unit and not outline"

# print JUnit report
pytest -junitxml=/path/for/output

pytest-bdd tests can be executed and filtered together with regular pytest tests. Tests can all be located within the same directory. Tags work just like pytest.mark.

All other pytest plugins should work, too. For example:

Pros and Cons

Just like for other BDD frameworks, pytest-bdd is best suited for black-box testing because it forces the developer to write test cases in plain, descriptive language. In my opinion, it is arguably the best BDD framework currently available for Python because it rests on the strength and extendability of pytest. It also has PyCharm support (in the Professional Edition). However, it can be more cumbersome to use than behave due to the extra code needed for declaring scenarios, implementing scenario outlines, and sharing steps. Nevertheless, I would still recommend pytest-bdd over behave for most users because it is more powerful – pytest is just awesome!

Speaking Pythonese

The Python community, like many groups, has its own language – and I don’t mean just Python. There are many words and phrases thrown around that may confuse people new to Python. I originally shared some terms in my article, Which Version of Python Should I Use?, but below are some more of those colloquialisms for quick reference:

Word or Phrase Meaning
Anaconda
  • Open-source implementation of Python (and R)
  • Meant for data scientists
  • Uses the conda package manager
Benevolent Dictator for Life (BDFL)
  • Guido van Rossum
  • The inventor of Python
  • Resigned in July 2018 but remains BDFL Emeritus
The CheeseShop
  • a fun code name for the Python Package Index
Class
  • A programming definition for creating objects
  • Combines attributes (variables) and behaviors (methods)
  • Useful for reusing code
Conda
  • Package manager for Python and other languages
  • Part of the Anaconda project
Core Developer
  • A developer who has commit privilege to the CPython codebase
  • Very few Pythonistas are core developers
CPython
  • The default and most widely used implementation of the Python language
  • Implemented in C
Django
  • A batteries-included Python Web framework for perfectionists with deadlines
  • Offers many features out of the box
  • The most popular Python framework in 2017
  • Size: Flask < Pyramid < Django
Flask
  • A microframework for Python Web development
  • Uses Werkzeug and Jinja2
  • Super-minimalist
  • Size: Flask < Pyramid < Django
Function
  • A definition for a callable subroutine
  • May take inputs
  • May return outputs
  • Great for code reuse
  • Functions are first-order values
The Hitchhiker’s Guide to Python
  • A popular online guide to Python
  • Opinionated
  • Shares many good best practices
Jinja2
  • A Python template engine
  • Inspired by Django’s templates
Jupyter
  • A project for interactive Python development
  • “Jupyter Notebooks” allow programmers to dynamically rewrite and rerun Python code, and then share code easily with others
  • Popular with data scientists
Module
  • A Python source code file containing definitions and statements
  • Every .py file is a module
  • May be imported by other modules to reuse code
NumPy
  • A popular Python package for scientific computing
Pandas
  • A popular Python package for data analysis
pip
  • The PyPA-recommended tool for installing Python packages
  • Command: “pip install “
  • Recursive acronym for “pip installs packages”
PyBites
  • A community of Pythoneers who improve their skills through code challenges
PyCharm
  • A popular Python IDE developed by JetBrains
  • Offers great development features
  • Has a free Community Edition and a paid Professional Edition
PyCon
  • The annual Python conference held in North America
  • GO – it will change your life!
  • Several other conferences are held worldwide
PyPy
  • An alternative Python implementation
  • Known for speed, memory usage, and compatibility
  • Good alternative to CPython for high performance workloads
Pyramid
  • A Python Web framework
  • Start small, finish big, stay finished
  • Provides many parts but not everything (such as ORM)
  • Size: Flask < Pyramid < Django
pytest
  • A lightweight-yet-powerful Python test framework (and arguably the best)
Python 2
  • The old version of Python
  • Will reach end-of-life in 2020
  • Final version will be 2.7.x
  • Please upgrade to version 3
Python 3
  • The current version of Python
  • Most packages now support Python 3
  • Has incompatibilities with Python 2
  • Please don’t use Python 2
Python Enhancement Proposal (PEP)
  • Official proposals for enhancing the Python language
Python Package Index (PyPI)
The Python Software Foundation (PSF)
  • Non-profit organization
  • Keeps Python going strong
  • Support them!
Pythoneer
  • A programmer who uses Python to solve problems
  • Styled off the word “engineer”
Pythonic
  • Describes idiomatic code for Python
  • Closely related to conciseness, readability, and elegance
  • Highly recommended
  • Follow style guidelines to learn how to be Pythonic
Pythonista
  • Someone who loves the Python language
  • Often an advanced Python programmer
Sphinx
  • A popular Python tool to generate documentation
virtualenv
  • Tool to create isolated Python environments
  • Enables programmers to use different versions of Python and packages for different projects
  • Also see venv and pipenv
Web Server Gateway Interface (WSGI)
  • A specification for how Web servers forward requests to Web applications and frameworks
  • A core piece of Python Web development
  • See PEP-333 and PEP-3333
The Zen of Python
  • The list of guiding principles for Python’s design
  • Run “import this” to see them
  • See PEP-20

The Software Engineer in Test

I am a Software Engineer in Test (SET). Many people don’t know quite what that means, though. Developers frequently refer to me as a “tester” or “QA,” and a former director once thought I did DevOps. While my work covers these areas, they aren’t the main focus. Let’s clarify what it means to be a SET.

What is a Software Engineer in Test?

A “Software Engineer in Test” (a.k.a. “Software Development Engineer in Test”) is a software developer who develops software for testing: tools, frameworks, and automated tests. SETs focus primarily on automation for running tests quickly and repeatedly. Test automation is a software product: just as front-end developers write web pages and back-end developers write microservices, SETs write automated tests. The same practices and coding skills apply. I frequently say that a Software Engineer in Test must have the heart of a developer.

So they just write test scripts?

No. Never say this to a SET. Test automation involves much more than just “writing test scripts.” A serious testing solution requires serious design and effort. The top-level automation of a test case is usually just a small piece. SETs are responsible for:

  • Collaborating with developers and product owners
    • Contributing to planning and design
    • Reviewing product code
    • Formulating test scenarios
  • Developing test automation frameworks
  • Automating test scenarios using the frameworks
  • Knowing and using design patterns where appropriate
  • Setting up the infrastructure to run tests
    • Running tests in CI/CD
    • Running tests in parallel
    • Running tests with the appropriate test data
  • Setting up dashboards for reporting test results in real time
  • Teaching others good quality and testing practices
  • Developing tools to assist manual and exploratory testing

Saying that testing is “just scripting” belittles the role of the SET and underestimates the workload it requires.

How did this role start?

Software “tester” and “QA” roles have existed for decades, but the SET role first became distinct in the 2000s when large-scale test automation became both feasible and necessary. According to Wikipedia, Microsoft coined the title “Software Developer Engineer in Test” (SDET) in 2005, and others like Amazon and Apple quickly adopted it. Google coined the name “Software Engineer in Test” for the same type of role. I personally prefer the SET title over the SDET title simply because it is more concise.

How is it different from “QA” or “testing” roles?

To me, the SET role is distinct from the traditional “QA” or “testing” role. Software testers historically focused on manual testing and thus didn’t need strong programming skills. Their fortes were product domain knowledge, intuition, out-of-the-box thinking, test planning, and test system setup. And these certainly are important, indispensable skills! SETs, however, live in both the development and testing worlds. They use developer skills to provide software solutions for testing problems. Automation is much more central to the SET role. These days, almost all “testing” job openings are SET roles; manual-only test roles have been largely deprecated. Nevertheless, there will always be a need for manual testing because there are some problems a human can catch much more easily than a script (or an AI agent).

Personally, I avoid using the titles “QA” and “software tester” for myself because they don’t accurately describe all that I do. I also avoid the title “automation engineer” because, again, it is reductionist. I tackle software testing with the heart of a developer, and I set up test automation solutions from the ground up. I’m proud to be a software engineer who specializes in testing.

 

As a bonus, check out Test and Code episode 47, in which Brian Okken and I discuss what it means to be a Software Engineer in Test (among other topics).

 

Book Review: pytest Quick Start Guide

tl;dr

Title pytest Quick Start Guide
(available from Packt and Amazon)
Author Bruno Oliveira (@nicoddemus)
Publication 2018 Packt Publishing
Summary “Write better Python code with simple and maintainable tests” – a very readable guide for pytest’s main features.
Prerequisites Intermediate-level Python programming.

Summary

pytest Quick Start Guide is a new book about using pytest for Python test automation. Bruno Oliveira explains not only how to use pytest but also why its features are useful. Even though this book is written as a “quick start” introduction, it nevertheless dives deep into pytest’s major features. It covers:

  1. An introduction to pytest
  2. Why pytest is superior to unittest
  3. Setting up pytest with pip and virtualenv
  4. Writing basic tests
  5. How assertions really work
  6. Common command line arguments
  7. Marks and parametrization
  8. Using and writing fixtures
  9. Popular plugins
  10. Converting unittest suites to pytest

Example code is hosted on GitHub at PacktPublishing/pytest-Quick-Start-Guide.

Praises

This book is an easy introduction to test automation in Python with pytest. Readers should have intermediate-level Python skills, but they do not need previous testing or automation skills. The progression of chapters makes it easy to start quick and then go deeper. Oliveira has a very accessible writing style, too.

The unittest refactoring guide is a hidden gem. The unittest module is great because it comes bundled with Python, but it is also clunky and not very Pythonic. Not all teams know the best way to modernize their test suites. Oliveira provides many pieces of practical advice for making the change, at varying degrees of conversion. The big changes use pytest’s fixtures and assertions.

A Comparison to Python Testing with pytest

Python Testing with pytest by Brian Okken is another popular pytest book. Both books are great resources for learning pytest, but each approaches the framework from a unique perspective. How do they compare? Here’s what I saw:

Oliveira’s book Okken’s book
Oliveira’s book is a great introductory guide for pytest. Oliveira’s writing style makes the reader feel like the author is almost teaching in person. The book’s main theme is getting the reader to use the main features of pytest pragmatically for general testing needs. The main differences in content are the unittest refactoring guide and some of the plugins and fixtures covered. This book is probably the best choice for beginners who want to learn the basics of pytest quickly. Okken’s book is introductory but also a great manual for future reference. Okken’s writing style is direct and concise, which covers more material in fewer pages. The format for each chapter is consistent: for each idea: idea → code → output → explanation; exercises at the end. Okken also covers how to create and share custom pytest plugins. This book is probably the best choice for people who want to master the ins and outs of pytest.

Ultimately, I recommend both books because they are both excellent.

Takeaways

Reading books on the same subject by different authors helps the reader learn the subject better. I’ve used pytest quite a lot myself, but I was able to learn new things from both pytest Quick Start Guide and Python Testing with pytest. Reading how experts use and think about the framework makes me a better engineer. Different writing styles and different opinions also challenge my own understandings. (It’s also funny that the authors of both pytest books have the same initials – “B.O.”)

pytest is really popular. There are now multiple good books on the subject. It’s becoming the de facto test automation framework for Python, outpacing unittest, nose, and others. These days, it seems more popular to write a pytest plugin than to create a new framework.