Year: 2017

Python Testing 101: Introduction

Python is an amazing programming language. Loved by beginners and experts alike, it is consistently ranked as one of the most in-demand languages today. At PyData Carolinas 2016, Josh Howes, a senior data science manager at MaxPoint at the time, described Python like this (in rough paraphrase):

Python is a magical tool that easily lets you solve the world’s toughest problems.

I first touched Python back in high school more than a decade ago, but I really started using it and loving it in recent years for test automation. This 101 series will teach how to do testing in Python. This introductory post will give basic orientation, and each subsequent post will focus on a different Python test framework in depth.

Why Use Python for Testing?

As mentioned in another post, The Best Programming Language for Test Automation, Python is concise, elegant, and readable – the precise attributes needed to effectively turn test cases into test scripts. It has richly-supported test packages to deftly handle both white-box and black-box testing. It is also command-line-friendly. Engineers who have never used Python tend to learn it quickly.

The following examples illustrate ways to use Python for test automation:

A developer embedding quick checks into function docstrings.
A developer writing unit tests for a module or package.
A tester writing integration tests for REST APIs.
A tester writing end-to-end web tests using Selenium.
A data scientist verifying functions in a Jupyter notebook.
The Three Amigos writing Given-When-Then scenarios for BDD testing.

Remember, Python can be used for any black-box testing, even if the software product under test isn’t written in Python!

Python Version

Choosing the right Python installation itself is no small decision. For an in-depth analysis, please refer to Which Version of Python Should I Use? Tl;dr:

For white-box testing, use the matching Python version.
For black-box testing, use CPython version 3 if not otherwise constrained.

Unless otherwise stated, this 101 series uses CPython 3.

Picking a Framework

There are so many Python test frameworks that choosing one may seem daunting – just look at the Python wiki, The Hitchhiker’s Guide to Python, and pythontesting.net. Despite choice overload, there are a few important things to consider:

Consider the type of testing. Basic unit tests could be handled by unittest or even doctest, but higher-level testing would do better with other frameworks like pytest. BDD testing would require behave, lettuce, or radish.
Consider the supported Python version. Python 2 and 3 are two different languages, with Python 2’s end-of-life slated for 2020. Different frameworks have different levels of version support, which could become especially problematic for white-box testing. Furthermore, some may have different features between Python versions.
Consider support and development. Typically, it is best to choose mature, actively-developed frameworks for future sustainability. For example, the once-popular nose is now deprecated.

Future posts in this series will document many frameworks in detail to empower you, as the reader, to pick the best one for your needs.

Virtual Environments

A virtual environment (VE) is like a local Python installation with a specific package set. Tools like venv (Python 3.3+), virtualenv (Python 2 and 3), and Conda (Python 2 and 3; for data scientists) make it easy to create virtual environments from the command line. Pipenv goes a step further by combining VE management with simple-yet-sophisticated package management. Creating at least one separate VE for each Python project is typically a good practice. VEs are extremely useful for test automation because:

VEs allow engineers to maintain multiple Python environments simultaneously.
- Engineers can develop and test packages for both versions of Python.
- Engineers can separate projects that rely on different package versions.
VEs allow users to install Python packages locally without changing global installations.
- Users may not have permissions to install packages globally.
- Global changes may disrupt other dependent Python software.
VEs can import and export package lists for easy reconstruction.

VEs become especially valuable in continuous integration and deployment because they can easily provide Python consistency. For example, a Jenkins build job can create a VE, install dependencies from PyPI in the VE, run Python tests, and safely tear down. Once the product under test is ready to be deployed, the same VE configuration can be used.

Recommended IDEs

Any serious test automation work needs an equally serious IDE. My favorite is JetBrains PyCharm. I really like its slick interface and intuitive nature, and it provides out-of-the-box support for a number of Python test frameworks. PyCharm may be downloaded as a standalone IDE or a plugin for JetBrains IntelliJ IDEA. The Community Edition is free and meets most automation needs, while the Professional Edition requires a license. PyDev is a nice alternative for those who prefer Eclipse. Eric satisfies the purists for being a Python IDE written in Python. While all three have a plugin framework, PyCharm and PyDev seem to take the advantage in popularity and support. There’s also the classic IDLE, but its use is strongly discouraged nowadays, due to bugs and better options.

Lightweight text editors can make small edits easy and fast. Visual Studio Code is a recent favorite. Notepad++ is always a winner on Windows. Atom is a newer, cross-platform editor developed by GitHub that’s gaining popularity. Of course, UNIX platforms typically provide vim or emacs.

Framework Survey

If this series is for you, then install an IDE, set up a virtual environment, and let’s roll! The next posts will each introduce a popular Python test framework.Each post should be used as an introduction for getting started or as a quick reference. Please refer to official framework documentation for full details – it would be imprudent for this blog to unnecessarily duplicate information.

The outline for each post will be:

Overview
Installation
Project Structure
Example Code
Test Launch
Pros and Cons

Cucumber-JVM Global Hook Workarounds

Almost all BDD automation frameworks have some sort of hooks that run before and after scenarios. However, not all frameworks have global hooks that run once at the beginning or end of a suite of scenarios – and Cucumber-JVM is one of these unlucky few. Cucumber-JVM GitHub Issue #515, which seeks to add @BeforeAll and @AfterAll hooks, has been open and active since 2013, but it looks unclear if the issue will ever be resolved. Thankfully, there are some workarounds to effect the same behavior as global hooks.

Workaround #1: Don’t Do It

From a purist’s perspective, each scenario (or test) should be completely independent, meaning it should not share parts with any other tests. Independence provides the following benefits:

Safety between tests
Consistency across tests
The ability to run any tests individually, in any order, or in parallel
More sensible, understandable tests

If not handled properly, global hooks can be dangerous because they make tests interdependent. Changes or failures in one test may cascade into others. Global test data would waste memory for tests that don’t use it. Furthermore, the fact that Issue #515 has been open for years indicates the difficulty of properly implementing global hooks.

However, the main cost of independence is runtime. Independent tests often repeat similar setup and cleanup routines. Even a few extra seconds per test can add up tremendously. Google Guava, for example, has over 286,000 tests – adding one second to each test would amount to nearly 80 hours! Performance becomes especially critical for continuous integration, in which wasted time means either delivery delays or coverage gaps. Certain operations like preparing a database or fetching authentication tokens may be pragmatic candidates for global hooks.

The best strategy is to use global hooks only when necessary for time-intensive setup that can be shared safely. Any shared test data should be immutable. Always question the need for global hooks. Most tests probably won’t need them.

Workaround #2: Static Variables

A basic hack for global hooks is actually provided in Issue #515. A static Boolean flag can indicate when the @Before hook has run more than once because it isn’t “reset” when a new scenario re-instantiates the step definition classes. The runtime shutdown hook will be called once all tests are done and the program exits. (Note that a static flag cannot be used in an @After hook due to the halting problem.) The example from the issue is shamelessly copied below:

public class GlobalHooks {
    private static boolean dunit = false;

    @Before
    public void beforeAll() {
        if(!dunit) {
            Runtime.getRuntime().addShutdownHook(afterAllThread);
            // do the beforeAll stuff...
            dunit = true;
        }
    }
}

Workaround #3: Singleton Caching

The basic hack is useful for simple setup and cleanup routines, but it becomes inelegant when objects must be shared by scenarios. Rather than polluting the class with static members, a singleton can cache test data between scenarios, and global setup logic may be put into the singleton’s constructor. Furthermore, if the singleton uses lazy initialization, then @Before hooks may not be needed at all. A “lazy” singleton will not be instantiated until the first time its getInstance method is called, meaning it will be skipped if the scenarios do not need them. This is a huge advantage when selectively running scenarios by name, tag, or feature. (Please refer to the previous post, Static or Singleton, for a deeper explanation of the singleton pattern.)

Consider scenarios that must generate authentication tokens (like OAuth) for API testing. A singleton “token holder” could cache tokens for usernames, rather than doing the authorization dance for every scenario. The snippet below shows how such a singleton could be called within a @When step definition with no @Before method.

public class ExampleSteps {
    ...
    @When("^some API is called$")
    public void whenSomeApiIsCalled() {
        // Get the token from the singleton cache lazily
        String token = TokenHolder.getInstance().getToken("user", "pass");
        // Use the token to call some API (method not shown)
        callSomeApi(token);
    }
    ...
}

And the singleton class could be defined like this:

public class TokenHolder {
    private static volatile TokenHolder instance = null;
    private HashMap<String, String> tokens;

    private TokenHolder() {
        tokens = new HashMap<String, String>();
    }

    public static TokenHolder getInstance() {
        // Lazy and thread-safe
        if (instance == null) {
            synchronized(TokenHolder.class) {
                if (instance == null) {
                    instance = new TokenHolder();
                }
            }
        }

        return instance;
    }
    
    public String getToken(String username, String password) {
        // This check could be extended to handle token expiration
        if (!tokens.containsKey(username)) {
            // Request a fresh authentication token (method not shown)
            String token = requestToken(username, password);
            // Cache the token for later
            tokens.put(username, token);
        }
        
        return tokens.get(username);
    }
    
    ...
}

Workaround #4: JUnit Class Annotations

Another workaround mentioned in Issue #515 and elsewhere is to use JUnit‘s @BeforeClass and @AfterClass annotations in the runner class, like this:

@RunWith(Cucumber.class)
@Cucumber.Options(format = {
    "html:target/cucumber-html-report",
    "json-pretty:target/cucumber-json-report.json"})
public class RunCukesTest {

    @BeforeClass
    public static void setup() {
        System.out.println("Ran the before");
    }

    @AfterClass
    public static void teardown() {
        System.out.println("Ran the after");
    }
}

While @BeforeClass and @AfterClass may look like the cleanest solution at first, they are not very practical to use. They work only when Cucumber-JVM is set to use the JUnit runner. Other runners, like TestNG, the command line runner, and special IDE runners, won’t pick up these hooks. Their methods must also be are static and would need static variables or singletons to share data anyway. Therefore, I personally discourage using these annotations in Cucumber-JVM.

What About Dependency Injection?

Dependency injection is a marvelous technique. As defined by Wikipedia:

In software engineering, dependency injection is a technique whereby one object supplies the dependencies of another object. A dependency is an object that can be used (a service). An injection is the passing of a dependency to a dependent object (a client) that would use it. The service is made part of the client’s state. Passing the service to the client, rather than allowing a client to build or find the service, is the fundamental requirement of the pattern.

Dependency injection can be a powerful alternative to singletons because DI provides finer control over the scope of objects. However, Cucumber-JVM’s dependency injection cannot be applied with global hooks because dependency objects, like step definition objects, are constructed and destroyed for each scenario.

Comparison Table

Ultimately, the best approach for global hooks in Cucumber-JVM is the one that best fits the tests’ needs. Below is a table to make workaround comparisons easier.

Workaround	Pros	Cons
Don’t Do It	Scenarios are completely independent. No complicated or risky workarounds.	Repeated setup and cleanup procedures may add significant execution time.
Static Variables	Simple yet effective implementation.	May need many static variables to share test data.
Singleton Caching	Abstracts test data and setup procedures. Easily handles lazy initialization and evaluation. May not need a @Before hook.	More complicated design.
JUnit Class Annotations	Clean look for basic setup and cleanup routines.	May be used only with the JUnit runner. Requires static variables or singletons to share test data anyway.

Static or Singleton

Let’s face it: there are times in object-oriented programming when we need to share something somewhat globally. When I say that, many purists will scream, “But global variables are evil and should never be used!” Hence, I used the word “somewhat” – attributes (variables) and behaviors (methods) can be class members, allowing them to be used wherever their class is in scope. Things like constants and stateless utility methods are perfect candidates to be class members; Java’s Math class is a perfect example. Many engineers use the word “static” interchangeably with “class member” because languages like Java and C# use the static keyword to denote class members.

Class Membership Limitations

While going “static” is great for one-off constants and stateless methods, it is a poor solution for managing global state. Class methods require class variables to manage state. Class variables are mutable (unlike constants which are immutable) and require extra precautions for handling. And as class variables, the state is divorced from many benefits offered by the object-oriented marriage between attributes and behaviors. A much better solution for global state is the singleton pattern.

The Singleton Pattern

The singleton pattern is one of the best known, and most controversial, design patterns in object-oriented programming. A singleton class is restricted to constructing only one instance of an object so that one instance may be shared somewhat globally. Love it or hate it, the singleton pattern is quite useful when applied appropriately – factories, state machines, and even test automation often employ singletons.

For reference, below is a thread-safe singleton class written in Java, courtesy of Wikipedia. The single instance may be accessed anywhere by simply calling Singleton.getInstance(). Any attributes or behaviors are added as instance members (without “static”).

public final class Singleton {
    private static volatile Singleton instance = null;

    private Singleton() {}

    public static Singleton getInstance() {
        if (instance == null) {
            synchronized(Singleton.class) {
                if (instance == null) {
                    instance = new Singleton();
                }
            }
        }
        return instance;
    }
}

Singleton Benefits

The singleton pattern is a much better solution for managing global state for several reasons.

True Object Orientation	Singleton objects may be treated as any other object (like POJOs). Classes themselves are not plain-old objects, thus limiting their usability.
Inheritance	Singleton classes may employ inheritance to add or modify attributes and behaviors. Class members cannot be manipulated through inheritance.
Lazy Initialization	Singleton classes can easily use lazy initialization to avoid constructing the instance until it is first used. Lazy initialization for class variables is possible but often more difficult. Many times, class variables are simply initialized when declared, which would unnecessarily bloat memory if the variables are never used.
Object Pooling	The singleton pattern is essentially a special case of the object pool pattern, in which the pool size is one. Thus, a singleton class could easily be updated to handle a pool of objects instead of just a single one.
Cleaner Implementation	Singleton state is nicely encapsulated. Singleton classes provide a sensible central place for shared stuff. Other classes don’t have the “static” clutter. Object references need not be passed around.

Anecdotally, engineers who use design patterns (like singleton) have a better grasp on good object-oriented principles, and they tend to make better, more thoughtful software design decisions. Adding static members to a class tends to be a hasty decision made for expediency – it will work, but it may not be the best practice.

TL;DR

Use class members for one-off constants and stateless methods.
Use singletons for managing global, mutable state.

The Dot-Dot-Dot

Warning: This post has nothing to do specifically with software. It is rather a personal musing over communication styles.

Throughout my years in the professional work environment, I’ve noticed a trend that bothers me: the inappropriate use of the ellipsis in textual communication. For example:

Hi Andy… Automated tests from the ABC build job are failing this morning… I don’t know why… Please do the needful… Thanks…

Dot-dot-dot… What meaning did the author intend to convey?

Did their finger get stuck on the keyboard?
Did they intend to use a period or a comma?
Do they want to textually capture pauses between their phrases?
Am I supposed to assume something that they haven’t said?
Are they giving me a specific task to do or simply speculating?
Did they lose their train of thought?
Do they doubt their words?
Are they half asleep?
Are they wishy-washy?
Are they being passive-aggressive?
Are they complaining to themselves?
Are they upset?
Are they in a bad mood?
Are they mad at me?
Am I in trouble?

I know I’m not the only one standing on this soap box – a friend recently triggered me by tweeting about the same problem. Blame my minor in creative writing.

This is not merely a minor nuance. Ellipsis abuse causes ambiguity, doubt, and stress. It can cause uncertainty in office relationships. Terse textual communication is already crude, and every jot and tittle implies meaning, whether intended or accidental.

In professional environments, always strive for clear, concise, and direct communication. Good communication skills are more than just a resume tagline. We should all pay close attention to how we write. Be on point, not on three points.

The Behavior-Driven Three Amigos

Recently, my manager said to me, “Andy, your BDD documentation looks great, but could you please mention The Three Amigos?” A brief flash of panic came over me – I had never heard of “The Three Amigos” before. My immediate thought was the 1986 film of the same name or the Disney film The Three Caballeros. After a little research, I knew exactly who they were; I just didn’t know they had a name.

Who are “The Three Amigos”?

“The Three Amigos” refers to a meeting of the minds of the three primary roles involved in producing software:

Business – Often named the “business analyst” (BA) or “product owner” (PO), the business role provides what problem must be solved. They provide requirements for the solution. Typically, the business role is non-technical.
Development – The developer role provides how the solution to the problem will be implemented. They build the software and must be very technical.
Testing – The testing role, sometimes named “quality assurance” (QA), verifies that the delivered software product works correctly. They also try to find defects. The tester role must be somewhat technical.

During software development, The Three Amigos should meet regularly to discuss how the product will be developed. It is a shift left practice to avoid misunderstandings (like a game of telephone), thus improving quality and avoiding missed deadlines. The discussions should include only the individuals who will actually work on the specific deliverable, not the whole team.

While The Three Amigos seems most popular in Agile, it can be applied to any software development process. Some (here and here) advocate regularly scheduled formal meetings. Others (here and here) interpret it as an attitude instead of a process, in which the roles continuously collaborate. Regardless of implementation, The Three Amigos need to touch base before development begins.

Applying BDD

The Three Amigos fits perfectly into behavior-driven development, especially as part of BDD with Agile. Behavior scenarios are meant to foster collaboration between technical and non-technical roles because they are simple, high-level, and written in plain language. Given-When-Then provides a common format for discussion.

Ideally, when The Three Amigos meet during grooming and planning, they would formalize acceptance criteria as Gherkin features. Those feature files are then used directly by the developer for direction and the tester for automation. They act like a receipt of purchase for the business role – the feature file says, “This is what you ordered.”

A great technique for Three Amigos collaboration is Example Mapping – it efficiently identifies rules for acceptance criteria, behavior examples, and open questions. Examples can easily be turned into Gherkin scenarios either during or after the meeting.

Since BDD relies on feature files as artifacts, The Behavior-Driven Three Amigos must be more than just an attitude. The point of the collaboration is to produce feature files early for process efficiency. Less formal meetings could quickly devolve into all-talk-no-action.

Don’t Presume Anything

Don’t presume that the three roles will naturally collaborate on their own. I’ve seen teams in which the testers don’t participate in planning. I’ve also seen organizations in which automation engineers don’t help to write the test cases that they need to automate! Developers often abdicate responsibility for testing considerations, because “that’s QA’s job.” And, specifically for BDD, I’ve noticed that product owners resist writing acceptance criteria in Gherkin because they think it is too technical and beyond their role.

The Three Amigos exists as a named practice because collaboration between roles does not always happen. It is an accountability measure. Remember, the ultimate purpose for The Three Amigos is higher quality in both the product and the process. Nobody wants more meetings on their calendar, but everyone can agree that quality is necessary.

White Guy in a Kurta: My Passage to India

Life never gives us what we want at the moment that we consider appropriate. Adventures do occur, but not punctually.

From “A Passage to India” by E. M. Forster

Back in July of 2013, NetApp sent me on an awesome business trip to Bangalore, India. I had already planned to take a personal trip to China during a company shutdown, and I convinced my senior manager to send me to Bangalore on my way back to the States. The main reason for the trip was to help our offshore contractors build up their test automation skills for an upcoming release. While they benefited from my assistance, I likewise benefited from a much deeper appreciation for international workers. This article documents my experiences and the lessons I learned from my trip.

Transportation

My connecting flight from Kuala Lumpur (on Malaysia Airlines – gasp!) landed in Bangalore at around midnight, local time, on a Monday. NetApp provided a private cab for my whole stay. I felt a little bad because he spoke minimal English with a thick accent, but he was very friendly. I couldn’t see much on the drive to the hotel except a bunch of road blocks and a half-constructed highway. When he dropped me off at the Hyatt, I slipped him a few crumpled rupees my aunt had given me before my trip – she had visited India the previous year and implored me to always leave small tips.

Every morning that week, my cabbie picked me up from the hotel at 8am and returned me at 8pm. (The days were long.) He drove a crossover-like vehicle that felt like a minivan but drove like a small SUV, with flowers and Hindu statues over the dashboard. Even though the office was only a few miles from the hotel, a one-way trip took about 25 minutes. And I saw everything: “two-wheelers” (bikes with or without motors), “three-wheelers” (auto-rickshaws), “four-wheelers” (what we could call “cars” in the USA), pedestrians young and old, buses with people hanging on the side, and even cows just moseying through the streets. It was freaky to see not only everyone driving on the left side of the road, but to realize that the roads did not have any lane markings painted on them. In traffic, you just go with the flow, rather than obey specific conventions. Going to lunch one day, my “substitute” cabbie (not my typical driver) actually hit a two-wheeler, tearing off part of the cab’s bumper. Thankfully, the other guy just spat some pithy words in Hindi and sped away.

Traffic is chaos. No lanes, few control mechanisms, and they drive on the left.

The view from the backseat of my cab: driver on the right, driving on the left (I think).

The quintessential auto-rickshaw, driving rain or shine. And apparently, Bangalore has Taco Bell!

A cow at the bus stop! Just chillin' there, no big deal...

Moo.

My Team

NetApp had a huge company site in Bangalore, but I spent all but one day at Mindteck, a nearby contracting firm hired by my organization. My team in Raleigh worked extensively with our group of Mindteck contractors. I felt delighted to meet them in person – they became more than broken voices and blurry profile pictures. And the team graciously welcomed me as one of their own. Every afternoon at about three or four o’clock, they tapped me to join them on the roof for masala chai (tea). There was never any shortage of Kinley water bottles, either. Embarrassingly, the team was so large that I struggled to remember who was who, but they never forgot who I was.

Mukesh, one of my team mates from Mindteck, on the right. Me, in my kurta, on the left.

Kinley, the bottled water choice of champions! We call it “Dasani” in the USA.

During my visit, my team and I worked a lot. Every day, we had meetings, meetings, and more meetings, all about the “big A” – automation. Hearing my Raleigh team mates on the other end of conference calls felt like a strange reversal. I led automation workshops to teach our contractors a new framework we had developed in Raleigh. They also shared a number of problems with me, in the hopes that I could somehow help. Some were simple programming problems that took nothing but a simple proofread to correct. Others were out of my league. One evening, I sat locked in a conference room with an IT guy trying to figure out some obscure network failure, while two managers stared at us silently for almost two hours. I did my best to not feel awkward, assuming this style of management-by-pressure was cultural.

One problem I couldn’t solve for Mindteck was red tape. NetApp put a lot of restrictions on what the contractors could and could not do. They were restricted from accessing certain systems. NetApp’s Bangalore site also wouldn’t give them a direct network connection, which slowed down their network speeds to the point where test scripts would regularly crash from timeouts. Mindteck showed me their lab in disarray and admitted that they didn’t always get the hardware they needed for proper testing. Most frustratingly, I learned that the contractors were not even permitted to ask engineers on the parent team in Sunnyvale, California for help! In Raleigh, getting help from Sunnyvale was always a struggle, but at least we never had our hands slapped for asking. We all vented loudly about that ridiculous rule.

I spent only one day with the employees at the NetApp site. When I arrived, nobody was ready for me, so I poked around the cubicles until I saw plaques with names I recognized. Thankfully, everyone was friendly there, too. I don’t remember much of the NetApp site except meeting a few friends, seeing the Engineering Support center, and getting evening snacks at 6pm.

Lunch with NetApp employees in my organization. I wish this picture turned out better.

NB gets snacks every afternoon. What are we doing wrong, Solomon Wube and Warren Campbell?

NetApp gave out snacks every day at 6pm. These were spicy!

The Hotel

NetApp approved only two or three hotels in Bangalore for travel. Arbitrarily, I chose the Hyatt, and they treated me like a king. Every manger, director, and VIP with a business card greeted me throughout the week. Bellhops always assisted me with luggage. Room service was always quick. And it was definitely a luxury hotel. Later, I learned that I was the first NetApper to stay there, and they sought to impress me in the hopes of future business from NetApp.

Exploring on my Own

The evenings were the only chance I had to explore Bangalore. As much as I wanted to hop an auto-rickshaw, I feared getting lost or scammed, so I stayed near my hotel on foot. Around the corner was a fancy shopping mall. The first store I entered was Big Bazaar, India’s version of Walmart. As a dumb American, my first mistake was attempting to enter Big Bazaar through the exit doors – a pair of armed guards quickly corrected my mistake by pointing me to the real entrance. Inside, I meandered through the store amazed by the essentials of Indian life, most of which were not much different from my own in America. When I tried to buy a few snacks from the grocery section, I made my second mistake by presuming the cashier would give me change – nope, I lost a few rupees. Before exiting through the proper exit doors, I detoured through the clothing department and bought myself a souvenir: a black kurta with red embroidery on the collar. I’ve since worn my kurta many times at Indian festivals in Raleigh.

BIG BAZAAR, India's version of a Walmart Super Center.

Big Bazaar – it was pretty big!

Across from Big Bazaar was a movie theater. Since I had nothing else to do, I moseyed over to see what films were playing, hoping to see a Bollywood movie with English subtitles. Instead, they had Despicable Me 2 playing, which I could not resist. The ticket plus samosa and Slice mango drink cost only a few dollars – a fraction of the typical movie cost in the USA. However, when I entered the theater, an armed guard who didn’t speak English forcibly patted me down. In public places, Indians took security very seriously. My coworkers told me the security was necessary to protect against Islamic terrorists.

Bangalore had such stark dichotomies. Near my hotel on MG Road were modern buildings with posh stores and fancy restaurants and security guards, while right around the corner were dilapidated houses with refuse burning on the street. Next to billboards for mobile phone plans were temples with innumerable statues. Two constants, though, were the crowds of people everywhere and, sadly, the pollution they left behind.

A mango peddler near Mindteck.

I passed many Hindu temples while riding around in my cab.

The Food

It’s a good thing that I loved Indian food. At work, my Indian counterparts showed much hospitality through their food. Every day, they provided lunch for me. One day, Mindteck catered a full lunch buffet as part of a meeting marathon. On the day I was on-site at NetApp, a friend bought me lunch in the cafe – 100 rupees, or about $1.50. For the remaining two days, both NetApp and Mindteck planned off-site lunches at delicious Indian restaurants. I ate it all: rice, naan, roti, curry of every color, chicken, lamb, paneer, aloo, palak, and gulab jamum for dessert. Nobody was surprised that I was “non-veg”, but they did appreciate my culinary adventurousness. For me, I was surprised to discover that, in India, eating with your bare hands is normal.

Gulab jamun, hot and fresh at an Indian buffet.

In the evenings, I had two fancy meals on my own. The best was my last night at the hotel. Since NetApp covered all travel expenses, I chose to eat dinner at the Hyatt’s fancy restaurant. It was totally empty, but that meant I received the best service. After serving my chaat and my curry, the head chef himself came out to greet me. The other fancy dinner was at a South Indian restaurant on the top floor of the nearby shopping mall – prawns, yellow-green curry, and round flatbread washed down with a Kingfisher. When the waiter served the food, he put the napkin on my lap for me. A third dinner I ate at the hotel’s standard restaurant while they played live music. On the night I went to the movies, I ate a Chicken Maharaja Mac at McDonald’s – let’s just say the Big Mac is better with beef. (Come on, I’m a red-blooded American.)

The South Indian dinner at the fancy shopping mall.

These were the best shrimp I have ever eaten. Indians always call them "prawns."

Some of the best “prawns” (not shrimp) I’ve ever eaten.

The Maharaja Mac is made using chicken, since it is illegal to kill cows. The combo cost about $3.25.

The Chicken Maharaja Mac Meal. #MURICA

Goodbyes

On my last day in Bangalore, I wore my kurta. As I left in the morning, the front desk lady at the Hyatt shouted out, “Very nice kurta, Mr. Knight!” My Mindteck team planned a proper sendoff in the afternoon. We took group photos, and they gifted to me a small desk clock that I keep to this day. And they offered one final treat – an ice cream bar. There were six small scoops of various colors, and I had to taste each to guess the flavor. To everyone’s surprise, I got them all right! I didn’t have the heart to tell them that I suffered from lactose intolerance. My belly ached during the whole flight home, but the experience was well worth it.

My flight out of Bangalore departed in the wee hours of the morning. My cab picked me up from the airport shortly after midnight. I flew to Qatar, then NYC, and finally to Raleigh for a nearly day-long transit. My biggest regret was not having enough time for sightseeing in India.

The Mindteck team and me: coworkers and friends.

Lactose intolerance in India sucks. I ate the ice cream anyway. These flavors are blueberry, mango, baked apple, jackfruit, lychee, and watermelon!

The ice cream bar, from left to right by row: strawberry, banana, lychee (?), jackfruit (?), mango, and blueberry.

Lessons Learned

I gained a genuine appreciation for our offshore teams after my trip. Oftentimes, I hear American workers complain that offshore workers are “stealing jobs,” and “you get what you pay for” in quality of work. But I learned firsthand that these folks are good people just trying to make a living. They aren’t trying to steal our jobs – they’re taking the opportunities in front of them as part of a global economy. Frankly, big companies hire offshore resources so they can throw more bodies at their projects. And quality suffers greatly when any team is not co-located due to challenges with communication, time zone, technology, bureaucracy, etc. Moreover, our guys and girls wanted to be more than just hired help: they sought to learn more, improve their skills, and increase their contribution.

Overall, I was very grateful for the opportunity to visit Bangalore, and I hope to return some day!

This post is dedicated to all of my friends from India.

12 Awesome Benefits of BDD

What can BDD do for you? Why adopt a new process with a new framework? Because it’s worth it! The main benefits of BDD are better collaboration and automation. This article expands those two into a dozen awesome benefits. (If you read the BDD 101 series, then these points should look familiar.)

#1: Inclusion

BDD is meant to be collaborative. Everyone from the customer to the tester should be able to easily engage in product development. And anyone can write behavior scenarios because they are written in plain language. Scenarios are:

Requirements for product owners
Acceptance criteria for developers
Test cases for testers
Scripts for automators
Description for other stakeholders

Essentially, BDD is an enhancement of The Three Amigos.

#2: Clarity

Scenarios focus on the expected behaviors of the product. Each scenario focuses on one specific thing. Behaviors are described in plain language, and any ambiguity can be clarified with a simple conversation or Example Mapping. There’s no unreadable code or obscure technical jargon, and there’s no game of telephone. Clarity ensures the customer gets what the customer wants.

#3: Streamlining

BDD is designed to speed up the development process. Everyone involved in development relies upon the same scenarios. Scenarios are requirements, acceptance criteria, test cases, and test scripts all in one – there is no need to write any other artifact. The modular nature of Gherkin syntax expedites test automation development. Furthermore, scenarios can be used as steps to reproduce failures for defect reports.

#4: Shift Left

“Shift left” is a buzzword for testing early in the development process. Testing earlier means fewer bugs later. In BDD, test case definition inherently becomes part of the requirements phase (for waterfall) or grooming (for Agile). As soon as behavior scenarios are written, testing and automation can theoretically begin.

#5: Artifacts

Scenarios form a collection of self-documenting test cases as a result of the BDD process. This ever-growing collection forms a perfect regression test suite. Scenarios can be run manually or with automation. Any tests not automated can be added to a backlog to automate in the future.

#6: Automation

BDD frameworks make it easy to turn scenarios into automated tests. The steps are already given by the scenarios – the automation engineer simply needs to write a method/function to perform each step’s operations.

#7: Test-Driven

BDD is an evolution of TDD. Writing scenarios from the beginning enforces quality-first and test-first mindsets. BDD automation can run scenarios to fail until the feature is implemented and causes tests to pass.

#8: Code Reuse

Given-When-Then steps can be reused between scenarios. The underlying implementation for each step does not change. Automation code becomes very modular.

#9: Parameterization

Scenario steps can be parameterized to be even more reusable. For example, a step to click a button can take in its ID. Parameterization can help a team adopt a common, reusable set of steps, and it inspires healthier discussion when writing scenarios.

#10: Variation

Scenario outlines make it easy to run the same scenario with different combinations of inputs. This is a simple but powerful way to expand test coverage without code duplication, which is the bane of test automation.

#11: Momentum

BDD has a snowball effect: scenarios become easier and faster to write and automate as more step definitions are added. Scenarios typically share common steps. Sometimes, new scenarios need nothing more than different step parameters or just one new line.

#12: Adaptability

BDD scenarios are easy to update as the product changes. Plain language is easy to edit. Modular design makes changes to automation code safer. Scenarios can also be filtered by tag name to decide what runs and what doesn’t.

Which Version of Python Should I Use?

Which version of Python should I use? Now, that’s a loaded question. While the answer is simple, the explanation is more complicated.

tl;dr

For most people:

Use the latest version of Python 3.
Use the CPython implementation.
Use pipenv to manage packages and installations.
Use Visual Studio Code or PyCharm for editing code.

Which Version?

Python 2 and Python 3 are actually slightly different languages. The differences go deeper than just print statements. The What’s New in Python page on the official doc site lists all the gory details, and decent articles showcasing differences can be found here, here, and here. Although Python 3 is newer, Python 2 remains prevalent. Most popular packages use Python packaging tools to support both versions.

The Python Wiki makes it clear that Python 3 is the better choice:

Python 2.x is legacy, Python 3.x is the present and future of the language

Furthermore, Python 2 will reach end-of-life in 2020. The Python team will continue to provide bug fixes for 2.7 until 2020 (PEP 373), but there will be no new language features and no 2.8 (PEP 404). (Originally, end-of-life was planned for 2015, but it was pushed back by 5 years.) There is even a Python 2.7 Countdown clock online.

Which Implementation?

In purest terms, “Python” is a language specification. An implementation provides the language processing tools (compiler, interpreter, etc.) to run Python programs. The Hitchhiker’s Guide to Python has a great article entitled Picking an Interpreter that provides a good summary of available interpreters. Others are listed on python.org and the Python Wiki. The table below provides a quick overview of the big ones.

Implementation	Points
CPython	most widely used implementation the reference implementation has the most libraries and support implemented in C supports Python 2 and 3
PyPy	much faster than CPython much more memory efficient implemented in RPython supports Python 2 and 3
Jython	implemented in Java runs on the JVM supports Python 2 only a sandbox for Python 3 no project updates since May 2015
IronPython	implemented for .NET lets Python libs call .NET and vice versa supports Python 2
Python for .NET	integrates CPython with .NET/Mono runtime supports Python 2 and 3
Stackless Python	branch of CPython with real threading
MicroPython	optimized for microcontrollers uses a subset of the standard library

Unless you have a very specific reason, just use CPython. In fact, most people are referring to CPython when they say “Python.” CPython has the most compatibility, the widest package library, and the richest support. If you really need speed, consider PyPy.

Managing Installations

pip is the standard tool for installing Python packages. The simplest way to install Python is to install it “globally” for the system. In fact, some operating systems like macOS and Ubuntu have Python pre-installed. However, global installation has limitations:

You may want to develop packages for both versions 2 and 3.
You may not have permissions to add new packages globally.
Different projects may require different versions of packages.

These problems can be solved by using “virtual” environments. A virtual environment is like a local Python installation with a specific package set. For example, I have created virtual environments for Python as part of Jenkins build jobs, since I did not have permission to install special automation packages globally on the Jenkins slaves.

The standard virtual environment tool for Python is venv, which has been packaged with (C)Python since 3.3. (venv had a command line wrapper named pyvenv, but this was deprecated in 3.6.) Another older but still popular third-party tool is virtualenv. As explained in this Reddit post, venv is the Python-sanctioned replacement for virtualenv. However, virtualenv supports Python 2, whereas venv does not. Conda is an environment manager popular with the science and data communities, and it can support other languages in addition to Python.

That being said, there is a relatively new package manager taking the Python world by storm: pipenv. Pipenv combines pip, Pipfile, and virtualenv into an easy workflow with simple commands. Personally, I find it to be very helpful. However, it has caused some controversy (see Reddit), and it may not be applicable for all scenarios (see Chris Warrick’s article). My recommendation is to use pipenv for new projects if it meets your needs.

Editors and IDEs

After setting up your Python environment, you are ready to start programming! There are two routes to take for text editing: source code editors and integrated development environments.

Source code editors are lightweight but often include basics like syntax highlighting and basic auto-completion. They’re great for quick edits and light scripting. Many have plugins. Popular choices are Visual Studio Code, Sublime, Atom, and Notepad++. My current favorite is Visual Studio Code because the Python extensions are stellar and settings are simple – just remember to install the extensions you need! I use it personally for Django development.

For more intense development, I highly recommend an IDE like JetBrains PyCharm, PyDev for Eclipse, Wing Python IDE, or Eric. IDEs provide rich development support, especially for larger apps that use frameworks like Django, Pyramid, and SQLAlchemy. They also make testing easier with plugins for test frameworks like pytest, behave, and others. PyCharm and PyDev are particularly nice because they can integrate into their larger IDEs (IntelliJ IDEA and Eclipse, respectively) to handle more languages. Personally, I prefer PyCharm, but advanced features require a paid license.

Pythonese

The Python community throws around a few terms you should know:

Word or Phrase	Meaning
Pythonic	describes idiomatic code for Python closely related to conciseness, readability, and elegance highly recommended to use follow style guidelines
Pythonista	someone who loves the Python language often an advanced Python programmer
Pythoneer	a programmer who uses Python to solve problems
The Zen of Python	the list of guiding principles for Python’s design run “import this” to see them
The Python Software Foundation (PSF)	non-profit org keeps Python going strong support them!
PyCon	the annual Python conference held in North America GO – it will change your life! several other conferences are held worldwide
Benevolent Dictator for Life (BDFL)	Guido van Rossum the inventor of Python resigned in July 2018 but remains BDFL Emeritus

[8/6/2018: I updated the recommendations for editors and IDEs.]

[8/15/2018: I added the Pythonese section.]

[8/17/2018: I updated pipenv information.]

This article is dedicated to my good friend Sudarsan,
who originally asked me the question in the title.

BDD 101: Frameworks

Every major programming language has a BDD automation framework. Some even have multiple choices. Building upon the structural basics from the previous post, this post provides a survey of the major frameworks available today. Since I cannot possibly cover every BDD framework in depth in this 101 series, my goal is to empower you, the reader, to pick the best framework for your needs. Each framework has support documentation online justifying its unique goodness and detailing how to use it, and I would prefer not to duplicate documentation. Use this post primarily as a reference. (Check the Automation Panda BDD page for the full table of contents.)

Major Frameworks

Most BDD frameworks are Cucumber versions, JBehave derivatives inspired by Dan North, or non-Gherkin spec runners. Some put behavior scenarios into separate files, while others put them directly into the source code.

C# and Microsoft .NET

SpecFlow, created by Gáspár Nagy, is arguably the most popular BDD framework for Microsoft .NET languages. Its tagline is “Cucumber for .NET” – thus fully compliant with Gherkin. SpecFlow also has polished, well-designed hooks, context injection, and parallel execution (especially with test thread affinity). The basic package is free and open source, but SpecFlow also sells licenses for SpecFlow+ extensions. The free version requires a unit test runner like MsTest, NUnit, or xUnit.net in order to run scenarios. This makes SpecFlow flexible but also feels jury-rigged and inelegant. The licensed version provides a slick runner named SpecFlow+ Runner (which is BDD-friendly) and a Microsoft Excel integration tool named SpecFlow+ Excel. Microsoft Visual Studio has extensions for SpecFlow to make development easier.

There are plenty of other BDD frameworks for C# and .NET, too. xBehave.net is an alternative that pairs nicely with xUnit.net. A major difference of xBehave.net is that scenario steps are written directly in the code, instead of in separate text (feature) files. LightBDD bills itself as being more lightweight than other frameworks and basically does some tricks with partial classes to make the code more readable. NSpec is similar to RSpec and Mocha and uses lambda expressions heavily. Concordion offers some interesting ways to write specs, too. NBehave is a JBehave descendant, but the project appears to be dead without any updates since 2014.

Java and JVM Languages

The main Java rivalry is between Cucumber-JVM and JBehave. Cucumber-JVM is the official Cucumber version for Java and other JVM languages (Groovy, Scala, Clojure, etc.). It is fully compliant with Gherkin and generates beautiful reports. The Cucumber-JVM driver can be customized, as well. JBehave is one of the first and foremost BDD frameworks available. It was originally developed by Dan North, the “father of BDD.” However, JBehave is missing key Gherkin features like backgrounds, doc strings, and tags. It was also a pure-Java implementation before Cucumber-JVM existed. Both frameworks are widely used, have plugins for major IDEs, and distribute Maven packages. This popular but older article compares the two in slight favor of JBehave, but I think Cucumber-JVM is better, given its features and support.

The Automation panda article Cucumber-JVM for Java is a thorough guide for the Cucumber-JVM framework.

Java also has a number of other BDD frameworks. JGiven uses a fluent API to spell out scenarios, and pretty HTML reports print the scenarios with the results. It is fairly clean and concise. Spock and JDave are spec frameworks, but JDave has been inactive for years. Scalatest for Scala also has spec-oriented features. Concordion also provides a Java implementation.

JavaScript

Almost all JavaScript BDD frameworks run on Node.js. Jasmine and Mocha are two of the most popular general-purpose JS test frameworks. They differ in that Jasmine has many features included (like assertions and spies) that Mocha does not. This makes Jasmine easier to get started (good for beginners) but makes Mocha more customizable (good for power users). Both claim to be behavior-driven because they structure tests using “describe” and “it-should” phrases in the code, but they do not have the advantage of separate, reusable steps like Gherkin. Personally, I consider Jasmine and Mocha to be behavior-inspired but not fully behavior-driven.

Other BDD frameworks are more true to form. Cucumber provides Cucumber.js for Gherkin-compliant happiness. Yadda is Gherkin-like but with a more flexible syntax. Vows provides a different way to approach behavior using more formalized phrase partitions for a unique form of reusability. The Cucumber blog argues that Cucumber.js is best due to its focus on good communication through plain language steps, whereas other JavaScript BDD frameworks are more code-y. (Keep in mind, though, that Cucumber would naturally boast of its own framework.) Other comparisons are posted here, here, here, and here.

PHP

The two major BDD frameworks for PHP are Behat and Codeception. Behat is the official Cucumber version for PHP, and as such is seen as the more “pure” BDD framework. Codeception is more programmer-focused and can handle other styles of testing. There are plenty of articles comparing the two – here, here, and here (although the last one seems out of date). Both seem like good choices, but Codeception seems more flexible.

Python

Python has a plethora of test frameworks, and many are BDD. behave and lettuce are probably the two most popular players. Feature comparison is analogous to Cucumber-JVM versus JBehave, respectively: behave is practically Gherkin compliant, while lettuce lacks a few language elements. Both have plugins for major IDEs. pytest-bdd is on the rise because it integrates with all the wonderful features of pytest. radish is another framework that extends the Gherkin language to include scenario loops, scenario preconditions, and variables. All these frameworks put scenarios into separate feature files. They all also implement step definitions as functions instead of classes, which not only makes steps feel simpler and more independent, but also avoids unnecessary object construction.

Other Python frameworks exist as well. pyspecs is a spec-oriented framework. Freshen was a BDD plugin for Nose, but both Freshen and Nose are discontinued projects.

Ruby

Cucumber, the gold standard for BDD frameworks, was first implemented in Ruby. Cucumber maintains the official Gherkin language standard, and all Cucumber versions are inspired by the original Ruby version. Spinach bills itself as an enhancement to Cucumber by encapsulating steps better. RSpec is a spec-oriented framework that does not use Gherkin.

Which One is Best?

There is no right answer – the best BDD framework is the one that best fits your needs. However, there are a few points to consider when weighing your options:

What programming language should I use for test automation?
Is it a popular framework that many others use?
Is the framework actively supported?
Is the spec language compliant with Gherkin?
What type of testing will you do with the framework?
What are the limitations as compared to other frameworks?

Frameworks that separate scenario text from implementation code are best for shift-left testing. Frameworks that put scenario text directly into the source code are better for white box testing, but they may look confusing to less experienced programmers.

Personally, my favorites are SpecFlow and pytest-bdd. At LexisNexis, I used SpecFlow and Cucumber-JVM. For Python, I used behave at MaxPoint, but I have since fallen in love with pytest-bdd since it piggybacks on the wonderfulness of pytest. (I can’t wait for this open ticket to add pytest-bdd support in PyCharm.) For skill transferability, I recommend Gherkin compliance, as well.

Reference Table

The table below categorizes BDD frameworks by language and type for quick reference. It also includes frameworks in languages not described above. Recommended frameworks are denoted with an asterisk (*). Inactive projects are denoted with an X (x).

Language	Framework	Type
C	Catch	In-line Spec
C++	Igloo	In-line Spec
C# and .NET	Concordion LightBDD NBehave x NSpec SpecFlow * xBehave.net	In-line Spec In-line Gherkin Separated semi-Gherkin In-line Spec Separated Gherkin In-line Gherkin
Golang	Ginkgo	In-line Spec
Java and JVM	Cucumber-JVM * JBehave JDave x JGiven * Scalatest Spock	Separated Gherkin Separated semi-Gherkin In-line Spec In-line Gherkin In-line Spec In-line Spec
JavaScript	Cucumber.js * Yadda Jasmine Mocha Vows	Separated Gherkin Separated semi-Gherkin In-line Spec In-line Spec In-line Spec
Perl	Test::BDD::Cucumber	Separated Gherkin
PHP	Behat Codeception *	Separated Gherkin Separated or In-line
Python	behave * freshen x lettuce pyspecs pytest-bdd * radish	Separated Gherkin Separated Gherkin Separated semi-Gherkin In-line Spec Separated semi-Gherkin Separated Gherkin-plus
Ruby	Cucumber * RSpec Spinach	Separated Gherkin In-line Spec Separated Gherkin
Swift / Objective C	Quick	In-line Spec

[4/22/2018] Update: I updated info for C# and Python frameworks.

BDD 101: Automation

Better automation is one of BDD’s hallmark benefits. In fact, the main goal of BDD could be summarized as rapidly turning conceptualized behavior into automatically tested behavior. While the process and the Gherkin are universal, the underlying automation could be built using one of many frameworks.

This post explains how BDD automation frameworks work. It focuses on the general structure of the typical framework – it is not a tutorial on how to use any specific framework. However, I wrote short examples for each piece using Python’s behave framework, since learning is easier with examples. I chose to use Python here simply for its conciseness. (Check the Automation Panda BDD page for the full table of contents.)

Framework Parts

Every BDD automation framework has five major pieces:

#1: Feature Files

Gherkin feature files are very much part of the automation. They act like test scripts – each scenario is essentially a test case. Previous posts covered Gherkin in depth.

Here is an example feature file named google_search.feature:

Feature: Google Searching
  As a web surfer, I want to search Google, so that I can learn new things.
  
  # This scenario should look familiar
  @automated @google-search @panda
  Scenario: Simple Google search
    Given a web browser is on the Google page
    When the search phrase "panda" is entered
    Then results for "panda" are shown

#2: Step Definitions

A step definition is a code block that implements the logic to execute a step. It is typically a method or function with the English-y step phrase as an annotation. Step definitions can take in arguments, doc strings, and step tables. They may also make assertions to pass or fail a scenario. In most frameworks, data can be passed between steps using some sort of context object. When a scenario is executed, the driver matches each scenario step phrase to its step definition. (Most frameworks use regular expressions for phrase matching.) Thus, every step in a feature file needs a step definition.

The step definitions would be written in a Python source file like this:

from behave import *

@given('a web browser is on the Google page')
def step_impl(context):
  context.google_page.load();

@when('the search phrase "{phrase}" is entered')
def step_impl(context, phrase):
  context.google_page.search(phrase)

@then('the results for "{phrase}" are shown')
def step_impl(context, phrase):
  assert context.google_page.has_results(phrase)

#3: Hooks

Certain automation logic cannot be handled by step definitions. For example, scenarios may need special setup and cleanup operations. Most BDD frameworks provide hooks that can insert calls before or after Gherkin sections, typically filterable using tags. Hooks are similar in concept to aspect-oriented programming.

In behave, hooks are written in a Python source file named environment.py:

import page_objects
from selenium import webdriver

def before_all(context):
  context.browser = webdriver.Chrome()

def before_scenario(context):
  context.google_page = page_objects.GooglePage(context.browser)

def after_all(context):
  context.browser.quit()

#4: Support Code

Support code (a.k.a libraries or packages) refers to any code called by step definitions and hooks. Support code could be dependency packages downloaded using managers like Maven (Java), NuGet (.NET), or PyPI (Python). For example, Selenium WebDriver is a well-known package for web browser automation. Support code could also be components to assist automation, such as page objects or other design patterns. As the cliché goes, “Don’t reinvent the wheel.” Step definitions and hooks should not contain all of the logic for running the actions – they should reuse common code as much as possible.

A Python page object class from the page_objects.py module could look like this:

class GooglePage(object):
  """A page object for the Google home page"""
  
  def __init__(self, browser):
    self.browser = browser
  
  def load():
    # put code here
    pass
  
  def search(phrase):
    # put code here
    pass
  
  def has_results(phrase):
    # put code here
    return False

#5: Driver

Every automation framework has a driver that runs tests, and BDD frameworks are no different. The driver executes each scenario in a feature file independently. Whenever a failure happens, the driver reports the failure and aborts the scenario. Drivers typically have discovery mechanisms for selecting scenarios to run based on tag names or file paths.

The behave driver can be launched from the command line like this:

> behave google_search.py --tags @panda

Automation Advantages

Even if a team does not apply behavior-driven practices to its full development process, BDD test frameworks still have some significant advantages over non-BDD test frameworks. First of all, steps make BDD automation very modular and thus reusable. Each step is an independent action, much like how each scenario is an independent behavior. Once a step definition is written, it may be reused by any number of scenarios. This is crucial, since most behaviors for a feature share common actions. And all steps are inherently self-documenting, since they are written in plain language. There is a natural connection between high-level behavior and low-level implementation.

Test execution also has advantages. Tags make it very easy to select tests to run, especially from the command line. Failures are very informative as well. The driver pinpoints precisely which step failed for which scenario. And since behaviors are isolated, a failure for one scenario is less likely to affect other test scenarios than would be the case for procedure-driven tests.

All of this is explained more thoroughly in the Automation Panda article, ‑‑BDD; Automation without Collaboration.

What About Test Data?

Test data is a huge concern for any automation framework. Simple test data values may be supplied directly in Gherkin as step arguments or table values, but larger test data sets require other strategies. Support code can be used to handle test data. Read BDD 101: Test Data for more information.

Available Frameworks

There are many BDD frameworks out there. The next post will introduce a few major frameworks for popular languages.