Author: Andy Knight

I'm a software engineer who specializes in test automation.

Static or Singleton

Let’s face it: there are times in object-oriented programming when we need to share something somewhat globally. When I say that, many purists will scream, “But global variables are evil and should never be used!” Hence, I used the word “somewhat” – attributes (variables) and behaviors (methods) can be class members, allowing them to be used wherever their class is in scope. Things like constants and stateless utility methods are perfect candidates to be class members; Java’s Math class is a perfect example. Many engineers use the word “static” interchangeably with “class member” because languages like Java and C# use the static keyword to denote class members.

Class Membership Limitations

While going “static” is great for one-off constants and stateless methods, it is a poor solution for managing global state. Class methods require class variables to manage state. Class variables are mutable (unlike constants which are immutable) and require extra precautions for handling. And as class variables, the state is divorced from many benefits offered by the object-oriented marriage between attributes and behaviors. A much better solution for global state is the singleton pattern.

The Singleton Pattern

The singleton pattern is one of the best known, and most controversial, design patterns in object-oriented programming. A singleton class is restricted to constructing only one instance of an object so that one instance may be shared somewhat globally. Love it or hate it, the singleton pattern is quite useful when applied appropriately – factories, state machines, and even test automation often employ singletons.

For reference, below is a thread-safe singleton class written in Java, courtesy of Wikipedia. The single instance may be accessed anywhere by simply calling Singleton.getInstance(). Any attributes or behaviors are added as instance members (without “static”).

public final class Singleton {
    private static volatile Singleton instance = null;

    private Singleton() {}

    public static Singleton getInstance() {
        if (instance == null) {
            synchronized(Singleton.class) {
                if (instance == null) {
                    instance = new Singleton();
                }
            }
        }
        return instance;
    }
}

Singleton Benefits

The singleton pattern is a much better solution for managing global state for several reasons.

True Object Orientation	Singleton objects may be treated as any other object (like POJOs). Classes themselves are not plain-old objects, thus limiting their usability.
Inheritance	Singleton classes may employ inheritance to add or modify attributes and behaviors. Class members cannot be manipulated through inheritance.
Lazy Initialization	Singleton classes can easily use lazy initialization to avoid constructing the instance until it is first used. Lazy initialization for class variables is possible but often more difficult. Many times, class variables are simply initialized when declared, which would unnecessarily bloat memory if the variables are never used.
Object Pooling	The singleton pattern is essentially a special case of the object pool pattern, in which the pool size is one. Thus, a singleton class could easily be updated to handle a pool of objects instead of just a single one.
Cleaner Implementation	Singleton state is nicely encapsulated. Singleton classes provide a sensible central place for shared stuff. Other classes don’t have the “static” clutter. Object references need not be passed around.

Anecdotally, engineers who use design patterns (like singleton) have a better grasp on good object-oriented principles, and they tend to make better, more thoughtful software design decisions. Adding static members to a class tends to be a hasty decision made for expediency – it will work, but it may not be the best practice.

TL;DR

Use class members for one-off constants and stateless methods.
Use singletons for managing global, mutable state.

The Dot-Dot-Dot

Warning: This post has nothing to do specifically with software. It is rather a personal musing over communication styles.

Throughout my years in the professional work environment, I’ve noticed a trend that bothers me: the inappropriate use of the ellipsis in textual communication. For example:

Hi Andy… Automated tests from the ABC build job are failing this morning… I don’t know why… Please do the needful… Thanks…

Dot-dot-dot… What meaning did the author intend to convey?

Did their finger get stuck on the keyboard?
Did they intend to use a period or a comma?
Do they want to textually capture pauses between their phrases?
Am I supposed to assume something that they haven’t said?
Are they giving me a specific task to do or simply speculating?
Did they lose their train of thought?
Do they doubt their words?
Are they half asleep?
Are they wishy-washy?
Are they being passive-aggressive?
Are they complaining to themselves?
Are they upset?
Are they in a bad mood?
Are they mad at me?
Am I in trouble?

I know I’m not the only one standing on this soap box – a friend recently triggered me by tweeting about the same problem. Blame my minor in creative writing.

This is not merely a minor nuance. Ellipsis abuse causes ambiguity, doubt, and stress. It can cause uncertainty in office relationships. Terse textual communication is already crude, and every jot and tittle implies meaning, whether intended or accidental.

In professional environments, always strive for clear, concise, and direct communication. Good communication skills are more than just a resume tagline. We should all pay close attention to how we write. Be on point, not on three points.

The Behavior-Driven Three Amigos

Recently, my manager said to me, “Andy, your BDD documentation looks great, but could you please mention The Three Amigos?” A brief flash of panic came over me – I had never heard of “The Three Amigos” before. My immediate thought was the 1986 film of the same name or the Disney film The Three Caballeros. After a little research, I knew exactly who they were; I just didn’t know they had a name.

Who are “The Three Amigos”?

“The Three Amigos” refers to a meeting of the minds of the three primary roles involved in producing software:

Business – Often named the “business analyst” (BA) or “product owner” (PO), the business role provides what problem must be solved. They provide requirements for the solution. Typically, the business role is non-technical.
Development – The developer role provides how the solution to the problem will be implemented. They build the software and must be very technical.
Testing – The testing role, sometimes named “quality assurance” (QA), verifies that the delivered software product works correctly. They also try to find defects. The tester role must be somewhat technical.

During software development, The Three Amigos should meet regularly to discuss how the product will be developed. It is a shift left practice to avoid misunderstandings (like a game of telephone), thus improving quality and avoiding missed deadlines. The discussions should include only the individuals who will actually work on the specific deliverable, not the whole team.

While The Three Amigos seems most popular in Agile, it can be applied to any software development process. Some (here and here) advocate regularly scheduled formal meetings. Others (here and here) interpret it as an attitude instead of a process, in which the roles continuously collaborate. Regardless of implementation, The Three Amigos need to touch base before development begins.

Applying BDD

The Three Amigos fits perfectly into behavior-driven development, especially as part of BDD with Agile. Behavior scenarios are meant to foster collaboration between technical and non-technical roles because they are simple, high-level, and written in plain language. Given-When-Then provides a common format for discussion.

Ideally, when The Three Amigos meet during grooming and planning, they would formalize acceptance criteria as Gherkin features. Those feature files are then used directly by the developer for direction and the tester for automation. They act like a receipt of purchase for the business role – the feature file says, “This is what you ordered.”

A great technique for Three Amigos collaboration is Example Mapping – it efficiently identifies rules for acceptance criteria, behavior examples, and open questions. Examples can easily be turned into Gherkin scenarios either during or after the meeting.

Since BDD relies on feature files as artifacts, The Behavior-Driven Three Amigos must be more than just an attitude. The point of the collaboration is to produce feature files early for process efficiency. Less formal meetings could quickly devolve into all-talk-no-action.

Don’t Presume Anything

Don’t presume that the three roles will naturally collaborate on their own. I’ve seen teams in which the testers don’t participate in planning. I’ve also seen organizations in which automation engineers don’t help to write the test cases that they need to automate! Developers often abdicate responsibility for testing considerations, because “that’s QA’s job.” And, specifically for BDD, I’ve noticed that product owners resist writing acceptance criteria in Gherkin because they think it is too technical and beyond their role.

The Three Amigos exists as a named practice because collaboration between roles does not always happen. It is an accountability measure. Remember, the ultimate purpose for The Three Amigos is higher quality in both the product and the process. Nobody wants more meetings on their calendar, but everyone can agree that quality is necessary.

White Guy in a Kurta: My Passage to India

Life never gives us what we want at the moment that we consider appropriate. Adventures do occur, but not punctually.

From “A Passage to India” by E. M. Forster

Back in July of 2013, NetApp sent me on an awesome business trip to Bangalore, India. I had already planned to take a personal trip to China during a company shutdown, and I convinced my senior manager to send me to Bangalore on my way back to the States. The main reason for the trip was to help our offshore contractors build up their test automation skills for an upcoming release. While they benefited from my assistance, I likewise benefited from a much deeper appreciation for international workers. This article documents my experiences and the lessons I learned from my trip.

Transportation

My connecting flight from Kuala Lumpur (on Malaysia Airlines – gasp!) landed in Bangalore at around midnight, local time, on a Monday. NetApp provided a private cab for my whole stay. I felt a little bad because he spoke minimal English with a thick accent, but he was very friendly. I couldn’t see much on the drive to the hotel except a bunch of road blocks and a half-constructed highway. When he dropped me off at the Hyatt, I slipped him a few crumpled rupees my aunt had given me before my trip – she had visited India the previous year and implored me to always leave small tips.

Every morning that week, my cabbie picked me up from the hotel at 8am and returned me at 8pm. (The days were long.) He drove a crossover-like vehicle that felt like a minivan but drove like a small SUV, with flowers and Hindu statues over the dashboard. Even though the office was only a few miles from the hotel, a one-way trip took about 25 minutes. And I saw everything: “two-wheelers” (bikes with or without motors), “three-wheelers” (auto-rickshaws), “four-wheelers” (what we could call “cars” in the USA), pedestrians young and old, buses with people hanging on the side, and even cows just moseying through the streets. It was freaky to see not only everyone driving on the left side of the road, but to realize that the roads did not have any lane markings painted on them. In traffic, you just go with the flow, rather than obey specific conventions. Going to lunch one day, my “substitute” cabbie (not my typical driver) actually hit a two-wheeler, tearing off part of the cab’s bumper. Thankfully, the other guy just spat some pithy words in Hindi and sped away.

Traffic is chaos. No lanes, few control mechanisms, and they drive on the left.

The view from the backseat of my cab: driver on the right, driving on the left (I think).

The quintessential auto-rickshaw, driving rain or shine. And apparently, Bangalore has Taco Bell!

A cow at the bus stop! Just chillin' there, no big deal...

Moo.

My Team

NetApp had a huge company site in Bangalore, but I spent all but one day at Mindteck, a nearby contracting firm hired by my organization. My team in Raleigh worked extensively with our group of Mindteck contractors. I felt delighted to meet them in person – they became more than broken voices and blurry profile pictures. And the team graciously welcomed me as one of their own. Every afternoon at about three or four o’clock, they tapped me to join them on the roof for masala chai (tea). There was never any shortage of Kinley water bottles, either. Embarrassingly, the team was so large that I struggled to remember who was who, but they never forgot who I was.

Mukesh, one of my team mates from Mindteck, on the right. Me, in my kurta, on the left.

Kinley, the bottled water choice of champions! We call it “Dasani” in the USA.

During my visit, my team and I worked a lot. Every day, we had meetings, meetings, and more meetings, all about the “big A” – automation. Hearing my Raleigh team mates on the other end of conference calls felt like a strange reversal. I led automation workshops to teach our contractors a new framework we had developed in Raleigh. They also shared a number of problems with me, in the hopes that I could somehow help. Some were simple programming problems that took nothing but a simple proofread to correct. Others were out of my league. One evening, I sat locked in a conference room with an IT guy trying to figure out some obscure network failure, while two managers stared at us silently for almost two hours. I did my best to not feel awkward, assuming this style of management-by-pressure was cultural.

One problem I couldn’t solve for Mindteck was red tape. NetApp put a lot of restrictions on what the contractors could and could not do. They were restricted from accessing certain systems. NetApp’s Bangalore site also wouldn’t give them a direct network connection, which slowed down their network speeds to the point where test scripts would regularly crash from timeouts. Mindteck showed me their lab in disarray and admitted that they didn’t always get the hardware they needed for proper testing. Most frustratingly, I learned that the contractors were not even permitted to ask engineers on the parent team in Sunnyvale, California for help! In Raleigh, getting help from Sunnyvale was always a struggle, but at least we never had our hands slapped for asking. We all vented loudly about that ridiculous rule.

I spent only one day with the employees at the NetApp site. When I arrived, nobody was ready for me, so I poked around the cubicles until I saw plaques with names I recognized. Thankfully, everyone was friendly there, too. I don’t remember much of the NetApp site except meeting a few friends, seeing the Engineering Support center, and getting evening snacks at 6pm.

Lunch with NetApp employees in my organization. I wish this picture turned out better.

NB gets snacks every afternoon. What are we doing wrong, Solomon Wube and Warren Campbell?

NetApp gave out snacks every day at 6pm. These were spicy!

The Hotel

NetApp approved only two or three hotels in Bangalore for travel. Arbitrarily, I chose the Hyatt, and they treated me like a king. Every manger, director, and VIP with a business card greeted me throughout the week. Bellhops always assisted me with luggage. Room service was always quick. And it was definitely a luxury hotel. Later, I learned that I was the first NetApper to stay there, and they sought to impress me in the hopes of future business from NetApp.

Exploring on my Own

The evenings were the only chance I had to explore Bangalore. As much as I wanted to hop an auto-rickshaw, I feared getting lost or scammed, so I stayed near my hotel on foot. Around the corner was a fancy shopping mall. The first store I entered was Big Bazaar, India’s version of Walmart. As a dumb American, my first mistake was attempting to enter Big Bazaar through the exit doors – a pair of armed guards quickly corrected my mistake by pointing me to the real entrance. Inside, I meandered through the store amazed by the essentials of Indian life, most of which were not much different from my own in America. When I tried to buy a few snacks from the grocery section, I made my second mistake by presuming the cashier would give me change – nope, I lost a few rupees. Before exiting through the proper exit doors, I detoured through the clothing department and bought myself a souvenir: a black kurta with red embroidery on the collar. I’ve since worn my kurta many times at Indian festivals in Raleigh.

BIG BAZAAR, India's version of a Walmart Super Center.

Big Bazaar – it was pretty big!

Across from Big Bazaar was a movie theater. Since I had nothing else to do, I moseyed over to see what films were playing, hoping to see a Bollywood movie with English subtitles. Instead, they had Despicable Me 2 playing, which I could not resist. The ticket plus samosa and Slice mango drink cost only a few dollars – a fraction of the typical movie cost in the USA. However, when I entered the theater, an armed guard who didn’t speak English forcibly patted me down. In public places, Indians took security very seriously. My coworkers told me the security was necessary to protect against Islamic terrorists.

Bangalore had such stark dichotomies. Near my hotel on MG Road were modern buildings with posh stores and fancy restaurants and security guards, while right around the corner were dilapidated houses with refuse burning on the street. Next to billboards for mobile phone plans were temples with innumerable statues. Two constants, though, were the crowds of people everywhere and, sadly, the pollution they left behind.

A mango peddler near Mindteck.

I passed many Hindu temples while riding around in my cab.

The Food

It’s a good thing that I loved Indian food. At work, my Indian counterparts showed much hospitality through their food. Every day, they provided lunch for me. One day, Mindteck catered a full lunch buffet as part of a meeting marathon. On the day I was on-site at NetApp, a friend bought me lunch in the cafe – 100 rupees, or about $1.50. For the remaining two days, both NetApp and Mindteck planned off-site lunches at delicious Indian restaurants. I ate it all: rice, naan, roti, curry of every color, chicken, lamb, paneer, aloo, palak, and gulab jamum for dessert. Nobody was surprised that I was “non-veg”, but they did appreciate my culinary adventurousness. For me, I was surprised to discover that, in India, eating with your bare hands is normal.

Gulab jamun, hot and fresh at an Indian buffet.

In the evenings, I had two fancy meals on my own. The best was my last night at the hotel. Since NetApp covered all travel expenses, I chose to eat dinner at the Hyatt’s fancy restaurant. It was totally empty, but that meant I received the best service. After serving my chaat and my curry, the head chef himself came out to greet me. The other fancy dinner was at a South Indian restaurant on the top floor of the nearby shopping mall – prawns, yellow-green curry, and round flatbread washed down with a Kingfisher. When the waiter served the food, he put the napkin on my lap for me. A third dinner I ate at the hotel’s standard restaurant while they played live music. On the night I went to the movies, I ate a Chicken Maharaja Mac at McDonald’s – let’s just say the Big Mac is better with beef. (Come on, I’m a red-blooded American.)

The South Indian dinner at the fancy shopping mall.

These were the best shrimp I have ever eaten. Indians always call them "prawns."

Some of the best “prawns” (not shrimp) I’ve ever eaten.

The Maharaja Mac is made using chicken, since it is illegal to kill cows. The combo cost about $3.25.

The Chicken Maharaja Mac Meal. #MURICA

Goodbyes

On my last day in Bangalore, I wore my kurta. As I left in the morning, the front desk lady at the Hyatt shouted out, “Very nice kurta, Mr. Knight!” My Mindteck team planned a proper sendoff in the afternoon. We took group photos, and they gifted to me a small desk clock that I keep to this day. And they offered one final treat – an ice cream bar. There were six small scoops of various colors, and I had to taste each to guess the flavor. To everyone’s surprise, I got them all right! I didn’t have the heart to tell them that I suffered from lactose intolerance. My belly ached during the whole flight home, but the experience was well worth it.

My flight out of Bangalore departed in the wee hours of the morning. My cab picked me up from the airport shortly after midnight. I flew to Qatar, then NYC, and finally to Raleigh for a nearly day-long transit. My biggest regret was not having enough time for sightseeing in India.

The Mindteck team and me: coworkers and friends.

Lactose intolerance in India sucks. I ate the ice cream anyway. These flavors are blueberry, mango, baked apple, jackfruit, lychee, and watermelon!

The ice cream bar, from left to right by row: strawberry, banana, lychee (?), jackfruit (?), mango, and blueberry.

Lessons Learned

I gained a genuine appreciation for our offshore teams after my trip. Oftentimes, I hear American workers complain that offshore workers are “stealing jobs,” and “you get what you pay for” in quality of work. But I learned firsthand that these folks are good people just trying to make a living. They aren’t trying to steal our jobs – they’re taking the opportunities in front of them as part of a global economy. Frankly, big companies hire offshore resources so they can throw more bodies at their projects. And quality suffers greatly when any team is not co-located due to challenges with communication, time zone, technology, bureaucracy, etc. Moreover, our guys and girls wanted to be more than just hired help: they sought to learn more, improve their skills, and increase their contribution.

Overall, I was very grateful for the opportunity to visit Bangalore, and I hope to return some day!

This post is dedicated to all of my friends from India.

12 Awesome Benefits of BDD

What can BDD do for you? Why adopt a new process with a new framework? Because it’s worth it! The main benefits of BDD are better collaboration and automation. This article expands those two into a dozen awesome benefits. (If you read the BDD 101 series, then these points should look familiar.)

#1: Inclusion

BDD is meant to be collaborative. Everyone from the customer to the tester should be able to easily engage in product development. And anyone can write behavior scenarios because they are written in plain language. Scenarios are:

Requirements for product owners
Acceptance criteria for developers
Test cases for testers
Scripts for automators
Description for other stakeholders

Essentially, BDD is an enhancement of The Three Amigos.

#2: Clarity

Scenarios focus on the expected behaviors of the product. Each scenario focuses on one specific thing. Behaviors are described in plain language, and any ambiguity can be clarified with a simple conversation or Example Mapping. There’s no unreadable code or obscure technical jargon, and there’s no game of telephone. Clarity ensures the customer gets what the customer wants.

#3: Streamlining

BDD is designed to speed up the development process. Everyone involved in development relies upon the same scenarios. Scenarios are requirements, acceptance criteria, test cases, and test scripts all in one – there is no need to write any other artifact. The modular nature of Gherkin syntax expedites test automation development. Furthermore, scenarios can be used as steps to reproduce failures for defect reports.

#4: Shift Left

“Shift left” is a buzzword for testing early in the development process. Testing earlier means fewer bugs later. In BDD, test case definition inherently becomes part of the requirements phase (for waterfall) or grooming (for Agile). As soon as behavior scenarios are written, testing and automation can theoretically begin.

#5: Artifacts

Scenarios form a collection of self-documenting test cases as a result of the BDD process. This ever-growing collection forms a perfect regression test suite. Scenarios can be run manually or with automation. Any tests not automated can be added to a backlog to automate in the future.

#6: Automation

BDD frameworks make it easy to turn scenarios into automated tests. The steps are already given by the scenarios – the automation engineer simply needs to write a method/function to perform each step’s operations.

#7: Test-Driven

BDD is an evolution of TDD. Writing scenarios from the beginning enforces quality-first and test-first mindsets. BDD automation can run scenarios to fail until the feature is implemented and causes tests to pass.

#8: Code Reuse

Given-When-Then steps can be reused between scenarios. The underlying implementation for each step does not change. Automation code becomes very modular.

#9: Parameterization

Scenario steps can be parameterized to be even more reusable. For example, a step to click a button can take in its ID. Parameterization can help a team adopt a common, reusable set of steps, and it inspires healthier discussion when writing scenarios.

#10: Variation

Scenario outlines make it easy to run the same scenario with different combinations of inputs. This is a simple but powerful way to expand test coverage without code duplication, which is the bane of test automation.

#11: Momentum

BDD has a snowball effect: scenarios become easier and faster to write and automate as more step definitions are added. Scenarios typically share common steps. Sometimes, new scenarios need nothing more than different step parameters or just one new line.

#12: Adaptability

BDD scenarios are easy to update as the product changes. Plain language is easy to edit. Modular design makes changes to automation code safer. Scenarios can also be filtered by tag name to decide what runs and what doesn’t.

Which Version of Python Should I Use?

Which version of Python should I use? Now, that’s a loaded question. While the answer is simple, the explanation is more complicated.

tl;dr

For most people:

Use the latest version of Python 3.
Use the CPython implementation.
Use pipenv to manage packages and installations.
Use Visual Studio Code or PyCharm for editing code.

Which Version?

Python 2 and Python 3 are actually slightly different languages. The differences go deeper than just print statements. The What’s New in Python page on the official doc site lists all the gory details, and decent articles showcasing differences can be found here, here, and here. Although Python 3 is newer, Python 2 remains prevalent. Most popular packages use Python packaging tools to support both versions.

The Python Wiki makes it clear that Python 3 is the better choice:

Python 2.x is legacy, Python 3.x is the present and future of the language

Furthermore, Python 2 will reach end-of-life in 2020. The Python team will continue to provide bug fixes for 2.7 until 2020 (PEP 373), but there will be no new language features and no 2.8 (PEP 404). (Originally, end-of-life was planned for 2015, but it was pushed back by 5 years.) There is even a Python 2.7 Countdown clock online.

Which Implementation?

In purest terms, “Python” is a language specification. An implementation provides the language processing tools (compiler, interpreter, etc.) to run Python programs. The Hitchhiker’s Guide to Python has a great article entitled Picking an Interpreter that provides a good summary of available interpreters. Others are listed on python.org and the Python Wiki. The table below provides a quick overview of the big ones.

Implementation	Points
CPython	most widely used implementation the reference implementation has the most libraries and support implemented in C supports Python 2 and 3
PyPy	much faster than CPython much more memory efficient implemented in RPython supports Python 2 and 3
Jython	implemented in Java runs on the JVM supports Python 2 only a sandbox for Python 3 no project updates since May 2015
IronPython	implemented for .NET lets Python libs call .NET and vice versa supports Python 2
Python for .NET	integrates CPython with .NET/Mono runtime supports Python 2 and 3
Stackless Python	branch of CPython with real threading
MicroPython	optimized for microcontrollers uses a subset of the standard library

Unless you have a very specific reason, just use CPython. In fact, most people are referring to CPython when they say “Python.” CPython has the most compatibility, the widest package library, and the richest support. If you really need speed, consider PyPy.

Managing Installations

pip is the standard tool for installing Python packages. The simplest way to install Python is to install it “globally” for the system. In fact, some operating systems like macOS and Ubuntu have Python pre-installed. However, global installation has limitations:

You may want to develop packages for both versions 2 and 3.
You may not have permissions to add new packages globally.
Different projects may require different versions of packages.

These problems can be solved by using “virtual” environments. A virtual environment is like a local Python installation with a specific package set. For example, I have created virtual environments for Python as part of Jenkins build jobs, since I did not have permission to install special automation packages globally on the Jenkins slaves.

The standard virtual environment tool for Python is venv, which has been packaged with (C)Python since 3.3. (venv had a command line wrapper named pyvenv, but this was deprecated in 3.6.) Another older but still popular third-party tool is virtualenv. As explained in this Reddit post, venv is the Python-sanctioned replacement for virtualenv. However, virtualenv supports Python 2, whereas venv does not. Conda is an environment manager popular with the science and data communities, and it can support other languages in addition to Python.

That being said, there is a relatively new package manager taking the Python world by storm: pipenv. Pipenv combines pip, Pipfile, and virtualenv into an easy workflow with simple commands. Personally, I find it to be very helpful. However, it has caused some controversy (see Reddit), and it may not be applicable for all scenarios (see Chris Warrick’s article). My recommendation is to use pipenv for new projects if it meets your needs.

Editors and IDEs

After setting up your Python environment, you are ready to start programming! There are two routes to take for text editing: source code editors and integrated development environments.

Source code editors are lightweight but often include basics like syntax highlighting and basic auto-completion. They’re great for quick edits and light scripting. Many have plugins. Popular choices are Visual Studio Code, Sublime, Atom, and Notepad++. My current favorite is Visual Studio Code because the Python extensions are stellar and settings are simple – just remember to install the extensions you need! I use it personally for Django development.

For more intense development, I highly recommend an IDE like JetBrains PyCharm, PyDev for Eclipse, Wing Python IDE, or Eric. IDEs provide rich development support, especially for larger apps that use frameworks like Django, Pyramid, and SQLAlchemy. They also make testing easier with plugins for test frameworks like pytest, behave, and others. PyCharm and PyDev are particularly nice because they can integrate into their larger IDEs (IntelliJ IDEA and Eclipse, respectively) to handle more languages. Personally, I prefer PyCharm, but advanced features require a paid license.

Pythonese

The Python community throws around a few terms you should know:

Word or Phrase	Meaning
Pythonic	describes idiomatic code for Python closely related to conciseness, readability, and elegance highly recommended to use follow style guidelines
Pythonista	someone who loves the Python language often an advanced Python programmer
Pythoneer	a programmer who uses Python to solve problems
The Zen of Python	the list of guiding principles for Python’s design run “import this” to see them
The Python Software Foundation (PSF)	non-profit org keeps Python going strong support them!
PyCon	the annual Python conference held in North America GO – it will change your life! several other conferences are held worldwide
Benevolent Dictator for Life (BDFL)	Guido van Rossum the inventor of Python resigned in July 2018 but remains BDFL Emeritus

[8/6/2018: I updated the recommendations for editors and IDEs.]

[8/15/2018: I added the Pythonese section.]

[8/17/2018: I updated pipenv information.]

This article is dedicated to my good friend Sudarsan,
who originally asked me the question in the title.

BDD 101: Frameworks

Every major programming language has a BDD automation framework. Some even have multiple choices. Building upon the structural basics from the previous post, this post provides a survey of the major frameworks available today. Since I cannot possibly cover every BDD framework in depth in this 101 series, my goal is to empower you, the reader, to pick the best framework for your needs. Each framework has support documentation online justifying its unique goodness and detailing how to use it, and I would prefer not to duplicate documentation. Use this post primarily as a reference. (Check the Automation Panda BDD page for the full table of contents.)

Major Frameworks

Most BDD frameworks are Cucumber versions, JBehave derivatives inspired by Dan North, or non-Gherkin spec runners. Some put behavior scenarios into separate files, while others put them directly into the source code.

C# and Microsoft .NET

SpecFlow, created by Gáspár Nagy, is arguably the most popular BDD framework for Microsoft .NET languages. Its tagline is “Cucumber for .NET” – thus fully compliant with Gherkin. SpecFlow also has polished, well-designed hooks, context injection, and parallel execution (especially with test thread affinity). The basic package is free and open source, but SpecFlow also sells licenses for SpecFlow+ extensions. The free version requires a unit test runner like MsTest, NUnit, or xUnit.net in order to run scenarios. This makes SpecFlow flexible but also feels jury-rigged and inelegant. The licensed version provides a slick runner named SpecFlow+ Runner (which is BDD-friendly) and a Microsoft Excel integration tool named SpecFlow+ Excel. Microsoft Visual Studio has extensions for SpecFlow to make development easier.

There are plenty of other BDD frameworks for C# and .NET, too. xBehave.net is an alternative that pairs nicely with xUnit.net. A major difference of xBehave.net is that scenario steps are written directly in the code, instead of in separate text (feature) files. LightBDD bills itself as being more lightweight than other frameworks and basically does some tricks with partial classes to make the code more readable. NSpec is similar to RSpec and Mocha and uses lambda expressions heavily. Concordion offers some interesting ways to write specs, too. NBehave is a JBehave descendant, but the project appears to be dead without any updates since 2014.

Java and JVM Languages

The main Java rivalry is between Cucumber-JVM and JBehave. Cucumber-JVM is the official Cucumber version for Java and other JVM languages (Groovy, Scala, Clojure, etc.). It is fully compliant with Gherkin and generates beautiful reports. The Cucumber-JVM driver can be customized, as well. JBehave is one of the first and foremost BDD frameworks available. It was originally developed by Dan North, the “father of BDD.” However, JBehave is missing key Gherkin features like backgrounds, doc strings, and tags. It was also a pure-Java implementation before Cucumber-JVM existed. Both frameworks are widely used, have plugins for major IDEs, and distribute Maven packages. This popular but older article compares the two in slight favor of JBehave, but I think Cucumber-JVM is better, given its features and support.

The Automation panda article Cucumber-JVM for Java is a thorough guide for the Cucumber-JVM framework.

Java also has a number of other BDD frameworks. JGiven uses a fluent API to spell out scenarios, and pretty HTML reports print the scenarios with the results. It is fairly clean and concise. Spock and JDave are spec frameworks, but JDave has been inactive for years. Scalatest for Scala also has spec-oriented features. Concordion also provides a Java implementation.

JavaScript

Almost all JavaScript BDD frameworks run on Node.js. Jasmine and Mocha are two of the most popular general-purpose JS test frameworks. They differ in that Jasmine has many features included (like assertions and spies) that Mocha does not. This makes Jasmine easier to get started (good for beginners) but makes Mocha more customizable (good for power users). Both claim to be behavior-driven because they structure tests using “describe” and “it-should” phrases in the code, but they do not have the advantage of separate, reusable steps like Gherkin. Personally, I consider Jasmine and Mocha to be behavior-inspired but not fully behavior-driven.

Other BDD frameworks are more true to form. Cucumber provides Cucumber.js for Gherkin-compliant happiness. Yadda is Gherkin-like but with a more flexible syntax. Vows provides a different way to approach behavior using more formalized phrase partitions for a unique form of reusability. The Cucumber blog argues that Cucumber.js is best due to its focus on good communication through plain language steps, whereas other JavaScript BDD frameworks are more code-y. (Keep in mind, though, that Cucumber would naturally boast of its own framework.) Other comparisons are posted here, here, here, and here.

PHP

The two major BDD frameworks for PHP are Behat and Codeception. Behat is the official Cucumber version for PHP, and as such is seen as the more “pure” BDD framework. Codeception is more programmer-focused and can handle other styles of testing. There are plenty of articles comparing the two – here, here, and here (although the last one seems out of date). Both seem like good choices, but Codeception seems more flexible.

Python

Python has a plethora of test frameworks, and many are BDD. behave and lettuce are probably the two most popular players. Feature comparison is analogous to Cucumber-JVM versus JBehave, respectively: behave is practically Gherkin compliant, while lettuce lacks a few language elements. Both have plugins for major IDEs. pytest-bdd is on the rise because it integrates with all the wonderful features of pytest. radish is another framework that extends the Gherkin language to include scenario loops, scenario preconditions, and variables. All these frameworks put scenarios into separate feature files. They all also implement step definitions as functions instead of classes, which not only makes steps feel simpler and more independent, but also avoids unnecessary object construction.

Other Python frameworks exist as well. pyspecs is a spec-oriented framework. Freshen was a BDD plugin for Nose, but both Freshen and Nose are discontinued projects.

Ruby

Cucumber, the gold standard for BDD frameworks, was first implemented in Ruby. Cucumber maintains the official Gherkin language standard, and all Cucumber versions are inspired by the original Ruby version. Spinach bills itself as an enhancement to Cucumber by encapsulating steps better. RSpec is a spec-oriented framework that does not use Gherkin.

Which One is Best?

There is no right answer – the best BDD framework is the one that best fits your needs. However, there are a few points to consider when weighing your options:

What programming language should I use for test automation?
Is it a popular framework that many others use?
Is the framework actively supported?
Is the spec language compliant with Gherkin?
What type of testing will you do with the framework?
What are the limitations as compared to other frameworks?

Frameworks that separate scenario text from implementation code are best for shift-left testing. Frameworks that put scenario text directly into the source code are better for white box testing, but they may look confusing to less experienced programmers.

Personally, my favorites are SpecFlow and pytest-bdd. At LexisNexis, I used SpecFlow and Cucumber-JVM. For Python, I used behave at MaxPoint, but I have since fallen in love with pytest-bdd since it piggybacks on the wonderfulness of pytest. (I can’t wait for this open ticket to add pytest-bdd support in PyCharm.) For skill transferability, I recommend Gherkin compliance, as well.

Reference Table

The table below categorizes BDD frameworks by language and type for quick reference. It also includes frameworks in languages not described above. Recommended frameworks are denoted with an asterisk (*). Inactive projects are denoted with an X (x).

Language	Framework	Type
C	Catch	In-line Spec
C++	Igloo	In-line Spec
C# and .NET	Concordion LightBDD NBehave x NSpec SpecFlow * xBehave.net	In-line Spec In-line Gherkin Separated semi-Gherkin In-line Spec Separated Gherkin In-line Gherkin
Golang	Ginkgo	In-line Spec
Java and JVM	Cucumber-JVM * JBehave JDave x JGiven * Scalatest Spock	Separated Gherkin Separated semi-Gherkin In-line Spec In-line Gherkin In-line Spec In-line Spec
JavaScript	Cucumber.js * Yadda Jasmine Mocha Vows	Separated Gherkin Separated semi-Gherkin In-line Spec In-line Spec In-line Spec
Perl	Test::BDD::Cucumber	Separated Gherkin
PHP	Behat Codeception *	Separated Gherkin Separated or In-line
Python	behave * freshen x lettuce pyspecs pytest-bdd * radish	Separated Gherkin Separated Gherkin Separated semi-Gherkin In-line Spec Separated semi-Gherkin Separated Gherkin-plus
Ruby	Cucumber * RSpec Spinach	Separated Gherkin In-line Spec Separated Gherkin
Swift / Objective C	Quick	In-line Spec

[4/22/2018] Update: I updated info for C# and Python frameworks.

BDD 101: Automation

Better automation is one of BDD’s hallmark benefits. In fact, the main goal of BDD could be summarized as rapidly turning conceptualized behavior into automatically tested behavior. While the process and the Gherkin are universal, the underlying automation could be built using one of many frameworks.

This post explains how BDD automation frameworks work. It focuses on the general structure of the typical framework – it is not a tutorial on how to use any specific framework. However, I wrote short examples for each piece using Python’s behave framework, since learning is easier with examples. I chose to use Python here simply for its conciseness. (Check the Automation Panda BDD page for the full table of contents.)

Framework Parts

Every BDD automation framework has five major pieces:

#1: Feature Files

Gherkin feature files are very much part of the automation. They act like test scripts – each scenario is essentially a test case. Previous posts covered Gherkin in depth.

Here is an example feature file named google_search.feature:

Feature: Google Searching
  As a web surfer, I want to search Google, so that I can learn new things.
  
  # This scenario should look familiar
  @automated @google-search @panda
  Scenario: Simple Google search
    Given a web browser is on the Google page
    When the search phrase "panda" is entered
    Then results for "panda" are shown

#2: Step Definitions

A step definition is a code block that implements the logic to execute a step. It is typically a method or function with the English-y step phrase as an annotation. Step definitions can take in arguments, doc strings, and step tables. They may also make assertions to pass or fail a scenario. In most frameworks, data can be passed between steps using some sort of context object. When a scenario is executed, the driver matches each scenario step phrase to its step definition. (Most frameworks use regular expressions for phrase matching.) Thus, every step in a feature file needs a step definition.

The step definitions would be written in a Python source file like this:

from behave import *

@given('a web browser is on the Google page')
def step_impl(context):
  context.google_page.load();

@when('the search phrase "{phrase}" is entered')
def step_impl(context, phrase):
  context.google_page.search(phrase)

@then('the results for "{phrase}" are shown')
def step_impl(context, phrase):
  assert context.google_page.has_results(phrase)

#3: Hooks

Certain automation logic cannot be handled by step definitions. For example, scenarios may need special setup and cleanup operations. Most BDD frameworks provide hooks that can insert calls before or after Gherkin sections, typically filterable using tags. Hooks are similar in concept to aspect-oriented programming.

In behave, hooks are written in a Python source file named environment.py:

import page_objects
from selenium import webdriver

def before_all(context):
  context.browser = webdriver.Chrome()

def before_scenario(context):
  context.google_page = page_objects.GooglePage(context.browser)

def after_all(context):
  context.browser.quit()

#4: Support Code

Support code (a.k.a libraries or packages) refers to any code called by step definitions and hooks. Support code could be dependency packages downloaded using managers like Maven (Java), NuGet (.NET), or PyPI (Python). For example, Selenium WebDriver is a well-known package for web browser automation. Support code could also be components to assist automation, such as page objects or other design patterns. As the cliché goes, “Don’t reinvent the wheel.” Step definitions and hooks should not contain all of the logic for running the actions – they should reuse common code as much as possible.

A Python page object class from the page_objects.py module could look like this:

class GooglePage(object):
  """A page object for the Google home page"""
  
  def __init__(self, browser):
    self.browser = browser
  
  def load():
    # put code here
    pass
  
  def search(phrase):
    # put code here
    pass
  
  def has_results(phrase):
    # put code here
    return False

#5: Driver

Every automation framework has a driver that runs tests, and BDD frameworks are no different. The driver executes each scenario in a feature file independently. Whenever a failure happens, the driver reports the failure and aborts the scenario. Drivers typically have discovery mechanisms for selecting scenarios to run based on tag names or file paths.

The behave driver can be launched from the command line like this:

> behave google_search.py --tags @panda

Automation Advantages

Even if a team does not apply behavior-driven practices to its full development process, BDD test frameworks still have some significant advantages over non-BDD test frameworks. First of all, steps make BDD automation very modular and thus reusable. Each step is an independent action, much like how each scenario is an independent behavior. Once a step definition is written, it may be reused by any number of scenarios. This is crucial, since most behaviors for a feature share common actions. And all steps are inherently self-documenting, since they are written in plain language. There is a natural connection between high-level behavior and low-level implementation.

Test execution also has advantages. Tags make it very easy to select tests to run, especially from the command line. Failures are very informative as well. The driver pinpoints precisely which step failed for which scenario. And since behaviors are isolated, a failure for one scenario is less likely to affect other test scenarios than would be the case for procedure-driven tests.

All of this is explained more thoroughly in the Automation Panda article, ‑‑BDD; Automation without Collaboration.

What About Test Data?

Test data is a huge concern for any automation framework. Simple test data values may be supplied directly in Gherkin as step arguments or table values, but larger test data sets require other strategies. Support code can be used to handle test data. Read BDD 101: Test Data for more information.

Available Frameworks

There are many BDD frameworks out there. The next post will introduce a few major frameworks for popular languages.

BDD 101: Behavior-Driven Agile

Previous posts in this 101 series have focused heavily upon Gherkin. They may have given the impression that Gherkin is merely a testing language, and that BDD is a test framework. Wrong: BDD is a full development process! Its practices complement Agile software development by bringing clearer communication and shift left testing. As such, BDD is a refinement, not an overhaul, of the Agile process. This post explains how to add behavior-driven practices to the Agile process. (Check the Automation Panda BDD page for the full table of contents.)

Common Agile Problems

User stories can sometimes seem like a game of telephone: the product owner says one thing, the developer makes another thing, and the tester writes a bad test. When the test fails, the tester goes back to the developer for clarification, who in turn goes back to the product owner. Hopefully, the misunderstanding is corrected before demo day, but time is nevertheless lost and resources are burned. Acceptance criteria for a user story should clarify how things should be, but often they are poorly stated or entirely missing.

Another Agile problem, especially in Scrum, is incomplete testing. Development work is often treated like a pipeline: design -> implement -> review -> test -> automate -> DONE. And stories have deadlines. When coding runs late, testers may not get testing done, let alone test automation. Add a game of telephone, and in-sprint testing can become perpetually impeded.

BDD to the Rescue

BDD solves both of these Agile problems beautifully through process efficiency. Let me break this down from a behavior-oriented perspective:

Acceptance criteria specify feature behavior.
Test cases validate feature behavior.
Gherkin feature files document feature behavior.

Therefore, when written in Gherkin, acceptance criteria are test cases! The Gherkin feature file is the formal specification of both the acceptance criteria and the test cases for a user story. One artifact covers both things!

The Behavior-Driven Three Amigos

“The Three Amigos” refers to the three primary roles engaged in producing software: business, development, and testing. Each role brings its own perspective to the product, and good software results when all can collaborate well. A common Agile practice is to hold meetings with the Three Amigos as part of grooming or planning.

The BDD process is an enhanced implementation of The Three Amigos. All stakeholders can participate in behavior-driven development because Gherkin is like plain English. Feature files mean different things to different people:

They are requirements for product owners.
They are acceptance criteria for developers.
They are test cases for testers.
They are scripts for automators.
They are descriptions for other stakeholders.

Thus, BDD fosters healthy collaboration because feature files are pertinent to all stakeholders (or “amigos”). Features files are like receipts – they’re a “proof of purchase” for the team. They document precisely what will be delivered.

Behavior-Driven Sprints

To see why this is a big deal, see what happens in a behavior-driven sprint:

Feature files begin at grooming. As the team prepares user stories in the backlog, acceptance criteria are written in Gherkin. Since Gherkin is easy to read, even non-technical people (namely product owners) can contribute when The Three Amigos meet. Techniques like Example Mapping can make this collaboration easy and efficient.
During planning, all stakeholders get a good understanding of how a feature should behave. Better conversations can happen. Clarifications can be written directly into feature files.
When the sprint starts, feature files are already written. Developers know what must be developed. Testers know what must be tested. There’s no ambiguity.
Test automation can begin immediately because the scenario steps are already written. Some step definitions may already exist, too. New step definitions are typically understandable enough to implement even before the developer commits the product code.
Manual testers know from the start which tests will be automated and which must be run manually. This enables them to make better test plans. It also frees them to do more exploratory testing, which is better for finding bugs.

Overall, Gherkinized acceptance criteria streamline development and improve quality. The team is empowered to shift left. On-time story completion and in-sprint automation become the norm.

(Arguably, these benefits would happen in Kanban as well as in Scrum.)

New Rules

In order to reap the benefits of BDD, the Agile process needs a few new rules. First, formalize all acceptance criteria as Gherkin feature files. In retrospect, it should seem odd that the user story itself is formalized (“As a ___, I want ___, so that ___”) if the acceptance criteria is not (“Given-When-Then”). Writing feature files adds more work to grooming, but it enables the collaboration and shift left testing.

Second, never commit to completing a user story that doesn’t have Gherkinized acceptance criteria. Don’t become sloppy out of expediency. Use the planning meeting as an accountability measure.

Third, include test automation in the definition of done. Stories should not be accepted without their tests completed and automated. Automation in the present guarantees regression coverage in the future, which in turn allows teams to respond to change safely and quickly.

Advanced Topics

Check out these other articles for further learning:

More Agility through Automation

BDD truly improves the Agile process by fixing its shortcomings. The next step is to learn BDD automation, which will be covered in the next post. Until then, I’ll leave this gem here:

BDD 101: Writing Good Gherkin

So, you and your team have decided to make test automation a priority. You plan to use behavior-driven development to shift left with testing. You read the BDD 101 Series up through the previous post. You picked a good language for test automation. You even peeked at Cucumber-JVM or another BDD framework on your own. That’s great! Big steps! And now, you are ready to write your first Gherkin feature file. You fire open Atom with a Gherkin plugin or Notepad++ with a Gherkin UDL, you type “Given” on the first line, and…

Writer’s block. How am I supposed to write my Gherkin steps?

Good Gherkin feature files are not easy to write at first. Writing is definitely an art. With some basic pointers, and a bit of practice, Gherkin becomes easier. This post will cover how to write top-notch feature files. (Check the Automation Panda BDD page for the full table of contents.)

The Golden Gherkin Rule: Treat other readers as you would want to be treated. Write Gherkin so that people who don’t know the feature will understand it.

Proper Behavior

The biggest mistake BDD beginners make is writing Gherkin without a behavior-driven mindset. They often write feature files as if they are writing “traditional” procedure-driven functional tests: step-by-step instructions with actions and expected results. HP ALM, qTest, AccelaTest, and many other test repository tools store tests in this format. These procedure-driven tests are often imperative and trace a path through the system that covers multiple behaviors. As a result, they may be unnecessarily long, which can delay failure investigation, increase maintenance costs, and create confusion.

For example, let’s consider a test that searches for images of pandas on Google. Below would be a reasonable test procedure:

Open a web browser.
1. Web browser opens successfully.
Navigate to https://www.google.com/.
1. The web page loads successfully and the Google image is visible.
Enter “panda” in the search bar.
1. Links related to “panda” are shown on the results page.
Click on the “Images” link at the top of the results page.
1. Images related to “panda” are shown on the results page.

I’ve seen many newbies translate a test like this into Gherkin like the following:

# BAD EXAMPLE! Do not copy.
Feature: Google Searching

  Scenario: Google Image search shows pictures
    Given the user opens a web browser
    And the user navigates to "https://www.google.com/"
    When the user enters "panda" into the search bar
    Then links related to "panda" are shown on the results page
    When the user clicks on the "Images" link at the top of the results page
    Then images related to "panda" are shown on the results page

This scenario is terribly wrong. All that happened was that the author put BDD buzzwords in front of each step of the traditional test. This is not behavior-driven, it is still procedure-driven.

The first two steps are purely setup: they just go to Google, and they are strongly imperative. Since they don’t focus on the desired behavior, they can be reduced to one declarative step: “Given a web browser is at the Google home page.” This new step is friendlier to read.

After the Given step, there are two When-Then pairs. This is syntactically incorrect: Given-When-Then steps must appear in order and cannot repeat. A Given may not follow a When or Then, and a When may not follow a Then. The reason is simple: any single When-Then pair denotes an individual behavior. This makes it easy to see how, in the test above, there are actually two behaviors covered: (1) searching from the search bar, and (2) performing an image search. In Gherkin, one scenario covers one behavior. Thus, there should be two scenarios instead of one. Any time you want to write more than one When-Then pair, write separate scenarios instead. (Note: Some BDD frameworks may allow disordered steps, but it would nevertheless be anti-behavioral.)

This splitting technique also reveals unnecessary behavior coverage. For instance, the first behavior to search from the search bar may be covered in another feature file. I once saw a scenario with about 30 When-Then pairs, and many were duplicate behaviors.

Do not be tempted to arbitrarily reassign step types to make scenarios follow strict Given-When-Then ordering. Respect the integrity of the step types: Givens set up initial state, Whens perform an action, and Thens verify outcomes. In the example above, the first Then step could have been turned into a When step, but that would be incorrect because it makes an assertion. Step types are meant to be guide rails for writing good behavior scenarios.

The correct feature file would look something like this:

Feature: Google Searching

  Scenario: Search from the search bar
    Given a web browser is at the Google home page
    When the user enters "panda" into the search bar
    Then links related to "panda" are shown on the results page

  Scenario: Image search
    Given Google search results for "panda" are shown
    When the user clicks on the "Images" link at the top of the results page
    Then images related to "panda" are shown on the results page

The second behavior arguably needs the first behavior to run first because the second needs to start at the search result page. However, since that is merely setup for the behavior of image searching and is not part of it, the Given step in the second scenario can basically declare (declaratively) that the “panda” search must already be done. Of course, this means that the “panda” search would be run redundantly at test time, but the separation of scenarios guarantees behavior-level independence.

The Cardinal Rule of BDD: One Scenario, One Behavior!

Remember, behavior scenarios are more than tests – they also represent requirements and acceptance criteria. Good Gherkin comes from good behavior.

(For deeper information about the Cardinal Rule of BDD and multiple When-Then pairs per scenario, please refer to my article, Are Gherkin Scenarios with Multiple When-Then Pairs Okay?)

Phrasing Steps

How you write a step matters. If you write a step poorly, it cannot easily be reused. Thankfully, some basic rules maintain consistent phrasing and maximum reusability.

Write all steps in third-person point of view. If first-person and third-person steps mix, scenarios become confusing. I even dedicated a whole blog post entirely to this point: Should Gherkin Steps Use First-Person or Third-Person? TL;DR: just use third-person at all times.

Write steps as a subject-predicate action phrase. It may tempting to leave parts of speech out of a step line for brevity, especially when using Ands and Buts, but partial phrases make steps ambiguous and more likely to be reused improperly. For example, consider the following example:

# BAD EXAMPLE! Do not copy.
Feature: Google Searching

  Scenario: Google search result page elements
    Given the user navigates to the Google home page
    When the user entered "panda" at the search bar
    Then the results page shows links related to "panda"
    And image links for "panda"
    And video links for "panda"

The final two And steps lack the subject-predicate phrase format. Are the links meant to be subjects, meaning that they perform some action? Or, are they meant to be direct objects, meaning that they receive some action? Are they meant to be on the results page or not? What if someone else wrote a scenario for a different page that also had image and video links – could they reuse these steps? Writing steps without a clear subject and predicate is not only poor English but poor communication.

Also, use appropriate tense and phrasing for each type of step. For simplicity, use present tense for all step types. Rather than take a time warp back to middle school English class, let’s illustrate tense with a bad example:

# BAD EXAMPLE! Do not copy.
Feature: Google Searching

  Scenario: Simple Google search
    Given the user navigates to the Google home page
    When the user entered "panda" at the search bar
    Then links related to "panda" will be shown on the results page

The Given step above uses present tense, but its subject is misleading. It indicates an action when it says, “Given the user navigates.” Actions imply the exercise of behavior. However, Given steps are meant to establish an initial state, not exercise a behavior. This may seem like a trivial nuance, but it can confuse feature file authors who may not be able to tell if a step is a Given or When. A better phrasing would be, “Given the Google home page is displayed.” It establishes a starting point for the scenario. Use present tense with an appropriate subject to indicate a state rather than an action.

The When step above uses past tense when it says, “The user entered.” This indicates that an action has already happened. However, When steps should indicate that an action is presently happening. Plus, past tense here conflicts with the tenses used in the other steps.

The Then step above uses future tense when it says, “The results will be shown.” Future tense seems practical for Then steps because it indicates what the result should be after the current action is taken. However, future tense reinforces a procedure-driven approach because it treats the scenario as a time sequence. A behavior, on the other hand, is a present-tense aspect of the product or feature. Thus, it is better to write Then steps in the present tense.

The corrected example looks like this:

Feature: Google Searching

  Scenario: Simple Google search
    Given the Google home page is displayed
    When the user enters "panda" into the search bar
    Then links related to "panda" are shown on the results page

And note, all steps are written in third-person. Read Should Gherkin Steps use Past, Present, or Future Tense? to learn more.

Good Titles

Good titles are just as important as good steps. The title is like the face of a scenario – it’s the first thing people read. It must communicate in one concise line what the behavior is. Titles are often logged by the automation framework as well. Specific pointers for writing good scenario titles are given in my article, Good Gherkin Scenario Titles.

Choices, Choices

Another common misconception for beginners is thinking that Gherkin has an “Or” step for conditional or combinatorial logic. People may presume that Gherkin has “Or” because it has “And”, or perhaps programmers want to treat Gherkin like a structured language. However, Gherkin does not have an “Or” step. When automated, every step is executed sequentially.

Below is a bad example based on a classic Super Mario video game, showing how people might want to use “Or”:

# BAD EXAMPLE! Do not copy.
Feature: SNES Mario Controls

  Scenario: Mario jumps
    Given a level is started
    When the player pushes the "A" button
    Or the player pushes the "B" button
    Then Mario jumps straight up

Clearly, the author’s intent is to say that Mario should jump when the player pushes either of two buttons. The author wants to cover multiple variations of the same behavior. In order to do this the right way, use Scenario Outline sections to cover multiple variations of the same behavior, as shown below:

Feature: SNES Mario Controls

  Scenario Outline: Mario jumps
    Given a level is started
    When the player pushes the "<letter>" button
    Then Mario jumps straight up
    
    Examples: Buttons
      | letter |
      | A      |
      | B      |

The Known Unknowns

Test data can be difficult to handle. Sometimes, it may be possible to seed data in the system and write tests to reference it, but other times, it may not. Google search is the prime example: the result list will change over time as both Google and the Internet change. To handle the known unknowns, write scenarios defensively so that changes in the underlying data do not cause test runs to fail. Furthermore, to be truly behavior-driven, think about data not as test data but as examples of behavior.

Consider the following example from the previous post:

Feature: Google Searching
  
  Scenario: Simple Google search
    Given a web browser is on the Google page
    When the search phrase "panda" is entered
    Then results for "panda" are shown
    And the following related results are shown
      | related       |
      | Panda Express |
      | giant panda   |
      | panda videos  |

This scenario uses a step table to explicitly name results that should appear for a search. The step with the table would be implemented to iterate over the table entries and verify each appeared in the result list. However, what if Panda Express were to go out of business and thus no longer be ranked as high in the results? (Let’s hope not.) The test run would then fail, not because the search feature is broken, but because a hard-coded variation became invalid. It would be better to write a step that more intelligently verified that each returned result somehow related to the search phrase, like this: “And links related to ‘panda’ are shown on the results page.” The step definition implementation could use regular expression parsing to verify the presence of “panda” in each result link.

Another nice feature of Gherkin is that step definitions can hide data in the automation when it doesn’t need to be exposed. Step definitions may also pass data to future steps in the automation. For example, consider another Google search scenario:

Feature: Google Searching

  Scenario: Search result linking
    Given Google search results for "panda" are shown
    When the user clicks the first result link
    Then the page for the chosen result link is displayed

Notice how the When step does not explicitly name the value of the result link – it simply says to click the first one. The value of the first link may change over time, but there will always be a first link. The Then step must know something about the chosen link in order to successfully verify the outcome, but it can simply reference it as “the chosen result link”. Behind the scenes, in the step definitions, the When step can store the value of the chosen link in a variable and pass the variable forward to the Then step.

Handling Test Data

Some types of test data should be handled directly within the Gherkin, but other types should not. Remember that BDD is specification by example – scenarios should be descriptive of the behaviors they cover, and any data written into the Gherkin should support that descriptive nature. Read Handling Test Data in BDD for comprehensive information on handling test data.

Less is More

Scenarios should be short and sweet. I typically recommend that scenarios should have a single-digit step count (<10). Long scenarios are hard to understand, and they are often indicative of poor practices. One such problem is writing imperative steps instead of declarative steps. I have touched on this topic before, but I want to thoroughly explain it here.

Imperative steps state the mechanics of how an action should happen. They are very procedure-driven. For example, consider the following When steps for entering a Google search:

When the user scrolls the mouse to the search bar
And the user clicks the search bar
And the user types the letter “p”
And the user types the letter “a”
And the user types the letter “n”
And the user types the letter “d”
And the user types the letter “a”
And the user types the ENTER key

Now, the granularity of actions may seem like overkill, but it illustrates the point that imperative steps focus very much on how actions are taken. Thus, they often need many steps to fully accomplish the intended behavior. Furthermore, the intended behavior is not always as self-documented as with declarative steps.

Declarative steps state what action should happen without providing all of the information for how it will happen. They are behavior-driven because they express action at a higher level. All of the imperative steps in the example above could be written in one line: “When the user enters ‘panda’ at the search bar.” The scrolling and keystroking is implied, and it will ultimately be handled by the automation in the step definition. When trying to reduce step count, ask yourself if your steps can be written more declaratively.

Another reason for lengthy scenarios is scenario outline abuse. Scenario outlines make it all too easy to add unnecessary rows and columns to their Examples tables. Unnecessary rows waste test execution time. Extra columns indicate complexity. Both should be avoided. Below are questions to ask yourself when facing an oversized scenario outline:

Does each row represent an equivalence class of variations?
- For example, searching for “elephant” in addition to “panda” does not add much test value.
Does every combination of inputs need to be covered?
- N columns with M inputs each generates M^N possible combinations.
- Consider making each input appear only once, regardless of combination.
Do any columns represent separate behaviors?
- This may be true if columns are never referenced together in the same step.
- If so, consider splitting apart the scenario outline by column.
Does the feature file reader need to explicitly know all of the data?
- Consider hiding some of the data in step definitions.
- Some data may be derivable from other data.

These questions are meant to be sanity checks, not hard-and-fast rules. The main point is that scenario outlines should focus on one behavior and use only the necessary variations.

Style and Structure

While style often takes a backseat during code review, it is a factor that differentiates good feature files from great feature files. In a truly behavior-driven team, non-technical stakeholders will rely upon feature files just as much as the engineers. Good writing style improves communication, and good communication skills are more than just resume fluff.

Below are a number of tidbits for good style and structure:

Focus a feature on customer needs.
Limit one feature per feature file. This makes it easy to find features.
Limit the number of scenarios per feature. Nobody wants a thousand-line feature file. A good measure is a dozen scenarios per feature.
Limit the number of steps per scenario to less than ten.
Limit the character length of each step. Common limits are 80-120 characters.
Use proper spelling.
Use proper grammar.
Capitalize Gherkin keywords.
Capitalize the first word in titles.
Do not capitalize words in the step phrases unless they are proper nouns.
Do not use punctuation (specifically periods and commas) at the end of step phrases.
Use single spaces between words.
Indent the content beneath every section header.
Separate features and scenarios by two blank lines.
Separate examples tables by 1 blank line.
Do not separate steps within a scenario by blank lines.
Space table delimiter pipes (“|”) evenly.
Adopt a standard set of tag names. Avoid duplicates.
Write all tag names in lowercase, and use hyphens (“-“) to separate words.
Limit the length of tag names.

Without these rules, you might end up with something like this:

# BAD EXAMPLE! Do not copy.

 Feature: Google Searching
     @AUTOMATE @Automated @automation @Sprint32GoogleSearchFeature
 Scenario outline: GOOGLE STUFF
Given a Web Browser is on the Google page,
 when The seach phrase "<phrase>" Enter,

 Then  "<phrase>" shown.
and The relatedd   results include "<related>".
Examples: animals
 | phrase | related |
| panda | Panda Express        |
| elephant    | elephant Man  |

Don’t do this. It looks horrible. Please, take pride in your profession. While the automation code may look hairy in parts, Gherkin files should look elegant.

Gherkinize Those Behaviors!

With these best practices, you can write Gherkin feature files like a pro. Don’t be afraid to try: nobody does things perfectly the first time. As a beginner, I broke many of the guidelines I put in this post, but I learned as I went. Don’t give up if you get stuck. Always remember the Golden Gherkin Rule and the Cardinal Rule of BDD!

This is the last of three posts in the series focused exclusively on Gherkin. The next post will address how to adopt behavior-driven practices into the Agile software development process.