TDD

What is BDD, and How Do We Practice It? (Webinar + Q&A)

On March 18, 2019, I gave a webinar entitled, “What is Behavior-Driven Development, and How Do We Practice It?” in collaboration with Paul Merrill and his company, Beaufort Fairmont. It was both a pleasure and an honor to do this webinar with them. Paul is a top-notch test automation expert, and Beaufort Fairmont is doing really exciting things. Check out their two-day BDD training offering, as well as their blog and other webinars.

To see my webinar recording, register here.

During the webinar, attendees asked more questions than we could answer. I’m excited that so many people asked questions. My answers are below.

Questions about Process

How is BDD different from TDD (Test-Driven Development)?

BDD is an evolution of TDD. In TDD, developers (1) write unit tests and watch them fail, (2) develop the feature to make the tests pass, (3) refactor the code to make it stronger, and (4) repeat the cycle. In BDD, teams do this same loop with feature tests (a.k.a “acceptance” or “black-box” tests) as well as unit tests. Furthermore, BDD adds shift left practices like Example Mapping and Specification by Example so that teams know what they are doing and focus on developing the right things.

Check out Dan North’s article, Introducing BDD, for a more thorough answer.

Can BDD be used with manual testing?

Yes! BDD is not merely an automation tool – it is a set of pragmatic practices to help teams develop better software. Gherkin scenarios are first and foremost behavior specs that help a team’s collaboration and accountability. They function secondarily as test cases that can be executed either manually or with automation.

Can we use BDD with technical stories or backend features?

Yes! If you can describe it, then you can do it.

How many Gherkin scenarios should one story have?

There’s no hard rule, but I recommend no more than a handful of rules per story, and no more than a handful of examples per rule. If you do Example Mapping and feel overwhelmed by the number of cards for a story, then the story should probably be broken into smaller stories.

Should we do Example Mapping for every story? Spending 20-30 minutes for each story would take a long time.

Try doing Example Mapping on one or two stories to start. The first time is always rough, but as you iterate on it, you’ll get better as a team. Even though Example Mapping has an upfront time cost, it will save a lot of time later in the sprint because (a) acceptance criteria is clear, (b) tests are already written, and (c) everyone has a mutual understanding of the story. The team won’t suffer through the inefficiencies of miscommunication and poor planning. You may even want to replace planning meeting with Example Mapping meetings.

What metrics should we use with BDD?

All metrics are flawed, but some metrics are useful. All the standard testing and Agile metrics still apply: code coverage, story velocity, etc. Here are some additional metrics you may consider for BDD:

  • the percentage of stories that undergo Example Mapping before the sprint
  • the number of rules and examples that get “missed” during Example Mapping and need to be added later
  • the percentage of Gherkin scenarios that get automated in the sprint

If you choose to track metrics, make sure their feedback is used to improve team practices. For more info on metrics, please read my Quality Metrics 101 series.

What were the resources you recommended at the end of the webinar?

Questions about Tools

What test management tools should we use with BDD?

I’m sure there are BDD plugins for test management tools, but I don’t have any that I can personally recommend. To be honest, I try to stay away from large test management tools like HP ALM, qTest, VersionOne. When doing BDD, the Gherkin feature files themselves should be the single source of truth for feature-level tests, and they should be version-controlled in a repository. Don’t fall into the trap of slapping “Given-When-Then” keywords onto existing functional tests – that’s not BDD.

Does Jira support Example Mapping?

I have not personally used any Jira plugin for Example Mapping. It looks like there is an Easy Agile User Story Maps plugin that is similar to but slightly different from Example Mapping.

Are there other good tools for BDD and Example Mapping?

What’s the difference between Gherkin, Cucumber, and SpecFlow?

  • Gherkin is the Given-When-Then spec language.
  • Cucumber is a company and its eponymous test framework that uses Gherkin.
  • SpecFlow is Cucumber for .NET.

Questions about Testing

Can BDD test frameworks be used for unit testing?

Yes, but I don’t recommend it. BDD frameworks shine for black-box feature testing. They’re a bit too verbose for code-level unit tests. Read BDD 101: Unit, Integration, and End-to-End Tests for more info.

Can BDD test frameworks be used for integration testing?

Yes! See BDD 101: Unit, Integration, and End-to-End Tests.

How long should Gherkin scenarios be?

Scenarios should be bite-sized. Each scenario should focus on one individual behavior. There’s no hard rule, but I recommend single-digit step counts. Read BDD 101: Writing Good Gherkin for more info.

What are “step definitions” in Cucumber?

Step definitions are the methods in the automation code that execute the steps. When a BDD framework runs a Gherkin scenario as a test, it “glues” each step to a step definition based on some sort of string matching.

How can we minimize duplicate code within a BDD test framework?

Know your steps. Always search for existing steps before writing new steps. Refactor existing steps whenever appropriate. Reuse steps when writing new scenarios. Do pair programming or mob programming when writing scenarios. Put scenarios through code reviews. Apply good coding practices – remember, test automation is software.

I write Gherkin scenarios, but I don’t write test automation code. What’s the best way to write Gherkin scenarios so that they can be automated?

Do pair programming with the automation engineers to write Gherkin scenarios together. Become familiar with existing steps by reading and searching feature files. Otherwise, the Gherkin steps you write in isolation might not be usable. Remember, BDD is a team effort!

The examples in the webinar were all fairly basic. Do you have any examples with more complex systems?

I have some example projects on GitHub in Python and Java with some basic unit, integration, and end-to-end tests, but I don’t have any large-scale examples that I can share publicly.

We wrote hundreds of SpecFlow tests without the other Amigos. Now, there are large test gaps, and many steps aren’t reusable. What should we do?

I’m sorry to hear that. It’s not an uncommon story. There are two paths: (1) refactoring or (2) starting over. Without really knowing the situation, I don’t think it’s my place to say which way is better. Here are some questions to help guide your decision:

  • What are your goals for testing and automation?
  • What’s your overall quality and testing strategy?
  • What parts of the code base are salvageable?
  • What parts of the code base should be removed?
  • If you started again from scratch, what would you do differently to make sure the same problems don’t reoccur?

I strongly recommend taking the Setting a Foundation for Successful Test Automation course from Test Automation University. (It’s free.) I also gave a talk about this very problem, Egad! How Do We Start Writing (Better) Tests?, at a few Python conferences.

We have a large BDD test suite with heavy coupling and slow execution times. The business amigos have also left the company. Should we try to fix what we have or just start over?

Sorry to hear that; same answer as before.

Final Questions

Why do you call yourself the “Automation Panda”?

Pandas are awesome. Everybody loves them. And nobody forgets my moniker.

Where can I get team training in BDD?

Beaufort Fairmont provides a one- or two-day course in BDD and writing Gherkin. Sign up for more information here.

BDD 101: Introducing BDD

Series Overview

BDD 101 is a blog series to teach the basics of behavior-driven development. It is both a “getting started” guide for BDD beginners, as well as a best-practice reference for pros. I wrote this series for anyone involved in the daily duties of software development: developers, testers, scrum masters, product owners, and managers. The content in this series comes from my experiences using BDD for many projects. It focuses on Gherkin-based specification, and test automation will be a major theme. If this series is for you, then let’s dive in!

The BDD 101 table of contents is given on the Automation Panda BDD page. Note that some articles in the series were posted months apart and will not all appear together using the “previous” and “next” article arrows.

The Big BDD Picture: The main goals of BDD are collaboration and automation.

What is a Behavior?

A behavior is how a product or feature operates. It is defined as a scenario of inputs, actions, and outcomes. A product or feature exhibits countless behaviors. Identifying behaviors individually brings clarity and simplicity. It also helps explain how behaviors are related. Below are examples of behaviors:

  • Logging into a web page
  • Clicking links on a navigation bar
  • Submitting forms
  • Making successful service calls
  • Receiving expected errors

Separating individual behaviors makes it easy to define a system without unnecessary repetition. For example, there may be multiple ways to navigate to the same page.

Nav Behaviors

Search from a text field and searching directly from URL parameters both lead to the same results page.

What is BDD?

Behavior­-Driven Development (BDD) is a test­-centric software development process that grew out of Test-­Driven Development (TDD). It has been around since roughly the mid-2000s. BDD focuses on clearly identifying the desired behavior of a feature from the very start. Behaviors are identified using specification by example: behavior specs are written to illustrate the desired behavior with realistic examples, rather than being written with abstract, generic jargon. They serve as both the product’s requirements/acceptance criteria (before development) and its test cases (after development). Gherkin is one of the most popular languages for writing formal behavior specifications – it captures behaviors as “Given-When-Then” scenarios. With the help of automation tools, scenarios can easily be turned into automated test cases. Anybody from engineers to product owners can write BDD scenarios, since they are just English phrases. BDD keeps developers focused on delivering precisely what the product owner wants. It also expedites testing. As such, BDD pairs nicely with Agile Software Development.

Quick Points

  • BDD is specification by example.
    • When someone says “BDD”, immediately think of “Given-When-Then”.
  • BDD focuses on behavior first.
    • Behavior scenarios are the cornerstone of BDD.
  • BDD is a refinement of the Agile process, not an overhaul.
    • It formalizes acceptance criteria and test coverage.
  • BDD is a paradigm shift.
    • Behaviors become the team’s main focus.

The Origins of BDD

The following quote comes from an article entitled Introducing BDD, written by Dan North (the “Father of BDD”) in March 2006:

I had a problem. While using and teaching agile practices like test-driven development (TDD) on projects in different environments, I kept coming across the same confusion and misunderstandings. Programmers wanted to know where to start, what to test and what not to test, how much to test in one go, what to call their tests, and how to understand why a test fails.

The deeper I got into TDD, the more I felt that my own journey had been less of a wax-on, wax-off process of gradual mastery than a series of blind alleys. I remember thinking “If only someone had told me that!” far more often than I thought “Wow, a door has opened.” I decided it must be possible to present TDD in a way that gets straight to the good stuff and avoids all the pitfalls.

My response is behaviour-driven development (BDD). It has evolved out of established agile practices and is designed to make them more accessible and effective for teams new to agile software delivery. Over time, BDD has grown to encompass the wider picture of agile analysis and automated acceptance testing.

12 Awesome Benefits

BDD improves the development process in a dozen ways:

Inclusion Anyone can write BDD scenarios, because they are written in plain English. Think of The Three Amigos.
Clarity Scenarios focus specifically on the expected behavior of the product under development, resulting in less ambiguity for what to develop.
Streamlining Requirements = acceptance criteria = test cases. Modular syntax expedites automation as well.
Shift-Left Test case definition inherently becomes part of grooming.
Artifacts Scenarios form a collection of test cases. Any tests not automated can be added to a known automation backlog.
Automation BDD frameworks make it easy to turn scenarios into automated tests.
Test­-Driven Most BDD frameworks can run scenarios to fail until the feature is implemented.
Code Reuse “Given-When­-Then” steps can be reused between scenarios.
Parameterization Steps can be parameterized. For example, a step to click a button can take in its ID.
Variation Using parameters, example tables make it easy to run the same scenario with different combinations of inputs.
Momentum Scenarios become easier and faster to write and automate as more step definitions are added.
Adaptability Scenarios are easy to rewrite as the products and features change.

Testing Recommendations

Since BDD focuses on actual feature behavior, behavior specs are best for higher-level, functional, black box tests. For example, BDD is great for testing APIs and web UIs. Gherkin excels for acceptance testing. However, behavior specs would be overkill for unit tests, and it is also not a good choice for performance tests that focus on metrics and not pass/fail results. Read more about this in the article BDD 101: Unit, Integration, and End-to-End Tests.

Next Steps

Lost yet? Don’t worry – this first post presented a lot of information. Things will make much more sense after learning how to write Gherkin test scenarios, which will be covered in the next post in this 101 series.