Author: Andy Knight

I'm a software engineer who specializes in test automation.

The Automation Panda Origin Story

In February 2021, Matthew Weeks interviewed me for the Work in Programming podcast. Matthew asked all sorts of questions about my story – how I got into programming, what I learned at different companies, and why I started blogging and speaking. I greatly enjoyed our conversation, so much so that it lasted an hour and a half!

If you’re interested to hear how my career has gone from high school to the present day, please give it a listen. There are some juicy anecdotes along the way. The link is below. Many thanks to Matthew for hosting and editing the interview. Be sure to listed to other Work in Programming interviews, too!

Work in Programming by Matthew Weeks: Andy Knight – Automation Panda origin story, BDD, test automation before it was cool

Solving: How to write good UI interaction tests? #GivenWhenThenWithStyle

Writing good Gherkin is a passion of mine. Good Gherkin means good behavior specification, which results in better features, better tests, and ultimately better software. To help folks improve their Gherkin skills, Gojko Adzic and SpecFlow are running a series of #GivenWhenThenWithStyle challenges. I love reading each new challenge, and in this article, I provide my answer to one of them.

The Challenge

Challenge 20 states:

This week, we’re looking into one of the most common pain points with Given-When-Then: writing automated tests that interact with a user interface. People new to behaviour driven development often misunderstand what kind of behaviour the specifications should describe, and they write detailed user interactions in Given-When-Then scenarios. This leads to feature files that are very easy to write, but almost impossible to understand and maintain.

Here’s a typical example:

Scenario: Signed-in users get larger capacity
 
Given a user opens https://www.example.com using Chrome
And the user clicks on "Upload Files"
And the page reloads
And the user clicks on "Spreadsheet Formats"
Then the buttons "XLS" and "XLSX" show
And the user clicks on "XLSX"
And the user selects "500kb-sheet.xlsx"
Then the upload completes
And the table "Uploaded Files" contains a cell with "500kb-sheet.xlsx" 
And the user clicks on "XLSX"
And the user selects "1mb-sheet.xlsx"
Then the upload fails
And the table "Uploaded Files" does not contain a cell with "1mb-sheet.xlsx" 
And the user clicks on "Login"
And the user enters "testuser123" into the "username" field
And the user enters "$Pass123" into the "password" field
And the user clicks on "Sign in"
And the page reloads
Then the table "Uploaded Files" contains a cell with "500kb-sheet.xlsx" 
And the table "Uploaded Files" does not contain a cell with "1mb-sheet.xlsx" 
And the user clicks on "spreadsheet formats"
Then the buttons "XLS" and "XLSX" show
And the user clicks on "XLSX"
And the user selects "1mb-sheet.xlsx"
Then the upload completes
And the table "Uploaded Files" contains a cell with "1mb-sheet.xlsx" 
And the table "Uploaded Files" contains a cell with "500kb-sheet.xlsx"

A common way to avoid such issues is to rewrite the specification to avoid the user interface completely. We’ve looked into that option several times in this article series. However, that solution only applies if the risk we’re testing is not in the user interface, but somewhere below. To make this challenge more interesting, let’s say that we actually want to include the user interface in the test, since the risk is in the UI interactions.

Indeed, most behavior-driven practitioners would generally recommend against phrasing steps using language specific to the user interface. However, there are times when testing a user interface itself is valid. For example, I work at PrecisionLender, a Q2 Company, and our main web app is very heavy on the front end. It has many, many interconnected fields for pricing commercial lending opportunities. My team has quite a few tests to cover UI-centric behaviors, such as verifying that entering a new interest rate triggers recalculation for summary amounts. If the target behavior is a piece of UI functionality, and the risk it bears warrants test coverage, then so be it.

Let’s break down the example scenario given above to see how to write Gherkin with style for user interface tests.

Understanding Behavior

Behavior is behavior. If you can describe it, then you can do it. Everything exhibits behavior, from the source code itself to the API, UIs, and full end-to-end workflows. Gherkin scenarios should use verbiage that reflects the context of the target behavior. Thus, the example above uses words like “click,” “select,” and “open.” Since the scenario explicitly covers a user interface, I think it is okay to use these words here. What bothers me, however, are two apparent code smells:

  1. The wall of text
  2. Out-of-order step types

The first issue is the wall of text this scenario presents. Walls of text are hard to read because they present too much information at once. The reader must take time to read through the whole chunk. Many readers simply read the first few lines and then skip the remainder. The example scenario has 27 Given-When-Then steps. Typically, I recommend Gherkin scenarios to have single-digit line length. A scenario with less than 10 steps is easier to understand and less likely to include unnecessary information. Longer scenarios are not necessarily “wrong,” but their longer lengths indicate that, perhaps, these scenarios could be rewritten more concisely.

The second issue in the example scenario is that step types are out of order. Given-When-Then is a formula for success. Gherkin steps should follow strict Given → When → Then ordering because this ordering demarcates individual behaviors. Each Gherkin scenario should cover one individual behavior so that the target behavior is easier to understand, easier to communicate, and easier to investigate whenever the scenario fails during testing. When scenarios break the order of steps, such as Given → Then → Given → Then in the example scenario, it shows that either the scenario covers multiple behaviors or that the author did not bring a behavior-driven understanding to the scenario.

The rules of good behavior don’t disappear when the type of target behavior changes. We should still write Gherkin with best practices in mind, even if our scenarios cover user interfaces.

Breaking Down Scenarios

If I were to rewrite the example scenario, I would start by isolating individual behaviors. Let’s look at the first half of the original example:

Given a user opens https://www.example.com using Chrome
And the user clicks on "Upload Files"
And the page reloads
And the user clicks on "Spreadsheet Formats"
Then the buttons "XLS" and "XLSX" show
And the user clicks on "XLSX"
And the user selects "500kb-sheet.xlsx"
Then the upload completes
And the table "Uploaded Files" contains a cell with "500kb-sheet.xlsx" 
And the user clicks on "XLSX"
And the user selects "1mb-sheet.xlsx"
Then the upload fails
And the table "Uploaded Files" does not contain a cell with "1mb-sheet.xlsx"

Here, I see four distinct behaviors covered:

  1. Clicking “Upload Files” reloads the page.
  2. Clicking “Spreadsheet Formats” displays new buttons.
  3. Uploading a spreadsheet file makes the filename appear on the page.
  4. Attempting to upload a spreadsheet file that is 1MB or larger fails.

If I wanted to purely retain the same coverage, then I would rewrite these behavior specs using the following scenarios:

Feature: Example site
 
 
Scenario: Choose to upload files
 
Given the Example site is displayed
When the user clicks the "Upload Files" link
Then the page displays the "Spreadsheet Formats" link
 
 
Scenario: Choose to upload spreadsheets
 
Given the Example site is ready to upload files
When the user clicks the "Spreadsheet Formats" link
Then the page displays the "XLS" and "XLSX" buttons
 
 
Scenario: Upload a spreadsheet file that is smaller than 1MB
 
Given the Example site is ready to upload spreadsheet files
When the user clicks the "XLSX" button
And the user selects "500kb-sheet.xlsx" from the file upload dialog
Then the upload completes
And the table "Uploaded Files" contains a cell with "500kb-sheet.xlsx" 
 
 
Scenario: Upload a spreadsheet file that is larger than or equal to 1MB
 
Given the Example site is ready to upload spreadsheet files
When the user clicks the "XLSX" button
And the user selects "1mb-sheet.xlsx" from the file upload dialog
Then the upload fails
And the table "Uploaded Files" does not contain a cell with "1mb-sheet.xlsx"

Now, each scenario covers each individual behavior. The first scenario starts with the Example site in a “blank” state: “Given the Example site is displayed”. The second scenario inherently depends upon the outcome of the first scenario. Rather than repeat all the steps from the first scenario, I wrote a new starting step to establish the initial state more declaratively: “Given the Example site is ready to upload files”. This step’s definition method may need to rerun the same operations as the first scenario, but it guarantees independence between scenarios. (The step could also optimize the operations, but that should be a topic for another challenge.) Likewise, the third and fourth scenarios have a Given step to establish the state they need: “Given the Example site is ready to upload spreadsheet files.” Both scenarios can share the same Given step because they have the same starting point. All three of these new steps are descriptive more than prescriptive. They declaratively establish an initial state, and they leave the details to the automation code in the step definition methods to determine precisely how that state is established. This technique makes it easy for Gherkin scenarios to be individually clear and independently executable.

I also added my own writing style to these scenarios. First, I wrote concise, declarative titles for each scenario. The titles dictate interaction over mechanics. For example, the first scenario’s title uses the word “choose” rather than “click” because, from the user’s perspective, they are “choosing” an action to take. The user will just happen to mechanically “click” a link in the process of making their choice. The titles also provide a level of example. Note that the third and fourth scenarios spell out the target file sizes. For brevity, I typically write scenario titles using active voice: “Choose this,” “Upload that,” or “Do something.” I try to avoid including verification language in titles unless it is necessary to distinguish behaviors.

Another stylistic element of mine was to remove explicit details about the environment. Instead of hard coding the website URL, I gave the site a proper name: “Example site.” I also removed the mention of Chrome as the browser. These details are environment-specific, and they should not be specified in Gherkin. In theory, this site could have multiple instances (like an alpha or a beta), and it should probably run in any major browser (like Firefox and Edge). Environmental characteristics should be specified as inputs to the automation code instead.I also refined some of the language used in the When and Then steps. When I must write steps for mechanical actions like clicks, I like to specify element types for target elements. For example, “When the user clicks the “Upload Files” link” specifies a link by a parameterized name. Saying the element is a link helps provides context to the reader about the user interface. I wrote other steps that specify a button, too. These steps also specified the element name as a parameter so that the step definition method could possibly perform the same interaction for different elements. Keep in mind, however, that these linguistic changes are neither “required” nor “perfect.” They make sense in the immediate context of this feature. While automating step definitions or writing more scenarios, I may revisit the verbiage and do some refactoring.

Determining Value for Each Behavior

The four new scenarios I wrote each covers an independent, individual behavior of the fictitious Example site’s user interface. They are thorough in their level of coverage for these small behaviors. However, not all behaviors may be equally important to cover. Some behaviors are simply more important than others, and thus some tests are more valuable than others. I won’t go into deep detail about how to measure risk and determine value for different tests in this article, but I will offer some suggestions regarding these example scenarios.

First and foremost, you as the tester must determine what is worth testing. These scenarios aptly specify behavior, and they will likely be very useful for collaborating with the Three Amigos, but not every scenario needs to be automated for testing. You as the tester must decide. You may decide that all four of these example scenarios are valuable and should be added to the automated test suite. That’s a fine decision. However, you may instead decide that certain user interface mechanics are not worth explicitly testing. That’s also a fine decision.

In my opinion, the first two scenarios could be candidates for the chopping block:

  1. Choose to upload files
  2. Choose to upload spreadsheets

Even though these are existing behaviors in the Example site, they are tiny. The tests simply verify that a user clicks makes certain links or buttons appear. It would be nice to verify them, but test execution time is finite, and user interface tests are notoriously slow compared to other tests. Consider the Rule of 1’s: typically, by orders of magnitude, a unit test takes about 1 millisecond, a service API test takes about 1 second, and a web UI test takes about 1 minute. Furthermore, these behaviors are implicitly exercised by the other scenarios, even if they don’t have explicit assertions.

One way to condense the scenarios could be like this:

Feature: Example site
 
 
Background:
 
Given the Example site is displayed
When the user clicks the "Upload Files" link
And the user clicks the "Spreadsheet Formats" link
And the user clicks the "XLSX" button
 
 
Scenario: Upload a spreadsheet file that is smaller than 1MB
 
When the user selects "500kb-sheet.xlsx" from the file upload dialog
Then the upload completes
And the table "Uploaded Files" contains a cell with "500kb-sheet.xlsx" 
 
 
Scenario: Upload a spreadsheet file that is larger than or equal to 1MB
 
When the user selects "1mb-sheet.xlsx" from the file upload dialog
Then the upload fails
And the table "Uploaded Files" does not contain a cell with "1mb-sheet.xlsx" 

This new feature file eliminates the first two scenarios and uses a Background section to cover the setup steps. It also eliminates the need for special Given steps in each scenario to set unique starting points. Implicitly, if the “Upload Files” or “Spreadsheet Formats” links fail to display the expected elements, then those steps would fail.

Again, this modification is not necessarily the “best” way or the “right” way to cover the desired behaviors, but it is a reasonably good way to do so. However, I would assert that both the 4-scenario feature file and the 2-scenario feature file are much better approaches than the original example scenario.

More Gherkin

What I showed in my answer to this Gherkin challenge is how I would handle UI-centric behaviors. I try to keep my Gherkin scenarios concise and focused on individual, independent behaviors. Try using these style techniques to rewrite the second half of Gojko’s original scenario. Feel free to drop your Gherkin in the comments below. I look forward to seeing how y’all write #GivenWhenThenWithStyle!

Extending Grace in Small Ways

Back in 2011, I was a recent college grad working at IBM as a “performance engineer” for z/OS mainframe software. Now, I didn’t know anything about mainframes, but I was thankful to have a job on the heels of the Great Recession.

At the time, IBM had recently released the Jazz platform with Rational Team Concert (RTC), a collaborative project management tool geared towards Agile software development. Teams company-wide started adopting RTC whether they wanted it or not. My team was no different: we created a team project in RTC and started writing work items in it. In my opinion, RTC was decent. It was very customizable, and its aesthetics and user experience were better than other tools at the time.

One day, I made a typo while trying to assign a work item to myself. When typing a name into the “owner” field, RTC would show a list of names from which to choose. For whatever reason, the list included all IBM employees, not just members from my team. IBM had nearly 400,000 employees worldwide at the time. I accidentally selected someone else with a similar name to mine. Blissfully unaware of my mistake, I proceeded to save the work item and start doing the actual work for it.

About a day later, I received a nastygram from another IBMer named Andrea Knight, demanding to know why I assigned her this work item in RTC. I had never met this person before, and she certainly wasn’t on my team. (To be honest, I don’t remember exactly what her name was, but for the sake of the story, we can call her Andrea.) At first, I felt perplexed. Then, once I read her message, I quickly realized that I must have accidentally listed her as the owner of the work item. I immediately corrected the mistake and humbly replied with an apology for my typo. No big deal, right?

Well, Andrea replied to my brief apology later that day to inform me that she was NOT responsible for that work item because she had NEVER seen it before and that she would NOT do any work for it.

Really GIFs | Tenor
What?

I was quite taken back by her response.

I let it go, but I couldn’t help but wonder why she would answer that way. Perhaps she was having a bad day? Perhaps her manager scrutinized all work items bearing her name? Perhaps the culture in her part of the company was toxic? Was my mistake that bad?

Even though this incident was small, it taught me one important lesson early in my career: a little bit of grace goes a long way. Poor reactions create awkward situations, hurt feelings, and wasted time. If we make a mistake, we should fix it and apologize. If someone else makes a mistake, we should strive to be gracious instead of unpleasant. I try to practice this myself, though, sometimes, I fail.

Nobody is perfect. That’s why we all need grace.

Improving Teamwork with SpecFlow+ LivingDoc

SpecFlow is an excellent Behavior-Driven Development test framework for .NET. Recently, SpecFlow released a new reporting tool called SpecFlow+ LivingDoc, which generates living documentation for features. It combines all scenarios from all SpecFlow feature files into one central HTML report. The report looks crisp and professional. It is filterable and can optionally show test results. Teams can generate updated reports as part of their Continuous Integration pipelines. The best part is that SpecFlow+ LivingDoc, along with all other features, is completely free to use – all you need to do is register for a free SpecFlow account. There is no reason for any SpecFlow project to not also use LivingDoc.

SpecFlow provides rich documentation on all of SpecFlow+ LivingDoc’s benefits, features, and configurations. In this article, I won’t simply repeat what the official docs already state. Instead, I’m going to share how my team and I at PrecisionLender, a Q2 Company, adopted SpecFlow+ LivingDoc into our test automation solution. I’ll start by giving a brief overview of how we test the PrecisionLender web app. Then, I’ll share why we wanted to make LivingDoc part of our quality workflow. Next, I’ll walk through how we added the new report to our testing pipelines. Finally, as an advanced technique, I’ll show how we modified some of the LivingDoc data files to customize our reports. My goal for this article is to demonstrate the value SpecFlow+ LivingDoc adds to BDD collaboration and automation practices.

PrecisionLender’s Test Automation

PrecisionLender is a web application that empowers commercial bankers with in-the-moment insights that help them structure and price commercial deals. Andi®, PrecisionLender’s intelligent virtual analyst, delivers these hyper-focused recommendations in real time allowing relationship managers to make data-driven decisions while pricing their commercial deals.

The PrecisionLender Opportunity Screen
(Picture taken from the PrecisionLender Support Center)

The PrecisionLender app is quite complex. It has several rich features to help bankers price any possible nuance for loan opportunities. Some banks also have unique configurations and additional features that make testing challenging.

On top of thorough unit testing, we run suites of end-to-end tests against the PrecisionLender web app. Our test automation solution is named “Boa,” and it is written in C# using SpecFlow for test cases and Boa Constrictor for Web UI and REST API interactions. We use BDD practices like Three Amigos, Example Mapping, and Good Gherkin to develop behaviors and cover them with automated tests. As of January 2021, Boa has over 1400 unique tests that target multiple test bank configurations. We run Boa tests continuously (for every code change), nightly (across all test banks), and “release-ly” (every two weeks before production deployments) at a rate of ~15K test iterations per week. Each test takes roughly half a minute to complete, and we run tests in parallel with up to 32 threads using Selenium Grid.

Introducing SpecFlow+ LivingDoc

SpecFlow+ LivingDoc is living documentation for features. SpecFlow started developing the tool a few years ago, but in recent months under Tricentis, they have significantly ramped up its development with the standalone generator and numerous feature enhancements. To learn about LivingDoc, watch this short introduction video:

When I saw the new SpecFlow+ LivingDoc reports, I couldn’t wait to try them myself. I love SpecFlow, and I’ve used it daily for the past few years. I knew it would bring value to my team at PrecisionLender.

Why Adopt SpecFlow+ LivingDoc?

My team and I wanted to bring SpecFlow+ LivingDoc into our testing workflow for a few reasons. First and foremost, we wanted to share our features with every team member, whether they were in business, development, or testing roles. I originally chose SpecFlow to be the core test framework for our Boa tests because I wanted to write all tests in plain-language Gherkin. That way, product owners and managers could read and understand our tests. We could foster better discussions about product behaviors, test coverage, and story planning. However, even though tests could be understood by anyone, we didn’t have an effective way to share them. Feature files for tests must be stored together with automation code in a repository. Folks must use Visual Studio or a version control tool like Git to view them. That’s fine for developers, but it’s inaccessible for folks who don’t code. SpecFlow+ LivingDoc breaks down that barrier. It combines all scenarios from all feature files into one consolidated HTML report. Folks could use a search bar to find the tests they need instead of plunging through directories of feature files. The report could be generated by Continuous Integration pipelines, published to a shared dashboard, or emailed directly to stakeholders. Pipelines could also update LivingDoc reports any time features change. SpecFlow+ LivingDoc would enable us to actually share our features instead of merely saying that we could.

SpecFlow+ LivingDoc Living Documentation for PrecisionLender Features

Second, we liked the concise test reporting that SpecFlow+ LivingDoc offered. The SpecFlow+ Runner report, which our team already used, provides comprehensive information about test execution: full log messages, duration times, and a complete breakdown of pass-or-fail results by feature and scenario. That information is incredibly helpful when determining why tests fail, but it is too much information when reporting failures to managers. LivingDoc provides just the right amount of information for reporting high-level status: the tests, the results per test, and the pass-or-fail totals. Folks can see test status at a glance. The visuals look nice, too.

SpecFlow+ LivingDoc Analytics for PL App Boa Tests

Third, we wanted to discover any unused step definitions in our C# automation code. The Boa test solution is a very large automation project. As of January 2020, it has over 1400 unique tests and over 1100 unique step definitions, and those numbers will increase as we continue to add new tests. Maintaining any project of this size is a challenge. Sometimes, when making changes to scenarios, old step definitions may no longer be needed, but testers may not think to remove them from the code. These unused step definitions then become “dead code” that bloats the repository. It’s easy to lose track of them. SpecFlow+ LivingDoc offers a special option to report unused step definitions on the Analytics tab. That way, the report can catch dead steps whenever they appear. When I generated the LivingDoc report for the Boa tests, I discovered over a hundred unused steps!

SpecFlow+ LivingDoc Unused Step Definitions

Fourth and finally, my team and I needed a test report that we could share with customers. At PrecisionLender, our customers are banks – and banks are very averse to risk. Some of our customers ask for our test reports so they can take confidence in the quality of our web app. When sharing any information with customers, we need to be careful about what we do share and what we don’t share. Internally, our Boa tests target multiple different system configurations, and we limit the test results we share with customers to the tests for the features they use. For example, if a bank doesn’t factor deposits into their pricing calculations, then that bank’s test report should not include any tests for deposits. The reports should also be high-level instead of granular: we want to share the tests, their scenarios, and their pass-or-fail results, but nothing more. SpecFlow+ LivingDoc fits this need perfectly. It provides Gherkin scenarios with their steps in a filterable tree, and it visually shows results for each test as well as in total. With just a little bit of data modification (as shown later in this article), the report can include exactly the intended scenarios. Our team could use LivingDoc instead of generating our own custom report for customers. LivingDoc would look better than any report we would try to make, too!

Setting Up SpecFlow+ LivingDoc

At PrecisionLender, we currently use JetBrains TeamCity to schedule and launch Boa tests. Some tests launch immediately after app deployments, while others are triggered based on the time of day. When a test pipeline is launched, it follows these steps:

  1. Check out the code repository.
  2. Build the Boa test automation solution.
  3. For each applicable bank configuration, run appropriate Boa tests.

We wanted to add SpecFlow+ LivingDoc in two places: after the build completes and after tests run for each configuration. The LivingDoc generated for the build step would not include test results. It would show all scenarios in all features, and it would also include the unused step definitions. This report would be useful for showing folks our tests and our coverage. The LivingDoc generated for each test run, however, would include test results. Since we run tests against multiple configurations, each run would need its own LivingDoc report. Not all tests run on each configuration, too. Generating LivingDoc reports at each pipeline step serve different needs.

Adding SpecFlow+ LivingDoc to our testing pipelines required only a few things to set up. The first step was to add the SpecFlow.Plus.LivingDocPlugin NuGet package to the .NET test project. Adding this NuGet package makes SpecFlow automatically save test results to a file named TestExecution.json every time tests run. The docs say you can customize this output path using specflow.json, too.

Required SpecFlow NuGet packages, including SpecFlow.Plus.LivingDoc
An example snippet of TestExecution.json

The next step was to install the LivingDoc CLI tool on our TeamCity agents. The CLI tool is a dotnet command line tool, so you need the .NET Core SDK 3.1 or higher. Also, note that you cannot install this package as a NuGet dependency for your .NET test project. (I tried to do that in the hopes of simplifying my build configuration, but NuGet blocks it.) You must install it to your machine’s command line. The installation command looks like this:

dotnet tool install --global SpecFlow.Plus.LivingDoc.CLI

After installing the LivingDoc CLI tool, the final step was to invoke it after each build and each test run. There are three sources from which to generate LivingDoc reports:

  1. Using a folder of feature files
  2. Using a SpecFlow test assembly (.dll)
  3. Using a feature data JSON file previously generated by the LivingDoc CLI tool

For generating LivingDoc after the build, I used this command in PowerShell to include unused steps but exclude test results:

livingdoc test-assembly "$TestAssemblyPath" --binding-assemblies "$TestAssemblyPath" --output-type HTML --output "$LivingDocDir\PLAppLivingDoc.html"

Then, for generating LivingDoc after test runs, I used this command in PowerShell that included TestExecution.json:

livingdoc test-assembly "$TestAssemblyPath" --test-execution-json "$TestExecutionPath" --output-type HTML --output "$HtmlReportPath" --title "PL App Boa Tests"

All the “$” variables are paths configured in our TeamCity projects. I chose to generate reports using the test assembly because I discovered that results wouldn’t appear in the report if I generated them from the feature folder.

Here’s what SpecFlow+ LivingDoc looks like when published as a TeamCity report:

SpecFlow+ LivingDoc report in TeamCity for the build (without test results)

Our team can view reports from TeamCity, or they can download them to view them locally.

Modifying SpecFlow+ LivingDoc Data

As I mentioned previously in this article, my team and I wanted to share SpecFlow+ LivingDoc reports with some of our customers. We just needed to tweak the contents of the report in two ways. First, we needed to remove scenarios that were inapplicable (meaning not executed) for the bank. Second, we needed to remove certain tags that we use internally at PrecisionLender. Scrubbing this data from the reports would give our customers what they need without including information that they shouldn’t see.

Thankfully, SpecFlow+ LivingDoc has a “backdoor” in its design that makes this kind of data modification easy. When generating a LivingDoc report, you can set the --output-type parameter to be “JSON” instead of “HTML” to generate a feature data JSON file. The feature data file contains all the data for the LivingDoc report in JSON format, including scenarios and tags. You can modify the data in this JSON file and then use it to generate an HTML LivingDoc report. Modifying JSON data is much simpler and cleaner than painfully splicing HTML text.

An example snippet of a feature data JSON file

I wrote two PowerShell scripts to modify feature data. Both are available publicly in GitHub at AndyLPK247/SpecFlowPlusLivingDocScripts. You can copy them from the repository to use them for your project, and you can even enhance them with your own changes. Note that the feature data JSON files they use must be generated from test assemblies, not from feature data folders.

The first script is RemoveSkippedScenarios.ps1. It takes in both a feature data JSON file and a test execution JSON file, and it removes all scenarios from the feature data that did not have results in the test execution data. It uses recursive functions to traverse the feature data JSON “tree” of folders, features, and scenarios. Removing unexecuted scenarios means that the LivingDoc report will only include scenarios with test results – none of the scenarios in it should be “skipped.” For my team, this means a LivingDoc report for a particular bank configuration will not include a bunch of skipped tests for other banks. Even though we currently have over 1400 unique tests, any given bank configuration may run only 1000 of those tests. The extra 400 skipped tests would be noise at best and a data privacy violation at worst.

The second script is RemoveTags.ps1. It takes in a list of tags and a feature data JSON file, and it removes all appearances of those tags from every feature, scenario, and example table. Like the script for removing skipped scenarios, it uses recursive functions to traverse the feature data JSON “tree.” The tags must be given as literal names, but the script could easily be adjusted to handle wildcard patterns or regular expressions.

With these new scripts, our test pipelines now look like this:

  1. Check out the code repository.
  2. Build the Boa test automation solution.
  3. Generate the SpecFlow+ LivingDoc report with unused steps but without test results.
  4. For each applicable bank configuration:
    1. Run appropriate Boa tests and get the test execution JSON file.
    2. Generate the feature data JSON file.
    3. Remove unexecuted scenarios from the feature data.
    4. Remove PrecisionLender-specific tags from the feature data.
    5. Generate the SpecFlow+ LivingDoc report using the modified feature data and the test results.

Below is an example of what the modified LivingDoc report looks like when we run our 12 smoke tests:

SpecFlow+ LivingDoc report using modified feature data after running only 12 smoke tests

(Note: At the time of writing this article, the most recent version of SpecFlow+ LivingDoc now includes a filter for test results in addition to its other filters. Using the test result filter, you can remove unexecuted scenarios from view. This feature is very helpful and could be used for our internal testing, but it would not meet our needs of removing sensitive data from reports for our customers.)

Conclusion

Ever since acquiring SpecFlow from TechTalk in January 2020, Tricentis has done great things to improve SpecFlow’s features and strengthen its community. SpecFlow+ LivingDoc is one of the many fruits of that effort. My team and I at PrecisionLender love these slick new reports, and we are already getting significant value out of them.

If you like SpecFlow+ LivingDoc, then I encourage you to check out some of SpecFlow’s other products. Everything SpecFlow offers is now free to use forever – you just need to register a free SpecFlow account. SpecFlow+ Runner is by far the best way to run SpecFlow tests (and, believe me, I’ve used the other runners for NUnit, xUnit.net, and MsTest). SpecMap is great for mapping and planning stories with Azure Boards. SpecFlow’s Online Gherkin Editor is also one of the best and simplest ways to write Gherkin without needing a full IDE.

Finally, if you use SpecFlow for test automation, give Boa Constrictor a try. Boa Constrictor is a .NET implementation of the Screenplay Pattern that my team and I developed at PrecisionLender. It helps you make better interactions for better automation, and it’s a significant step up from the Page Object Model. It’s now an open source project – all you need to do is install the Boa.Constrictor NuGet package! If you’re interested, be sure to check out the SpecFlow livestream in which Andi Willich and I teamed up to convert an existing SpecFlow project from page objects and drivers to Boa Constrictor’s Screenplay calls. SpecFlow and Boa Constrictor work together beautifully.

Using Domain-Specific Languages for Security Testing

I love programming languages. They have fascinated me ever since I first learned to program my TI-83 Plus calculator in ninth grade, many years ago. When I studied computer science in college, I learned how parsers, interpreters, and compilers work. During my internships at IBM, I worked on a language named Enterprise Generation Language as both a tester and a developer. At NetApp, I even developed my own language named DS for test automation. Languages are so much fun to learn, build, and extend.

Today, even though I do not actively work on compilers, I still do some pretty interesting things with languages and testing. I strongly advocate for Behavior-Driven Development and its domain-specific language (DSL) Gherkin. In fact, as I wrote in my article Behavior-Driven Blasphemy, I support using Gherkin-based BDD test frameworks for test automation even if a team is not also doing BDD’s collaborative activities. Why? Gherkin is the world’s first major off-the-shelf DSL for test automation, and it doesn’t require the average tester to know the complexities of compiler theory. DSLs like Gherkin can make tests easier to read, faster to write, and more reliable to run. They provide a healthy separation of concerns between test cases and test code. After working on successful large-scale test automation projects with C# and SpecFlow, I don’t think I could go back to traditional test frameworks.

I’m not the only one who thinks this way. Here’s a tweet from Dinis Cruz, CTO and CISO at Glasswall, after he read one of my articles:

Dinis then tweeted at me to invite me to speak about using DSLs for testing at the Open Security Summit in 2021:

Now, I’m not a “security guy” at all, but I do know a thing or two about DSLs and testing. So, I gladly accepted the invitation to speak! I delivered my talk, “Using DSLs for Security Testing” virtually on Thursday, January 14, 2021 at 10am US Eastern. I also uploaded my slides to GitHub at AndyLPK247/using-dsls-for-security-testing. Check out the YouTube recording here:

This talk was not meant to be a technical demo or tutorial. Instead, it was meant to be a “think big” proposal. The main question I raised was, “How can we use DSLs for security testing?” I used my own story to illustrate the value languages deliver, particularly for testing. My call to action breaks that question down into three parts:

  1. Can DSLs make security testing easier to do and thereby more widely practiced?
  2. Is Gherkin good enough for security testing, or do we need to make a DSL specific to security?
  3. Would it be possible to write a set of “standard” or “universal” security tests using a DSL that anyone could either run directly or use as a template?

My goal for this talk was to spark a conversation about DSLs and security testing. Immediately after my talk, Luis Saiz shared two projects he’s working on regarding DSLs and security: SUSTO and Mist. Dinis also invited me back for a session at the Open Source Summit Mini Summit in February to have a follow-up roundtable discussion for my talk. I can’t wait to explore this idea further. It’s an exciting new space for me.

If this topic sparks your interest, be sure to watch my talk recording, and then join us live in February 2021 for the next Open Source Summit event. Virtual sessions are free to join. Many thanks again to Dinis and the whole team behind Open Source Summit for inviting me to speak and organizing the events.

I’m Writing a Software Testing Book!

That’s right! You read the title. I’m writing a book about software testing!

One of the most common questions people ask me is, “What books can you recommend on software testing and automation?” Unfortunately, I don’t have many that I can recommend. There are plenty of great books, but most of them focus on a particular tool, framework, or process. I haven’t found a modern book that covers software testing as a whole. Trust me, I looked – when I taught my college course on software testing at Wake Tech, the textbook’s copyright date was 2002. Its content felt just as antiquated.

I want to write a book worthy of answering that question. I want to write a treatise on software testing for our current generation of software professionals. My goal is ambitious, but I think I can do it. It will probably take a year to write. I hope to find deep joy in this endeavor.

Manning Publications will be the publisher. They accepted my proposal, and we signed a contract. The working title of the book is The Way to Test Software. The title pays homage to Julia Child’s classic, The Way to Cook. Like Julia Child, I want to teach “master recipes” that can be applied to any testing situations.

I don’t want to share too many details this early in the process, but the tentative table of contents has the following parts:

  1. Orientation
  2. Testing Code
  3. Testing Features
  4. Testing Performance
  5. Running Tests
  6. Development Practices

Python will be the language of demonstration. This should be no surprise to anyone. I chose Python because I love the language. I also think it’s a great language for test automation. Python will be easy for both beginners and experts to learn. Besides, the book is about testing, not programming – Python will be just the linguistic tool for automation.

If you’re as excited about this book as I am, please let me know! I need all the encouragement I can get. This book probably won’t enter print until 2022, given the breadth of its scope. I’ll work to get it done as soon as I can.

Learning Python Test Automation

Do you want to learn how to automate tests in Python? Python is one of the best languages for test automation because it is easy to learn, concise to write, and powerful to scale. These days, there’s a wealth of great content on Python testing. Here’s a brief reference to help you get started.

If you are new to Python, read How Do I Start Learning Python? to find the best way to start learning the language.

If you want to roll up your sleeves, check out Test Automation University. I developed a “trifecta” of Python testing courses for TAU with videos, transcripts, quizzes, and example code. You can take them for FREE!

  1. Introduction to pytest
  2. Selenium WebDriver with Python
  3. Behavior-Driven Python with pytest-bdd

If you wants some brief articles for reference, check out my Python Testing 101 blog series:

  1. Python Testing 101: Introduction
  2. Python Testing 101: unittest
  3. Python Testing 101: doctest
  4. Python Testing 101: pytest
  5. Python Testing 101: behave
  6. Python Testing 101: pytest-bdd
  7. Python BDD Framework Comparison

RealPython also has excellent guides:

I’ve given several talks about Python testing:

If you prefer to read books, here are some great titles:

Here are links to popular Python test tools and frameworks:

Do you have any other great resources? Drop them in the comments below! Happy testing!

SpecFlow’s Online Gherkin Editor

Finding a good Gherkin editor is difficult. Some editors like Visual Studio Code and similar IDEs work great for engineers but aren’t suitable for product owners and non-programmer Amigos who want to contribute. Other editors like Notepad++ and Atom are lighter in weight but still require extensions and a little expertise. Fancy BDD tools like CucumberStudio and Cucumber for Jira provide Gherkin editors together with a bunch of other nifty features, but they require paid licenses.

For years, I’ve wanted a lightweight Gherkin editor that’s easy to use and accessible to all. Now, one finally exists: the Online Gherkin Editor by SpecFlow!

SpecFlow is the most popular BDD test automation framework for .NET. It’s also my favorite BDD framework. Over the past few years, I’ve built two large-scale test automation solutions with SpecFlow.

The Online Gherkin Editor by SpecFlow is just an editor on a web page. When you first load the page, the editor has example scenarios for you to reference. You can type your own Gherkin into the text area, and the editor highlights it for you. The editor provides line numbers and visual scrolling, too. My language is English, but if you happen to speak German, French, Spanish, or Dutch, then you can change the language setting via a dropdown. Once you’re done writing your Gherkin, you can clear it, copy it to the clipboard, or download it as a feature file using icons in the top-right corner. Be warned, though, that this editor won’t save your Gherkin in the cloud.

If you want to give this new editor a try, here’s the link: https://specflow.org/gherkin-editor/

You can also read SpecFlow’s official announcement here: https://specflow.org/blog/introducing-the-specflow-online-gherkin-editor/

Thanks, SpecFlow! Happy “Gherk-ing”!

Boa Constrictor Intro Video with Transcript

The Video

Boa Constrictor is the .NET Screenplay Pattern, and I’m its lead developer. Check out this intro video to learn why we need the Screenplay Pattern and how to use it with Boa Constrictor.

The Transcript

[Camera]

Hello, everyone! My name is Andrew Knight, or “Pandy” for short. I’m the Automation Panda – I build solutions to testing problems. Be sure to read my blog and follow me on Twitter at “AutomationPanda”.

Today, I’m going to introduce you to a new test automation library called Boa Constrictor, the .NET Screenplay Pattern. Boa Constrictor can help you make better interactions for better automation. Its primary use cases are Web UI and REST API interactions, but it can be extended to handle any type of interaction.

My team and I at PrecisionLender originally developed Boa Constrictor as the cornerstone of our .NET end-to-end test automation solution. We found the Screenplay Pattern to be a great way to scale our test development, avoid duplicate code, and stay focused on behaviors. In October 2020, together with help from our parent company Q2, we released Boa Constrictor as an open source project.

In this video, we will cover three things:

  1. First, problems with traditional ways of automating interactions.
  2. Second, why the Screenplay Pattern is a better way.
  3. Third, how to use the Screenplay Pattern with Boa Constrictor in C#.

Since Boa Constrictor is open source, you can check out its repository. I’ll paste the link below: https://github.com/q2ebanking/boa-constrictor. The repository also has a hands-on tutorial you can try. Make sure to have Visual Studio and some .NET skills because the code is written in C#.

My main goal with the Boa Constrictor project is to help improve test automation practices. For so long, our industry has relied on page objects, and I think it’s time we talk about a better way. Boa Constrictor strives to make that easy.

[Slide]

To start, let’s define that big “I” word I kept tossing around:

[Slide]

Interactions.

[Slide]

Simply put, interactions are how users operate software. For this video, I’ll focus on Web UI interactions, like clicking buttons and scraping text.

[Slide]

Interactions are indispensable to testing. The simplest way to define “testing” is interaction plus verification. That’s it! You do something, and you make sure it works.

Think about any functional test case that you have ever written or executed. The test case was a step-by-step procedure, in which each step had interactions and verifications.

[Slide]

Here’s an example of a simple DuckDuckGo search test. DuckDuckGo is a search engine like Google or Yahoo. The steps here are very straightforward.

[Slide]

Opening the search engine requires navigation.

[Slide]

Searching for a phrase requires entering keystrokes and clicking the search button.

[Slide]

Verifying results requires scraping the page title and result links from the new page. 

Interactions are everywhere!

[Slide]

Unfortunately, our industry struggles to handle automated Web UI interactions well. Even though most teams use Selenium WebDriver in their test automation code, every team seems to use it differently. There’s lots of duplicate code and flakiness, too. Let’s take a look at the way many teams evolve their WebDriver-based interactions. I will use C# for code examples, and I will continue to use DuckDuckGo for testing.

[Slide]

When teams first start writing test automation code using Selenium WebDriver, they frequently write raw calls. Anyone familiar with the WebDriver API should recognize these calls.

[Slide]

The WebDriver object is initialized using, say, ChromeDriver for the Chrome browser.

[Slide]

The first step to open the search engine calls “driver dot navigate dot go to URL” with the DuckDuckGo website address.

[Slide]

The second step performs the search by fetching Web elements using “driver dot find element” with locators and then calling methods like “send keys” and “click”.

[Slide]

The third step uses assertions to verify the contents of the page title and the existence of result links.

[Slide]

Finally, at the end of the test, the WebDriver quits the browser for cleanup.

Like I said, these are all common WebDriver calls. Unfortunately, there’s a big problem in this code.

[Slide]

Race conditions. There are three race conditions in this code in which the automation does NOT wait for the page to be ready before making interactions! WebDriver does not automatically wait for elements to load or titles to appear. Waiting is a huge challenge for Web UI automation, and it is one of the main reasons for “flaky” tests.

[Slide]

You could set an implicit wait that will make calls wait until target elements appear, but they don’t work for all cases, such as the title in race condition #2.

[Slide]

Explicit waits provide much more control over waiting timeout and conditions. They use a “WebDriverWait” object with a pre-set timeout value, and they must be placed explicitly throughout the code. Here, they are placed in the three spots where race conditions could happen. Each “wait dot until” call takes in a function that returns true when the condition is satisfied.

[Slide]

These waits are necessary, but they cause new problems. First, they cause duplicate code because Web element locators are used multiple times. Notice how “search form input homepage” is called twice.

[Slide]

Second, raw calls with explicit waits makes code less intuitive. If I remove the comments from each paragraph of code, what’s left is a wall of text. It is difficult to understand what this code does as a glance.

[Slide]

To remedy these problems, most teams use the Page Object Pattern. In the Page Object Pattern, each page is modeled as a class with locator variables and interaction methods. So, a “search page” class could look like this.

[Slide]

At the top, there could be a constant for the page URL and variables for the search input and search button locators. Notice how each has an intuitive name.

[Slide]

Next, there could be a variable to hold the WebDriver reference. This reference would come via dependency injection through the constructor.

[Slide]

The first method would be a “load” method that navigates the browser to the page’s URL.

[Slide]

And, the second method would be a “search” method that waits for the elements to appear, enters the phrase into the input field, and clicks the search button.

This page object class has a decent structure and a mild separation of concerns. Locators and interactions have meaningful names. Page objects require a few more lines of code that raw calls at first, but their parts can easily be reused.

[Slide]

The original test steps can be rewritten using this new SearchPage class. Notice how much cleaner this new code looks.

[Slide]

The other steps can be rewritten using page objects, too.

[Slide]

Unfortunately, page objects themselves suffer problems with duplication in their interaction methods.

[Slide]

Suppose a page object needs a method to click an element. We already know the logic: wait for the element to exist, and then click it.

But what about clicking another element? This method is essentially hard coded for one button.

[Slide]

A second “click” method is needed to click the other button.

[Slide]

Unfortunately, the code for both methods is the same. The code will be the same for any other click method, too. This is copy pasta, and it happens all the time in page objects. I’ve seen page objects grow to be thousands of lines long due to duplicative methods like this.

At this point, some teams will say, “Aha! More duplicate code? We can solve this problem with more Object-Oriented Programming!”

[Slide]

And they’ll create the infamous “base page”, a parent class for all other page object classes.

[Slide]

The base page will have variables for the WebDriver and the wait object.

[Slide]

It will also provide common interaction methods, such as this click method that can click on any element. Abstraction for the win!

[Slide]

Child pages will inherit everything from the base page. Child page interaction methods frequently just call base page methods.

I’ve seen many teams stop here and say, “This is good enough.” Unfortunately, this really isn’t very good at all!

[Slide]

The base page helps mitigate code duplication, but it doesn’t solve its root cause. Page objects inherently combine two separate concerns: page structure and interactions. Interactions are often generic enough to be used on any Web element. Coupling interaction code with specific locators or pages forces testers to add new page object methods for every type of interaction needed for an element. Every element could potentially need a click, a text, a “displayed”, or any other type of WebDriver interaction. That’s a lot of extra code that shouldn’t be necessary. The base page also becomes very top-heavy as testers add more and more code to share.

[Slide]

Most frustratingly, the page object code I showed here is merely one type of implementation. What do your page objects look like? I’d bet dollars to doughnuts that they look different than mine. Page objects are completely free form. Every team implements them differently. There’s no official version of the Page Object Pattern. There’s no conformity in its design. Even worse, within its design, there is almost no way for the pattern to enforce good practices. That’s why people argue whether page object locators should be public or private. Page objects would be better described as a “convention” than as a true design pattern.

[Slide]

There must be a better way to handle interactions. Thankfully, there is.

[Slide]

Let’s take a closer look at how interactions happen.

[Slide]

First, there is someone who initiates the interactions. Usually, this is a user. They are the ones making the clicks and taking the scrapes. Let’s call them the “Actor”.

[Slide]

Second, there is the thing under test. For our examples in this video, that’s a Web app. It has pages with elements. Web page structure is modeled using locators to access page elements from the DOM. Keep in mind, the thing under test could also be anything else, like a mobile app, a microservice, or even a command line.

[Slide]

Third, there are the interactions themselves. For Web apps, they could be simple clicks and keystrokes, or they could be more complex interactions like logging into the app or searching for a phrase. Each interaction will do the same type of operation on whatever target page or element it is given.

[Slide]

Finally, there are objects that enable Actors to perform certain types of Interactions. For example, browser interactions need a tool like Selenium WebDriver to make clicks and scrapes. Let’s call these things “Abilities”.

Actors, Abilities, and Interactions are each different types of concerns. We could summarize their relationship in one line.

[Slide]

Actors use Abilities to perform Interactions.

Actors use Abilities to perform Interactions.

[Slide]

This is the heart of the Screenplay Pattern. In the Page Object Convention, page objects become messy because concerns are all combined. The Screenplay Pattern separates concerns for maximal reusability and scalability.

[Slide]

So, let’s learn how to Screenplay, using Boa Constrictor.

[Slide]

“Boa Constrictor” is an open source C# implementation of the Screenplay Pattern my team and I developed at PrecisionLender. Like I said before, it is the cornerstone of PrecisionLender’s end-to-end test automation solution. It can be used with any .NET test framework, like SpecFlow or NUnit. The GitHub repository name is q2ebanking/boa-constrictor, and the NuGet package name is Boa.Constrictor.

[Slide]

Let’s rewrite that DuckDuckGo search test from before using Boa Constrictor. As you watch this video, I recommend just reading along with the code as it appears on screen to get the concepts. Trying to code along in real time might be challenging. After this video, you can take the official Boa Constrictor tutorial to get hands-on with the code.

[Slide]

To use Boa Constrictor, you will need to install the Boa Constrictor and Selenium WebDriver NuGet packages. My example code will also use Fluent Assertions and ChromeDriver.

[Slide]

The Actor is the entity that initiates Interactions. All Screenplay calls start with an Actor. Most test cases need only one Actor.

The Actor class optionally takes two arguments. The first argument is a name, which can help describe who the actor is. The name will appear in logged messages. The second argument is a logger, which will send log messages from Screenplay calls to a target destination. Loggers must implement Boa Constrictor’s ILogger interface. ConsoleLogger is a class that will log messages to the system console. You can define your own custom loggers by implementing ILogger.

[Slide]

Abilities enable Actors to initiate Interactions. For example, an Actor needs a Selenium WebDriver instance to click elements on a Web page.

Read this new line in plain English: “The actor can browse the Web with a new ChromeDriver.” Boa Constrictor’s fluent-like syntax makes its call chains very readable. “actor dot Can” adds an Ability to an Actor.

[Slide]

“BrowseTheWeb” is the Ability that enables Actors to perform Web UI Interactions. “BrowseTheWeb dot With” provides the WebDriver object that the Actor will use, which, in this case, is a new ChromeDriver object. Boa Constrictor supports all browser types.

All Abilities must implement the IAbility interface. Actors can be given any number of Abilities. “BrowseTheWeb” simply holds a reference to the WebDriver object. Web UI Interactions will retrieve this WebDriver object from the Actor.

[Slide]

Before the Actor can call any WebDriver-based Interactions, the Web pages under test need models. These models should be static classes that include locators for elements on the page and possibly page URLs. Page classes should only model structure – they should not include any interaction logic.

The Screenplay Pattern separates the concerns of page structure from interactions. That way, interactions can target any element, maximizing code reusability. Interactions like clicks and scrapes work the same regardless of the target elements.

The SearchPage class has two members. The first member is a URL string named Url. The second member is a locator for the search input element named SearchInput.

A locator has two parts. First, it has a plain-language Description that will be used for logging. Second, it has a Query that is used to find the element on the page. Boa Constrictor uses Selenium WebDriver’s By queries. For convenience, locators can be constructed using the statically imported L method.

[Slide]

The Screenplay Pattern has two types of Interactions. The first type of Interaction is called a Task. A Task performs actions without returning a value. Examples of Tasks include clicking an element, refreshing the browser, and loading a page. These interactions all “do” something rather than “get” something.

Boa Constrictor provides a Task named Navigate for loading a Web page using a target URL. Read this line in plain English: “The actor attempts to navigate to the URL for the search page.” Again, Boa Constrictor’s fluent-like syntax is very readable. Clearly, this line will load the DuckDuckGo search page. 

[Slide]

“Actor dot attempts to” calls a Task. All Tasks must implement the ITask interface. When the Actor calls “AttemptsTo” on a task, it calls the task’s “PerformAs” method.

[Slide]

“Navigate” is the name of the task, and “dot to URL” provides the target URL.

[Slide]

The Navigate Task’s “PerformAs” method fetches the WebDriver object from the Actor’s Ability and uses it to load the given URL.

[Slide]

“Search page dot URL” comes from the SearchPage class we previously wrote. Putting the URL in the page class makes it universally available.

[Slide]

The second type of Interaction is called a Question. A Question returns an answer after performing actions. Examples of Questions include getting an element’s text, location, and appearance. Each of these interactions return some sort of value. 

Boa Constrictor provides a Question named ValueAttribute that gets the “value” of the text currently inside an input field. Read this line in plain English: “The actor asking for the value attribute of the search page’s search input element should be empty.”

[Slide]

“Actor dot asking for” calls a Question. All Questions must implement the IQuestion interface. When the Actor calls “AskingFor” or the equivalent “AsksFor” method, it calls the question’s “RequestAs” method.

[Slide]

“ValueAttribute” is the name of the Question, and “dot Of” provides the target Web element’s locator. 

[Slide]

The ValueAttribute’s “RequestAs” method fetches the WebDriver object, waits for the target element to exist on the page, and scrapes and returns its value attribute.

[Slide]

“Search page dot search input” is the locator for the search input field. It comes from the SearchPage class.

[Slide]

Finally, once the value is obtained, the test must make an assertion on it. “Should be empty” is a Fluent Assertion that verifies that the search input field is empty when the page is first loaded.

[Slide]

The test case’s next step is to enter a search phrase. Doing this requires two interactions: typing the phrase into the search input and clicking the search button. However, since searching is such a common operation, we can create a custom interaction for search by composing the lower-level interactions together.

[Slide]

The “SearchDuckDuckGo” task takes in a search phrase.

[Slide]

In its “PerformAs” method, it calls two other interactions: “SendKeys” and “Click”.

[Slide]

Using one task to combine these lower-level interactions makes the test code more readable and understandable. It also improves automation reusability. Read this line in plain English now: “The actor attempts to search DuckDuckGo for ‘panda’.” That’s concise and intuitive!

[Slide]

The last test case step should verify that result links appear after entering a search phrase. Unfortunately, this step has a race condition: the result page takes a few seconds to display result links. Automation must wait for those links to appear. Checking too early will make the test case fail.

Boa Constrictor makes waiting easy. Read this line in plain English: “The actor attempts to wait until the appearance of result page result links is equal to true.” In simpler terms, “Wait until the result links appear.”

[Slide]

“Wait” is a special Task. It will repeatedly call a Question until the answer meets a given condition.

[Slide]

For this step, the Question is the appearance of result links on the result page. Before links are loaded, this Question will return “false”. Once links appear, it will return “true”.

[Slide]

The Condition for waiting is for the answer value to become “true”. Boa Constrictor provides several conditions out of the box, such as equality, mathematical comparisons, and string matching. You can also implement custom conditions by implementing the “ICondition” interface.

[Slide]

Waiting is smart – it will repeatedly ask the question until the answer is met, and then it will move on. This makes waiting much more efficient than hard sleeps. If the answer does not meet the condition within the timeout, then the wait will raise an exception. The timeout defaults to 30 seconds, but it can be overridden.

Many of Boa Constrictor’s WebDriver-based interactions already handle waiting. Anything that uses a target element, such as “Click”, “SendKeys”, or “Text” will wait for the element to exist before attempting the operation. We saw this in some of the previous example code. However, there are times where explicit waits are needed. Interactions that query appearance or existence do not automatically wait.

[Slide]

The final step is to quit the browser. Boa Constrictor’s “QuitWebDriver” task does this. If you don’t quit the browser, then it will remain open and turn into a zombie. Always quit the browser. Furthermore, in whatever test framework you use, put the step to quit the browser in a cleanup or teardown routine so that it is called even when the test fails.

[Slide]

And there we have our completed test using Boa Constrictor’s Screenplay Pattern. All the separated concerns come together beautifully to handle interactions in a much better way.

[Slide]

As we said before, the Screenplay Pattern can be summed up in one line:

[Slide]

Actors [Slide] use Abilities [Slide] to perform Interactions.

It’s that simple. Actors use Abilities to perform Interactions.

[Slide]

For those who like Object-Oriented Programming, the Screenplay Pattern is, in a sense, a SOLID refactoring of the Page Object Convention. SOLID refers to five design principles for maintainability and extensibility. I won’t go into detail about each principle here because the information is a bit dense, but if you’re interested, then pause the video, snap a quick screenshot, and check out each of these principles later. Wikipedia is a good source. You’ll find that the Screenplay Pattern follows each one nicely.

[Slide]

So, why should you use the Screenplay Pattern over Page Object Convention or raw WebDriver calls? There are a few key reasons.

[Slide]

First, the Screenplay Pattern, and specifically the Boa Constrictor project, provide rich, reusable, reliable interactions out of the box. Boa Constrictor already has Tasks and Questions for every type of WebDriver-based interaction. Each one is battle-hardened and safe.

[Slide]

Second, Screenplay interactions are composable. Like we saw with searching for a phrase, you can easily combine interactions. This makes code easier to use and reuse, and it avoids lots of duplication.

[Slide]

Third, the Screenplay Pattern makes waiting easy using existing questions and conditions. Waiting is one of the toughest parts of black box automation.

[Slide]

Fourth, Screenplay calls are readable and understandable. They use a fluent-like syntax that reads more like prose than code.

[Slide]

Finally, the Screenplay Pattern, at its core, is a design pattern for any type of interaction. In this video, I showed how to use it for Web UI interactions, but the Screenplay Pattern could also be used for mobile, REST API, and other platforms. You can make your own interactions, too!

[Slide]

Overall, the Screenplay Pattern [Slide] provides better interactions [Slide] for better automation.

That’s the point. It’s not just another Selenium WebDriver wrapper. It’s not just a new spin on page objects. Screenplay is a great way to exercise any feature behaviors under test.

And, as we saw before…

[Slide]

The Screenplay Pattern isn’t that complicated. Actors use Abilities to perform Interactions. That’s it. The programming behind it just has some nifty dependency injection.

[Slide]

If you’d like to start using the Screenplay Pattern for your test automation, there are a few ways to get started.

[Slide]

If you are programming in C#, you can use Boa Constrictor, the library I showed in the examples. You can download Boa Constrictor as a NuGet package. It works with any .NET test framework, like SpecFlow and NUnit. I recommend taking the hands-on tutorial so you can develop a test automation project yourself with Boa Constrictor. Also, since Boa Constrictor is an open source project, I’d love for you to contribute!

[Slide]

If you are programming in Java or JavaScript, you can use Serenity BDD – a mature, complete test automation framework that includes the Screenplay Pattern. Serenity BDD greatly influenced Boa Constrictor, but the two are entirely separate projects. Boa Constrictor is NOT Serenity BDD for .NET. Instead, Boa Constrictor aims to be a simpler, standalone implementation of the Screenplay Pattern.

[Slide]

If none of those options suit you, then you could create your own. The Screenplay Pattern does require a bit of boilerplate code, but it’s worthwhile in the end. You can always reference code from Boa Constrictor and Serenity BDD.

[Slide]

Thank you so much for taking the time to learn more about the Screenplay Pattern and Boa Constrictor. I’d like to give special thanks to everyone at PrecisionLender and Q2 who helped make Boa Constrictor’s open source release happen.

Again, my name is Andrew Knight. I’m the Automation Panda. Be sure to read my blog, follow me on Twitter, and reach out to me if you’d like to join the Boa Constrictor project! Thank you.

How Python Decorators Function

Have you ever seen those “@” tags on top of Python functions and classes? Those are decorators – functions that wrap around other functions. Confusing? At first, but they’re easy with practice. Useful? Very!

The Talk

I delivered a talk on decorators at PyGotham TV 2020, PyCon India 2020, PyTexas 2020, and DevNation Day 2021. Here’s the recording from PyTexas:

And here’s the recording from PyCon India:

Of course, I mentioned testing in this talk, too:

Is it even a Pandy Knight talk if testing is not talked about?
“Is it even a Pandy Knight talk if testing is not talked about?”

The Transcript

[Camera]

Hello, PyTexas 2020! It’s Pandy Knight here. I’m the Automation Panda, and I’m a big Python fan, just like y’all.

Have you ever seen those “@” tags on top of Python functions? Maybe you’ve seen them on top of methods and classes, too. Those are decorators, one of Python’s niftiest language features. Decorators are essentially wrappers – they wrap additional code around existing definitions. When used right, they can clean up your code better than OxiClean! Let’s learn how to use them.

[Slide]

So, here’s a regular old “hello world” function. When we run it, …

[Slide]

…It prints “Hello World!” Nothing fancy here.

[Slide]

Now, let’s take that function…

[Slide]

…And BAM! Add a decorator. Using this “@” sign, we just added a decorator named “tracer” to “hello_world”. So, what is this decorator?

[Slide]

“Tracer” is just another function. But, it’s special because it takes in another function as an argument!

[Slide]

Since “tracer” decorates “hello_world”, the “hello_world” function is passed into “tracer” as an argument. Wow!

So, what’s inside “tracer”?

[Slide]

This decorator has an inner function named “wrapper”. Can you even do that? With Python, yes, you can! The “wrapper” function prints “Entering”, calls the function originally passed into the decorator, and then prints “Exiting”.

[Slide]

When “tracer” decorates “hello_world”, that means “hello_world” will be wrapped by “Entering” and “Exiting” print statements.

[Slides]

Finally, the decorator returns the new “wrapper” function. Any time the decorated function is called, it will effectively be replaced by this new wrapper function. 

[Slides]

So, when we call “hello_world”, the trace statements are now printed, too. Wow! That’s amazing. That’s how decorators work.

[Slide] Decorators [Slide] wrap [Slide] functions [Slide] around [Slide] functions!

[Slide]

Think about them like candy bars. The decorator is like the foil wrapper, and the decorated function is like the chocolate inside.

[Slide]

But how is this even possible? That decorator code looks confusing!

[Slide]

Decorators are possible because, in Python, functions are objects. In fancy language, we say functions are “first-order” values. Since functions are just objects, …

[Slide]

…We can pass them into other functions as arguments, …

[Slide]

…define new functions inside existing functions, …

[Slide]

…and return a function from a function.

[Slide]

This is all part of a paradigm called “Functional Programming.” Python supports functional programming because functions can be treated like objects. That’s awesome!

[Slide]

So, using functions as objects, decorators change how functions are called.

[Slide]

Decorators create an “outer” decorator function around an “inner” decorated function. Remember, the outer function is like the foil wrapper, and the inner function is like the chocolate.

[Slide]

Creating an outer function lets you add new code around the inner function. Some people call this “advice.” You can add advice before or after the inner function. You could even skip the inner function!

[Slide]

The best part is, decorators can be applied to any function. They make sharing code easy so you don’t repeat yourself!

[Slide]

Decorators are reminiscent of a paradigm called “Aspect-Oriented Programming,” in which code can be cleverly inserted before and after points of execution. Neat!

[Slide]

So remember, decorators wrap functions around functions, like candy bars!

[Slide]

Hold on, now! We have a problem in that Python code!

[Slide]

If the “wrapper” function effectively replaces “hello_world”, then what identity does “hello_world” report?

[Slide]

Its name is “wrapper”…

[Slide]

And its help is also “wrapper”! That’s not right!

[Slide]

Never fear! There’s an easy solution. The “functools” module provides a decorator named “wraps”. Put “functools.wraps” on the “wrapper” function and pass in the inner function object, and decorated functions once again show the right identity. That’s awesome.

[Slide]

But wait, there’s another problem!

[Slide]

How do decorators work with inputs and outputs? What if we decorate a function with parameters and a return value?

[Slide]

If we try to use the current “tracer”, …

[Slide]

…We get an error! Arguments broke it!

[Slide]

We can fix it! First, add “star args” and “star-star k-w-args” to the “wrapper” function’s parameters, and then pass them through to the inner function. This will make sure all arguments go through the decorator into the decorated function.

[Slide]

Then, capture the inner function’s return value and return it from the “wrapper” function. This makes sure return values also pass through. If the inner function has no return value, don’t worry – the decorator will pass through a “None” value.

[Slide]

When we call the function with the updated “tracer”, …

[Slide]

…we see tracing is now successful again!

[Slide]

When we check the return value, …

[Slide]

…it’s exactly what we expect. It works!

[Slide]

Wow, that’s awesome!

[Slide]

But wait, there’s more!

[Slide]

You can write a decorator to call a function twice!

[Slide]

Start with the decorator template…

[Slide]

…and call the inner function twice! Return the final return value for continuity.

[Slide]

BAM! It works!

[Slide]

But wait, there’s more!

[Slide]

You can write a timer decorator!

[Slide]

Start with the template, …

[Slide]

…call the inner function, …

[Slide]

…and surround it with timestamps using the “time” module!

[Slide]

BAM! Now you can time any function!

[Slide]

But wait, there’s more!

[Slide]

You can also add more than one decorator to a function! This is called “nesting”. Order matters. Decorators are executed in order of closeness to the inner function. So, in this case, …

[Slide]

…”call_twice” is applied first, and then “timer” is applied.

[Slide]

If these decorators are reversed, …

[Slide]

…then each inner function call is timed. Cool!

[Slide]

But wait, there’s more!

[Slide]

You can scrub and validate function arguments! Check out these two simple math functions.

[Slide]

Create a decorator to scrub and validate inputs as integers.

[Slide]

Add the wrapper function, and make sure it has positional args.

[Slide]

Then, cast all args as ints before passing them into the inner function.

[Slide]

Now, when calling those math functions, all numbers are integers! Using non-numeric inputs also raises a ValueError!

[Slide]

But wait, there’s more!

[Slide]

You can create decorators with parameters! Here’s a decorator that will repeat a function 5 times.

[Slide]

The “repeat” function is a little different. Instead of taking in the inner function object, it takes in the parameter, which is the number of times to repeat the inner function.

[Slide]

Inside, there’s a “repeat_decorator” function that has a parameter for the inner function. The “repeat” function returns the “repeat_decorator” function.

[Slide]

Inside “repeat_decorator” is the “wrapper” function. It uses “functools.wraps” and passes through all arguments. “repeat_decorator” returns “wrapper”.

[Slide]

Finally, “wrapper” contains the logic for calling the inner function multiple times, according to the “repeat” decorator’s parameter value.

[Slide]

Now, “hello_world” runs 5 times. Nifty!

[Slide]

But wait, there’s more!

[Slide]

Decorators can be used to save state! Here’s a decorator that will count the number of times a function is called.

[Slide]

“count_calls” has the standard decorator structure.

[Slide]

Outside the wrapper, a “count” attribute is initialized to 0. This attribute is added to the wrapper function object.

[Slide]

Inside the wrapper, the count is incremented before calling the inner function. The “count” value will persist across multiple calls.

[Slide]

Initially, the “hello_world” count value is 0.

[Slide]

After two calls, the count value goes up! Awesome!

[Slide]

But wait, there’s more!

[Slide]

Decorators can be used in classes! Here, the “timer” decorator is applied to this “hello” method.

[Slider]

As long as parameters and return values are set up correctly, decorators can be applied equally to functions and methods.

[Slide]

Decorators can also be applied directly to classes!

[Slide]

When a decorator is applied to a class, it wraps the constructor.

[Slide]

Note that it does not wrap each method in the class.

[Slide]

Since decorators can wrap classes and methods in addition to functions, it would technically be more correct to say that decorators wrap callables around callables!

[Slide]

So all that’s great, but can decorators be tested? Good code must arguably be testable code. Well, today’s your lucky day, because yes, you can test decorators!

[Slide]

Testing decorators can be a challenge. We should always try to test the code we write, but decorators can be tricky. Here’s some advice:

[Slide]

First, separate tests for decorator functions from decorated functions. For decorator functions, focus on intended outcomes. Try to focus on the “wrapper” instead of the “inner” function. Remember, decorators can be applied to any callable, so cover the parts that make decorators unique. Decorated functions should have their own separate unit tests.

[Slide]

Second, apply decorators to “fake” functions used only for testing. These functions can be simple or mocked. That way, unit tests won’t have dependencies on existing functions that could change. Tests will also be simpler if they use slimmed-down decorated functions.

[Slide]

Third, make sure decorators have test coverage for every possible way it could be used. Cover decorator parameters, decorated function arguments, and return values. Make sure the “name” and “help” are correct. Check any side effects like saved state. Try it on methods and classes as well as functions. With decorators, most failures happen due to overlooked edge cases.

[Slide]

Let’s look at a few short decorator tests. We’ll use the “count_calls” decorator from earlier.

There are two decorated functions to use for testing. The first one is a “no operation” function that does nothing. It has no parameters or returns. The second one is a function that takes in one argument and returns it. Both are very simple, but they represent two equivalences classes of decoratable functions.

[Slide]

The test cases will verify outcomes of using the decorator. For “count_calls”, that means tests will focus on the “count” attribute added to decorated functions.

The first test case verifies that the initial count value for any function is zero.

[Slide]

The second test calls a function three times and verifies that count is three.

[Slide]

The third test exercises the “same” function to make sure arguments and return values work correctly. It calls the “same” function, asserts the return value, and asserts the count value.

This collection of tests is by no means complete. It simply shows how to start writing tests for decorators. It also shows that you don’t need to overthink unit tests for decorators. Simple is better than complex!

[Slide]

Up to this point, we’ve covered how to write your own decorators. However, Python has several decorators available in the language and in various modules that you can use, absolutely free!

[Slide]

Decorators like “classmethod”, “staticmethod”, and “property” can apply to methods in a class. Frameworks like Flask and pytest have even more decorators. Let’s take a closer look.

[Slide]

Let’s start by comparing the “classmethod” and “staticmethod” decorators. We’ll revisit the “Greeter” class we saw before.

[Slide]

The “classmethod” decorator will turn any method into a “class” method instead of an “instance” method. That means this “hello” method here can be called directly from the class itself instead of from an object of the class. This decorator will pass a reference to the class into the method so the method has some context of the class. Here, the reference is named “c-l-s”, and the method uses it to get the class’s name. The method can be called using “Greeter.hello”. Wow!

[Slide]

The “staticmethod” decorator works almost the same as the “classmethod” decorator, except that it does not pass a reference to the class into the method.

[Slide]

Notice how the method parameters are empty – no “class” and no “self”. Methods are still called from the class, like here with “Greeter.goodbye”. You could say that “staticmethod” is just a simpler version of “classmethod”.

[Slide]

Next, let’s take a look at the “property” decorator. To show how to use it, we’ll create a class called “Accumulator” to keep count of a tally.

[Slide]

Accumulator’s “init” method initializes a “count” attribute to 0.

[Slide]

An “add” method adds an amount to the count. So far, nothing unusual.

[Slide]

Now, let’s add a property. This “count” method has the “property” decorator on it. This means that “count” will be callable as an attribute instead of a method, meaning that it won’t need parentheses. It is effectively a “getter”. The calls to “count” in the “init” and “add” methods will actually call this property instead of a raw variable.

Inside the “count” property, the method returns an attribute named “underscore-count”. The underscore means that this variable should be private. However, this class hasn’t set that variable yet.

[Slide]

That variable is set in the “setter” method. Setters are optional for properties. Here, the setter validates that the value to set is not negative. If the value is good, then it sets “underscore-count”. If the value is negative, then it raises a ValueError.

“underscore-count” is handled internally, while “count” is handled publicly as the property. The getter and setter controls added by the “property” decorator let you control how the property is handled. In this class, the setter protects the property against bad values!

[Slide]

So, let’s use this class. When an Accumulator object is constructed, its initial count is 0.

[Slide]

After adding an amount to the object, its count goes up.

[Slide]

Its count can be directly set to non-negative values. Attempting to set the count directly to a negative value raises an exception, as expected. Protection like that is great!

[Slide]

Python packages also frequently contain decorators. For example, Flask is a very popular Web “micro-framework” that enables you to write Web APIs with very little Python code.

[Slide]

Here’s an example “Hello World” Flask app taken directly from the Flask docs online. It imports the “flask” module, creates the app, and defines a single endpoint at the root path that returns the string, “Hello, World!” Flask’s “app.route” decorator can turn any function into a Web API endpoint. That’s awesome!

[Slide]

Another popular Python package with decorators is pytest, Python’s most popular test framework.

[Slide]

One of pytest’s best features is the ability to parametrize test functions to run for multiple input combinations. Test parameters empower data driven testing for wider test coverage!

[Slide]

To show how this works, we’ll use a simple test for basic arithmetic: “test addition”. It asserts that a plus b equals c.

[Slide]

The values for a, b, and c must come from a list of tuples. For example, 1 plus 2 equals 3, and so forth.

[Slide]

The “pytest.mark.parametrize” decorator connects the list of test values to the test function. It runs the test once for each tuple in the list, and it injects the tuple values into the test case as function arguments. This test case would run four times. Test parameters are a great way to rerun test logic without repeating test code.

[Slide]

So, act now, before it’s too late!

[Slide]

When should you use decorators in your Python code?

[Slide]

Use decorators for aspects.

[Slide]

An aspect is a special cross-cutting concern. They are things that happen in many parts of the code, and they frequently require repetitive calls.

Think about something like logging. If you want to add logging statements to different parts of the code, then you need to write multiple logging calls in all those places. Logging itself is one concern, but it cross-cuts the whole code base. One solution for logging could be to use decorators, much like we saw earlier with the “tracer” decorator.

[Slide]

Good use cases for decorators include logging, profiling, input validation, retries, and registries. These are things that typically require lots of extra calls inserted in duplicative ways. Ask yourself this:

[Slide]

Should the code wrap something else? If yes, then you have a good candidate for a decorator.

[Slide]

However, decorators aren’t good for all circumstances. You should avoid decorators for “main” behaviors, because those should probably be put directly in the body of the decorated function. Avoid logic that’s complicated or has heavy conditionals, too, because simple is better than complex. You should also try to avoid completely side-stepping the decorated function – that could confuse people!

[Slide]

Ask yourself this: is the code you want to write the wrapper or the candy bar itself? Wrappers make good decorators, but candy bars do not.

[Slide]

I hope you’ve found this infomercial about decorators useful! If you want to learn more, …

[Slide]

…check out this Real Python tutorial by Geir Arne Hjelle named “Primer on Python Decorators”. It covers everything I showed here, plus more.

[Slide]

Thank you very much for listening! Again, my name is Pandy Knight – the Automation Panda and a bona fide Pythonista. Please read my blog at AutomationPanda.com, and follow me on Twitter @AutomationPanda. I’d love to hear what y’all end up doing with decorators!