Are Automated Test Retries Good or Bad?

What happens when a test fails? If someone is manually running the test, then they will pause and poke around to learn more about the problem. However, when an automated test fails, the rest of the suite keeps running. Testers won’t get to view results until the suite is complete, and the automation won’t perform any extra exploration at the time of failure. Instead, testers must review logs and other artifacts gathered during testing, and they even might need to rerun the failed test to check if the failure is consistent.

Since testers typically rerun failed tests as part of their investigation, why not configure automated tests to automatically rerun failed tests? On the surface, this seems logical: automated retries can eliminate one more manual step. Unfortunately, automated retries can also enable poor practices, like ignoring legitimate issues.

So, are automated test retries good or bad? This is actually a rather controversial topic. I’ve heard many voices strongly condemn automated retries as an antipattern (see here, here, and here). While I agree that automated retries can be abused, I nevertheless still believe they can add value to test automation. A deeper understanding needs a nuanced approach.

So, how do automated retries work?

To avoid any confusion, let’s carefully define what we mean by “automated test retries.”

Let’s say I have a suite of 100 automated tests. When I run these tests, the framework will execute each test individually and yield a pass or fail result for the test. At the end of the suite, the framework will aggregate all the results together into one report. In the best case, all tests pass: 100/100.

However, suppose that one of the tests fails. Upon failure, the test framework would capture any exceptions, perform any cleanup routines, log a failure, and safely move onto the next test case. At the end of the suite, the report would show 99/100 passing tests with one test failure.

By default, most test frameworks will run each test one time. However, some test frameworks have features for automatically rerunning test cases that fail. The framework may even enable testers to specify how many retries to attempt. So, let’s say that we configure 2 retries for our suite of 100 tests. When that one test fails, the framework would queue that failing test to run twice more before moving onto the next test. It would also add more information to the test report. For example, if one retry passed but another one failed, the report would show 99/100 passing tests with a 1/3 pass rate for the failing test.

In this article, we will focus on automated retries for test cases. Testers could also program other types of retries into automated tests, such as retrying browser page loads or REST requests. Interaction-level retries require sophisticated, context-specific logic, whereas test-level retry logic works the same for any kind of test case. (Interaction-level retries would also need their own article.)

Automated retries can be a terrible antipattern

Let’s see how automated test retries can be abused:

Jeremy is a member of a team that runs a suite of 300 automated tests for their web app every night. Unfortunately, the tests are notoriously flaky. About a dozen different tests fail every night, and Jeremy spends a lot of time each morning triaging the failures. Whenever he reruns failed tests individually on his laptop, they almost always pass.

To save himself time in the morning, Jeremy decides to add automatic retries to the test suite. Whenever a test fails, the framework will attempt one retry. Jeremy will only investigate tests whose retries failed. If a test had a passing retry, then he will presume that the original failure was just a flaky test.

Ouch! There are several problems here.

First, Jeremy is using retries to conceal information rather than reveal information. If a test fails but its retries pass, then the test still reveals a problem! In this case, the underlying problem is flaky behavior. Jeremy is using automated retries to overwrite intermittent failures with intermittent passes. Instead, he should investigate why the test are flaky. Perhaps automated interactions have race conditions that need more careful waiting. Or, perhaps features in the web app itself are behaving unexpectedly. Test failures indicate a problem – either in test code, product code, or infrastructure.

Second, Jeremy is using automated retries to perpetuate poor practices. Before adding automated retries to the test suite, Jeremy was already manually retrying tests and disregarding flaky failures. Adding retries to the test suite merely speeds up the process, making it easier to sidestep failures.

Third, the way Jeremy uses automated retries indicates that the team does not value their automated test suite very much. Good test automation requires effort and investment. Persistent flakiness is a sign of neglect, and it fosters low trust in testing. Using retries is merely a “band-aid” on both the test failures and the team’s attitude about test automation.

In this example, automated test retries are indeed a terrible antipattern. They enable Jeremy and his team to ignore legitimate issues. In fact, they incentivize the team to ignore failures because they institutionalize the practice of replacing red X’s with green checkmarks. This team should scrap automated test retries and address the root causes of flakiness.

green check red x
Testers should not conceal failures by overwriting them with passes.

Automated retries are not the main problem

Ignoring flaky failures is unfortunately all too common in the software industry. I must admit that in my days as a newbie engineer, I was guilty of rerunning tests to get them to pass. Why do people do this? The answer is simple: intermittent failures are difficult to resolve.

Testers love to find consistent, reproducible failures because those are easy to explain. Other developers can’t push back against hard evidence. However, intermittent failures take much more time to isolate. Root causes can become mind-bending puzzles. They might be triggered by environmental factors or awkward timings. Sometimes, teams never figure out what causes them. In my personal experience, bug tickets for intermittent failures get far less traction than bug tickets for consistent failures. All these factors incentivize folks to turn a blind eye to intermittent failures when convenient.

Automated retries are just a tool and a technique. They may enable bad practices, but they aren’t inherently bad. The main problem is willfully ignoring certain test results.

Automated retries can be incredibly helpful

So, what is the right way to use automated test retries? Use them to gather more information from the tests. Test results are simply artifacts of feedback. They reveal how a software product behaved under specific conditions and stimuli. The pass-or-fail nature of assertions simplifies test results at the top level of a report in order to draw attention to failures. However, reports can give more information than just binary pass-or-fail results. Automated test retries yield a series of results for a failing test that indicate a success rate.

For example, SpecFlow and the SpecFlow+ Runner make it easy to use automatic retries the right way. Testers simply need to add the retryFor setting to their SpecFlow+ Runner profile to set the number of retries to attempt. In the final report, SpecFlow records the success rate of each test with color-coded counts. Results are revealed, not concealed.

Here is a snippet of the SpecFlow+ Report showing both intermittent failures (in orange) and consistent failures (in red).

This information jumpstarts analysis. As a tester, one of the first questions I ask myself about a failing test is, “Is the failure reproducible?” Without automated retries, I need to manually rerun the test to find out – often at a much later time and potentially within a different context. With automated retries, that step happens automatically and in the same context. Analysis then takes two branches:

  1. If all retry attempts failed, then the failure is probably consistent and reproducible. I would expect it to be a clear functional failure that would be fast and easy to report. I jump on these first to get them out of the way.
  2. If some retry attempts passed, then the failure is intermittent, and it will probably take more time to investigate. I will look more closely at the logs and screenshots to determine what went wrong. I will try to exercise the product behavior manually to see if the product itself is inconsistent. I will also review the automation code to make sure there are no unhandled race conditions. I might even need to rerun the test multiple times to measure a more accurate failure rate.

I do not ignore any failures. Instead, I use automated retries to gather more information about the nature of the failures. In the moment, this extra info helps me expedite triage. Over time, the trends this info reveals helps me identify weak spots in both the product under test and the test automation.

Automated retries are most helpful at high scale

When used appropriate, automated retries can be helpful for any size test automation project. However, they are arguably more helpful for large projects running tests at high scale than small projects. Why? Two main reasons: complexities and priorities.

Large-scale test projects have many moving parts. For example, at PrecisionLender, we presently run 4K-10K end-to-end tests against our web app every business day. (We also run ~100K unit tests every business day.) Our tests launch from TeamCity as part of our Continuous Integration system, and they use in-house Selenium Grid instances to run 50-100 tests in parallel. The PrecisionLender application itself is enormous, too.

Intermittent failures are inevitable in large-scale projects for many different reasons. There could be problems in the test code, but those aren’t the only possible problems. At PrecisionLender, Boa Constrictor already protects us from race conditions, so our intermittent test failures are rarely due to problems in automation code. Other causes for flakiness include:

  • The app’s complexity makes certain features behave inconsistently or unexpectedly
  • Extra load on the app slows down response times
  • The cloud hosting platform has a service blip
  • Selenium Grid arbitrarily chokes on a browser session
  • The DevOps team recycles some resources
  • An engineer makes a system change while tests were running
  • The CI pipeline deploys a new change in the middle of testing

Many of these problems result from infrastructure and process. They can’t easily be fixed, especially when environments are shared. As one tester, I can’t rewrite my whole company’s CI pipeline to be “better.” I can’t rearchitect the app’s whole delivery model to avoid all collisions. I can’t perfectly guarantee 100% uptime for my cloud resources or my test tools like Selenium Grid. Some of these might be good initiatives to pursue, but one tester’s dictates do not immediately become reality. Many times, we need to work with what we have. Curt demands to “just fix the tests” come off as pedantic.

Automated test retries provide very useful information for discerning the nature of such intermittent failures. For example, at PrecisionLender, we hit Selenium Grid problems frequently. Roughly 1/10000 Selenium Grid browser sessions will inexplicably freeze during testing. We don’t know why this happens, and our investigations have been unfruitful. We chalk it up to minor instability at scale. Whenever the 1/10000 failure strikes, our suite’s automated retries kick in and pass. When we review the test report, we see the intermittent failure along with its exception method. Based on its signature, we immediately know that test is fine. We don’t need to do extra investigation work or manual reruns. Automated retries gave us the info we needed.

Selenium Grid
Selenium Grid is a large cluster with many potential points of failure.
(Image source: LambdaTest.)

Another type of common failure is intermittently slow performance in the PrecisionLender application. Occasionally, the app will freeze for a minute or two and then recover. When that happens, we see a “brick wall” of failures in our report: all tests during that time frame fail. Then, automated retries kick in, and the tests pass once the app recovers. Automatic retries prove in the moment that the app momentarily froze but that the individual behaviors covered by the tests are okay. This indicates functional correctness for the behaviors amidst a performance failure in the app. Our team has used these kinds of results on multiple occasions to identify performance bugs in the app by cross-checking system logs and database queries during the time intervals for those brick walls of intermittent failures. Again, automated retries gave us extra information that helped us find deep issues.

Automated retries delineate failure priorities

That answers complexity, but what about priority? Unfortunately, in large projects, there is more work to do than any team can handle. Teams need to make tough decisions about what to do now, what to do later, and what to skip. That’s just business. Testing decisions become part of that prioritization.

In almost all cases, consistent failures are inherently a higher priority than intermittent failures because they have a greater impact on the end users. If a feature fails every single time it is attempted, then the user is blocked from using the feature, and they cannot receive any value from it. However, if a feature works some of the time, then the user can still get some value out of it. Furthermore, the rarer the intermittency, the lower the impact, and consequentially the lower the priority. Intermittent failures are still important to address, but they must be prioritized relative to other work at hand.

Automated test retries automate that initial prioritization. When I triage PrecisionLender tests, I look into consistent “red” failures first. Our SpecFlow reports make them very obvious. I know those failures will be straightforward to reproduce, explain, and hopefully resolve. Then, I look into intermittent “orange” failures second. Those take more time. I can quickly identify issues like Selenium Grid disconnections, but other issues may not be obvious (like system interruptions) or may need additional context (like the performance freezes). Sometimes, we may need to let tests run for a few days to get more data. If I get called away to another more urgent task while I’m triaging results, then at least I could finish the consistent failures. It’s a classic 80/20 rule: investigating consistent failures typically gives more return for less work, while investigating intermittent failures gives less return for more work. It is what it is.

The only time I would prioritize an intermittent failure over a consistent failure would be if the intermittent failure causes catastrophic or irreversible damage, like wiping out an entire system, corrupting data, or burning money. However, that type of disastrous failure is very rare. In my experience, almost all intermittent failures are due to poorly written test code, automation timeouts from poor app performance, or infrastructure blips.

Context matters

Automated test retries can be a blessing or a curse. It all depends on how testers use them. If testers use retries to reveal more information about failures, then retries greatly assist triage. Otherwise, if testers use retries to conceal intermittent failures, then they aren’t doing their jobs as testers. Folks should not be quick to presume that automated retries are always an antipattern. We couldn’t achieve our scale of testing at PrecisionLender without them. Context matters.

Should Gherkin Steps use Past, Present, or Future Tense?

Gherkin’s Given-When-Then syntax is a great structure for specifying behaviors. However, while writing Gherkin may seem easy, writing good Gherkin can be a challenge. One aspect to consider is the tense used for Gherkin steps. Should Gherkin steps use past, present, or future tense?

One approach is to use present tense for all steps, like this:

Scenario: Simple Google search
    Given the Google home page is displayed
    When the user searches for "panda"
    Then the results page shows links related to "panda"

Notice the tense of each verb:

  1. the home page is – present
  2. the user searches – present
  3. the results page shows – present

Present tense is the simplest verb tense to use. It is the least “wordy” tense, and it makes the scenario feel active.

An alternative approach is to use past-present-future tense for Given-When-Then steps respectively, like this:

Scenario: Simple Google search
    Given the Google home page was displayed
    When the user searches for "panda"
    Then the results page will show links related to "panda"

Notice the different verb tenses in this scenario:

  1. the home page was – past
  2. the user searches – present
  3. the result page will show – future

Scenarios exercise behavior. Writing When steps using present tense centers the scenario’s main actions in the present. Since Given steps must happen before the main actions, they would be written using past tense. Likewise, since Then steps represent expected outcomes after the main actions, they would be written using future tense.

Both of these approaches – using all present tense or using past-present-future in order – are good. Personally, I prefer to write all steps using present tense. It’s easier to explain to others, and it frames the full scenario in the moment. However, I don’t think other approaches are good. For example, writing all steps using past tense or future tense would seem weird, and writing steps in order of future-present-past tense would be illogical. Scenarios should be centered in the present because they should timelessly represent the behaviors they cover.

Want to learn more? Check out my other BDD articles, especially Writing Good Gherkin.

Managing the Test Data Nightmare

On April 22, 2021, I delivered a talk entitled “Managing the Test Data Nightmare” at SauceCon 2021. SauceCon is Sauce Labs’ annual conference for the testing community. Due to the COVID-19 pandemic, the conference was virtual, but I still felt a bit of that exciting conference buzz.

My talk covers the topic of test data, which can be a nightmare to handle. Data must be prepped in advance, loaded before testing, and cleaned up afterwards. Sometimes, teams don’t have much control over the data in their systems under test—it’s just dropped in, and it can change arbitrarily. Hard-coding values into tests that reference system tests can make the tests brittle, especially when running tests in different environments.

In this talk, I covered strategies for managing each type of test data: test case variations, test control inputs, config metadata, and product state. I also covered how to “discover” test data instead of hard-coding it, how to pass inputs into automation (including secrets like passwords), and how to manage data in the system. After watching this talk, you can wake up from the nightmare and handle test data cleanly and efficiently like a pro!

Here are some other articles I wrote about test data:

As usual, I hit up Twitter throughout the conference. Here are some action shots:

Many thanks to Sauce Labs and all the organizers who made SauceCon 2021 happen. If SauceCon was this awesome as a virtual event, then I can’t wait to attend in person (hopefully) in 2022!

Announcing Boa Constrictor Docs!

Doc site:
https://q2ebanking.github.io/boa-constrictor/

Boa Constrictor is a C# implementation of the Screenplay Pattern. My team and I at PrecisionLender, a Q2 Company, developed Boa Constrictor as part of our test automation solution. Its primary use case is Web UI and REST API test automation. Boa Constrictor helps you make better interactions for better automation!

Our team released Boa Constrictor as an open source project on GitHub in October 2020. This week, we published a full documentation site for Boa Constrictor. They include an introduction to the Screenplay Pattern, a quick-start guide, a full tutorial, and ways to contribute to the project. The doc site itself uses GitHub Pages, Jekyll, and Minimal Mistakes.

Our team hopes that the docs help you with testing and automation. Enjoy!

Testing GitHub Pages without Local Jekyll Setup

TL;DR: If you want to test your full GitHub Pages site before publishing but don’t want to set up Ruby and Jekyll on your local machine, then:

  1. Commit your doc changes to a new branch.
  2. Push the new branch to GitHub.
  3. Temporarily change the repository’s GitHub Pages publishing source to the new branch.
  4. Reload the GitHub Pages site, and review the changes.

If you have a GitHub repository, did you know that you can create your own documentation site for it within GitHub? Using GitHub Pages, you can write your docs as a set of Markdown pages and then configure your repository to generate and publish a static web site for those pages. All you need to do is configure a publishing source for your repository. Your doc site will go live at:

https://<user>.github.io/<repository>

If this is new to you, then you can learn all about this cool feature from the GitHub docs here: Working with GitHub Pages. I just found out about this cool feature myself!

GitHub Pages are great because they make it easy to develop docs and code together as part of the same workflow without needing extra tools. Docs can be written as Markdown files, Liquid templates, or raw assets like HTML and CSS. The docs will be version-controlled for safety and shared from a single source of truth. GitHub Pages also provides free hosting with a decent domain name for the doc site. Clearly, the theme is simplicity.

Unfortunately, I hit one challenge while trying GitHub Pages for the first time: How could I test the doc site before publishing it? A repository using GitHub Pages must be configured with a specific branch and folder (/ (root) or /docs) as the publishing source. As soon as changes are committed to that source, the updated pages go live. However, I want a way to view the doc site in its fullness before committing any changes so I don’t accidentally publish any mistakes.

One way to test pages is to use a Markdown editor. Many IDEs have Markdown editors with preview panes. Even GitHub’s web editor lets you preview Markdown before committing it. Unfortunately, while editor previews may help catch a few typos, they won’t test the full end result of static site generation and deployment. They may also have trouble with links or templates.

GitHub’s docs recommend testing your site locally using Jekyll. Jekyll is a static site generator written in Ruby. GitHub Pages uses Jekyll behind the scenes to turn doc pages into full doc sites. If you want to keep your doc development simple, you can just edit Markdown files and let GitHub do the dirty work. However, if you want to do more hands-on things with your docs like testing site generation, then you need to set up Ruby and Jekyll on your local machine. Thankfully, you don’t need to know any Ruby programming to use Jekyll.

I followed GitHub’s instructions for setting up a GitHub Pages site with Jekyll. I installed Ruby and Jekyll and then created a Jekyll site in the /docs folder of my repository. I verified that I could edit and run my site locally in a branch. However, the setup process felt rather hefty. I’m not a Ruby programmer, so setting up a Ruby environment with a few gems felt like a lot of extra work just to verify that my doc pages looked okay. Plus, I could foresee some developers getting stuck while trying to set up these doc tools, especially if the repository’s main code isn’t a Ruby project. Even if setting up Jekyll locally would be the “right” way to develop and test docs, I still wanted a lighter, faster alternative.

Thankfully, I found a workaround that didn’t require any tools outside of GitHub: Commit doc changes to a branch, push the branch to GitHub, and then temporarily change the repository’s GitHub Pages source to the branch! I originally configured my repository to publish docs from the /docs folder in the main branch. When I changed the publishing source to another branch, it regenerated and refreshed the GitHub Pages site. When I changed it back to main, the site reverted without any issues. Eureka! This is a quick, easy hack for testing changes to docs before merging them. You get to try the full site in the main environment without needing any additional tools or setup.

Above is a screenshot of the GitHub Pages settings for one of my repositories. You can find these settings under Settings -> Options for any repository, as long as you have the administrative rights. In this screenshot, you can see how I changed the publishing source’s branch from main to docs/test. As soon as I selected this change, GitHub Pages republished the repository’s doc site.

Now, I recognize that this solution is truly a hack. Changing the publishing source affects the “live”, “production” version of the site. It effectively does publish the changes, albeit temporarily. If some random reader happens to visit the site during this type of testing, they may see incorrect or even broken pages. I’d recommend changing the publishing source’s branch only for small projects and for short periods of time. Don’t forget to revert the branch once testing is complete, too. If you are working on a larger, more serious project, then I’d recommend doing full setup for local doc development. Local setup would be safer and would probably make it easier to try more advanced tricks, like templates and themes.

The Automation Panda Origin Story

In February 2021, Matthew Weeks interviewed me for the Work in Programming podcast. Matthew asked all sorts of questions about my story – how I got into programming, what I learned at different companies, and why I started blogging and speaking. I greatly enjoyed our conversation, so much so that it lasted an hour and a half!

If you’re interested to hear how my career has gone from high school to the present day, please give it a listen. There are some juicy anecdotes along the way. The link is below. Many thanks to Matthew for hosting and editing the interview. Be sure to listed to other Work in Programming interviews, too!

Work in Programming by Matthew Weeks: Andy Knight – Automation Panda origin story, BDD, test automation before it was cool

Solving: How to write good UI interaction tests? #GivenWhenThenWithStyle

Writing good Gherkin is a passion of mine. Good Gherkin means good behavior specification, which results in better features, better tests, and ultimately better software. To help folks improve their Gherkin skills, Gojko Adzic and SpecFlow are running a series of #GivenWhenThenWithStyle challenges. I love reading each new challenge, and in this article, I provide my answer to one of them.

The Challenge

Challenge 20 states:

This week, we’re looking into one of the most common pain points with Given-When-Then: writing automated tests that interact with a user interface. People new to behaviour driven development often misunderstand what kind of behaviour the specifications should describe, and they write detailed user interactions in Given-When-Then scenarios. This leads to feature files that are very easy to write, but almost impossible to understand and maintain.

Here’s a typical example:

Scenario: Signed-in users get larger capacity
 
Given a user opens https://www.example.com using Chrome
And the user clicks on "Upload Files"
And the page reloads
And the user clicks on "Spreadsheet Formats"
Then the buttons "XLS" and "XLSX" show
And the user clicks on "XLSX"
And the user selects "500kb-sheet.xlsx"
Then the upload completes
And the table "Uploaded Files" contains a cell with "500kb-sheet.xlsx" 
And the user clicks on "XLSX"
And the user selects "1mb-sheet.xlsx"
Then the upload fails
And the table "Uploaded Files" does not contain a cell with "1mb-sheet.xlsx" 
And the user clicks on "Login"
And the user enters "testuser123" into the "username" field
And the user enters "$Pass123" into the "password" field
And the user clicks on "Sign in"
And the page reloads
Then the table "Uploaded Files" contains a cell with "500kb-sheet.xlsx" 
And the table "Uploaded Files" does not contain a cell with "1mb-sheet.xlsx" 
And the user clicks on "spreadsheet formats"
Then the buttons "XLS" and "XLSX" show
And the user clicks on "XLSX"
And the user selects "1mb-sheet.xlsx"
Then the upload completes
And the table "Uploaded Files" contains a cell with "1mb-sheet.xlsx" 
And the table "Uploaded Files" contains a cell with "500kb-sheet.xlsx"

A common way to avoid such issues is to rewrite the specification to avoid the user interface completely. We’ve looked into that option several times in this article series. However, that solution only applies if the risk we’re testing is not in the user interface, but somewhere below. To make this challenge more interesting, let’s say that we actually want to include the user interface in the test, since the risk is in the UI interactions.

Indeed, most behavior-driven practitioners would generally recommend against phrasing steps using language specific to the user interface. However, there are times when testing a user interface itself is valid. For example, I work at PrecisionLender, a Q2 Company, and our main web app is very heavy on the front end. It has many, many interconnected fields for pricing commercial lending opportunities. My team has quite a few tests to cover UI-centric behaviors, such as verifying that entering a new interest rate triggers recalculation for summary amounts. If the target behavior is a piece of UI functionality, and the risk it bears warrants test coverage, then so be it.

Let’s break down the example scenario given above to see how to write Gherkin with style for user interface tests.

Understanding Behavior

Behavior is behavior. If you can describe it, then you can do it. Everything exhibits behavior, from the source code itself to the API, UIs, and full end-to-end workflows. Gherkin scenarios should use verbiage that reflects the context of the target behavior. Thus, the example above uses words like “click,” “select,” and “open.” Since the scenario explicitly covers a user interface, I think it is okay to use these words here. What bothers me, however, are two apparent code smells:

  1. The wall of text
  2. Out-of-order step types

The first issue is the wall of text this scenario presents. Walls of text are hard to read because they present too much information at once. The reader must take time to read through the whole chunk. Many readers simply read the first few lines and then skip the remainder. The example scenario has 27 Given-When-Then steps. Typically, I recommend Gherkin scenarios to have single-digit line length. A scenario with less than 10 steps is easier to understand and less likely to include unnecessary information. Longer scenarios are not necessarily “wrong,” but their longer lengths indicate that, perhaps, these scenarios could be rewritten more concisely.

The second issue in the example scenario is that step types are out of order. Given-When-Then is a formula for success. Gherkin steps should follow strict Given → When → Then ordering because this ordering demarcates individual behaviors. Each Gherkin scenario should cover one individual behavior so that the target behavior is easier to understand, easier to communicate, and easier to investigate whenever the scenario fails during testing. When scenarios break the order of steps, such as Given → Then → Given → Then in the example scenario, it shows that either the scenario covers multiple behaviors or that the author did not bring a behavior-driven understanding to the scenario.

The rules of good behavior don’t disappear when the type of target behavior changes. We should still write Gherkin with best practices in mind, even if our scenarios cover user interfaces.

Breaking Down Scenarios

If I were to rewrite the example scenario, I would start by isolating individual behaviors. Let’s look at the first half of the original example:

Given a user opens https://www.example.com using Chrome
And the user clicks on "Upload Files"
And the page reloads
And the user clicks on "Spreadsheet Formats"
Then the buttons "XLS" and "XLSX" show
And the user clicks on "XLSX"
And the user selects "500kb-sheet.xlsx"
Then the upload completes
And the table "Uploaded Files" contains a cell with "500kb-sheet.xlsx" 
And the user clicks on "XLSX"
And the user selects "1mb-sheet.xlsx"
Then the upload fails
And the table "Uploaded Files" does not contain a cell with "1mb-sheet.xlsx"

Here, I see four distinct behaviors covered:

  1. Clicking “Upload Files” reloads the page.
  2. Clicking “Spreadsheet Formats” displays new buttons.
  3. Uploading a spreadsheet file makes the filename appear on the page.
  4. Attempting to upload a spreadsheet file that is 1MB or larger fails.

If I wanted to purely retain the same coverage, then I would rewrite these behavior specs using the following scenarios:

Feature: Example site
 
 
Scenario: Choose to upload files
 
Given the Example site is displayed
When the user clicks the "Upload Files" link
Then the page displays the "Spreadsheet Formats" link
 
 
Scenario: Choose to upload spreadsheets
 
Given the Example site is ready to upload files
When the user clicks the "Spreadsheet Formats" link
Then the page displays the "XLS" and "XLSX" buttons
 
 
Scenario: Upload a spreadsheet file that is smaller than 1MB
 
Given the Example site is ready to upload spreadsheet files
When the user clicks the "XLSX" button
And the user selects "500kb-sheet.xlsx" from the file upload dialog
Then the upload completes
And the table "Uploaded Files" contains a cell with "500kb-sheet.xlsx" 
 
 
Scenario: Upload a spreadsheet file that is larger than or equal to 1MB
 
Given the Example site is ready to upload spreadsheet files
When the user clicks the "XLSX" button
And the user selects "1mb-sheet.xlsx" from the file upload dialog
Then the upload fails
And the table "Uploaded Files" does not contain a cell with "1mb-sheet.xlsx"

Now, each scenario covers each individual behavior. The first scenario starts with the Example site in a “blank” state: “Given the Example site is displayed”. The second scenario inherently depends upon the outcome of the first scenario. Rather than repeat all the steps from the first scenario, I wrote a new starting step to establish the initial state more declaratively: “Given the Example site is ready to upload files”. This step’s definition method may need to rerun the same operations as the first scenario, but it guarantees independence between scenarios. (The step could also optimize the operations, but that should be a topic for another challenge.) Likewise, the third and fourth scenarios have a Given step to establish the state they need: “Given the Example site is ready to upload spreadsheet files.” Both scenarios can share the same Given step because they have the same starting point. All three of these new steps are descriptive more than prescriptive. They declaratively establish an initial state, and they leave the details to the automation code in the step definition methods to determine precisely how that state is established. This technique makes it easy for Gherkin scenarios to be individually clear and independently executable.

I also added my own writing style to these scenarios. First, I wrote concise, declarative titles for each scenario. The titles dictate interaction over mechanics. For example, the first scenario’s title uses the word “choose” rather than “click” because, from the user’s perspective, they are “choosing” an action to take. The user will just happen to mechanically “click” a link in the process of making their choice. The titles also provide a level of example. Note that the third and fourth scenarios spell out the target file sizes. For brevity, I typically write scenario titles using active voice: “Choose this,” “Upload that,” or “Do something.” I try to avoid including verification language in titles unless it is necessary to distinguish behaviors.

Another stylistic element of mine was to remove explicit details about the environment. Instead of hard coding the website URL, I gave the site a proper name: “Example site.” I also removed the mention of Chrome as the browser. These details are environment-specific, and they should not be specified in Gherkin. In theory, this site could have multiple instances (like an alpha or a beta), and it should probably run in any major browser (like Firefox and Edge). Environmental characteristics should be specified as inputs to the automation code instead.I also refined some of the language used in the When and Then steps. When I must write steps for mechanical actions like clicks, I like to specify element types for target elements. For example, “When the user clicks the “Upload Files” link” specifies a link by a parameterized name. Saying the element is a link helps provides context to the reader about the user interface. I wrote other steps that specify a button, too. These steps also specified the element name as a parameter so that the step definition method could possibly perform the same interaction for different elements. Keep in mind, however, that these linguistic changes are neither “required” nor “perfect.” They make sense in the immediate context of this feature. While automating step definitions or writing more scenarios, I may revisit the verbiage and do some refactoring.

Determining Value for Each Behavior

The four new scenarios I wrote each covers an independent, individual behavior of the fictitious Example site’s user interface. They are thorough in their level of coverage for these small behaviors. However, not all behaviors may be equally important to cover. Some behaviors are simply more important than others, and thus some tests are more valuable than others. I won’t go into deep detail about how to measure risk and determine value for different tests in this article, but I will offer some suggestions regarding these example scenarios.

First and foremost, you as the tester must determine what is worth testing. These scenarios aptly specify behavior, and they will likely be very useful for collaborating with the Three Amigos, but not every scenario needs to be automated for testing. You as the tester must decide. You may decide that all four of these example scenarios are valuable and should be added to the automated test suite. That’s a fine decision. However, you may instead decide that certain user interface mechanics are not worth explicitly testing. That’s also a fine decision.

In my opinion, the first two scenarios could be candidates for the chopping block:

  1. Choose to upload files
  2. Choose to upload spreadsheets

Even though these are existing behaviors in the Example site, they are tiny. The tests simply verify that a user clicks makes certain links or buttons appear. It would be nice to verify them, but test execution time is finite, and user interface tests are notoriously slow compared to other tests. Consider the Rule of 1’s: typically, by orders of magnitude, a unit test takes about 1 millisecond, a service API test takes about 1 second, and a web UI test takes about 1 minute. Furthermore, these behaviors are implicitly exercised by the other scenarios, even if they don’t have explicit assertions.

One way to condense the scenarios could be like this:

Feature: Example site
 
 
Background:
 
Given the Example site is displayed
When the user clicks the "Upload Files" link
And the user clicks the "Spreadsheet Formats" link
And the user clicks the "XLSX" button
 
 
Scenario: Upload a spreadsheet file that is smaller than 1MB
 
When the user selects "500kb-sheet.xlsx" from the file upload dialog
Then the upload completes
And the table "Uploaded Files" contains a cell with "500kb-sheet.xlsx" 
 
 
Scenario: Upload a spreadsheet file that is larger than or equal to 1MB
 
When the user selects "1mb-sheet.xlsx" from the file upload dialog
Then the upload fails
And the table "Uploaded Files" does not contain a cell with "1mb-sheet.xlsx" 

This new feature file eliminates the first two scenarios and uses a Background section to cover the setup steps. It also eliminates the need for special Given steps in each scenario to set unique starting points. Implicitly, if the “Upload Files” or “Spreadsheet Formats” links fail to display the expected elements, then those steps would fail.

Again, this modification is not necessarily the “best” way or the “right” way to cover the desired behaviors, but it is a reasonably good way to do so. However, I would assert that both the 4-scenario feature file and the 2-scenario feature file are much better approaches than the original example scenario.

More Gherkin

What I showed in my answer to this Gherkin challenge is how I would handle UI-centric behaviors. I try to keep my Gherkin scenarios concise and focused on individual, independent behaviors. Try using these style techniques to rewrite the second half of Gojko’s original scenario. Feel free to drop your Gherkin in the comments below. I look forward to seeing how y’all write #GivenWhenThenWithStyle!

Extending Grace in Small Ways

Back in 2011, I was a recent college grad working at IBM as a “performance engineer” for z/OS mainframe software. Now, I didn’t know anything about mainframes, but I was thankful to have a job on the heels of the Great Recession.

At the time, IBM had recently released the Jazz platform with Rational Team Concert (RTC), a collaborative project management tool geared towards Agile software development. Teams company-wide started adopting RTC whether they wanted it or not. My team was no different: we created a team project in RTC and started writing work items in it. In my opinion, RTC was decent. It was very customizable, and its aesthetics and user experience were better than other tools at the time.

One day, I made a typo while trying to assign a work item to myself. When typing a name into the “owner” field, RTC would show a list of names from which to choose. For whatever reason, the list included all IBM employees, not just members from my team. IBM had nearly 400,000 employees worldwide at the time. I accidentally selected someone else with a similar name to mine. Blissfully unaware of my mistake, I proceeded to save the work item and start doing the actual work for it.

About a day later, I received a nastygram from another IBMer named Andrea Knight, demanding to know why I assigned her this work item in RTC. I had never met this person before, and she certainly wasn’t on my team. (To be honest, I don’t remember exactly what her name was, but for the sake of the story, we can call her Andrea.) At first, I felt perplexed. Then, once I read her message, I quickly realized that I must have accidentally listed her as the owner of the work item. I immediately corrected the mistake and humbly replied with an apology for my typo. No big deal, right?

Well, Andrea replied to my brief apology later that day to inform me that she was NOT responsible for that work item because she had NEVER seen it before and that she would NOT do any work for it.

Really GIFs | Tenor
What?

I was quite taken back by her response.

I let it go, but I couldn’t help but wonder why she would answer that way. Perhaps she was having a bad day? Perhaps her manager scrutinized all work items bearing her name? Perhaps the culture in her part of the company was toxic? Was my mistake that bad?

Even though this incident was small, it taught me one important lesson early in my career: a little bit of grace goes a long way. Poor reactions create awkward situations, hurt feelings, and wasted time. If we make a mistake, we should fix it and apologize. If someone else makes a mistake, we should strive to be gracious instead of unpleasant. I try to practice this myself, though, sometimes, I fail.

Nobody is perfect. That’s why we all need grace.

Improving Teamwork with SpecFlow+ LivingDoc

SpecFlow is an excellent Behavior-Driven Development test framework for .NET. Recently, SpecFlow released a new reporting tool called SpecFlow+ LivingDoc, which generates living documentation for features. It combines all scenarios from all SpecFlow feature files into one central HTML report. The report looks crisp and professional. It is filterable and can optionally show test results. Teams can generate updated reports as part of their Continuous Integration pipelines. The best part is that SpecFlow+ LivingDoc, along with all other features, is completely free to use – all you need to do is register for a free SpecFlow account. There is no reason for any SpecFlow project to not also use LivingDoc.

SpecFlow provides rich documentation on all of SpecFlow+ LivingDoc’s benefits, features, and configurations. In this article, I won’t simply repeat what the official docs already state. Instead, I’m going to share how my team and I at PrecisionLender, a Q2 Company, adopted SpecFlow+ LivingDoc into our test automation solution. I’ll start by giving a brief overview of how we test the PrecisionLender web app. Then, I’ll share why we wanted to make LivingDoc part of our quality workflow. Next, I’ll walk through how we added the new report to our testing pipelines. Finally, as an advanced technique, I’ll show how we modified some of the LivingDoc data files to customize our reports. My goal for this article is to demonstrate the value SpecFlow+ LivingDoc adds to BDD collaboration and automation practices.

PrecisionLender’s Test Automation

PrecisionLender is a web application that empowers commercial bankers with in-the-moment insights that help them structure and price commercial deals. Andi®, PrecisionLender’s intelligent virtual analyst, delivers these hyper-focused recommendations in real time allowing relationship managers to make data-driven decisions while pricing their commercial deals.

The PrecisionLender Opportunity Screen
(Picture taken from the PrecisionLender Support Center)

The PrecisionLender app is quite complex. It has several rich features to help bankers price any possible nuance for loan opportunities. Some banks also have unique configurations and additional features that make testing challenging.

On top of thorough unit testing, we run suites of end-to-end tests against the PrecisionLender web app. Our test automation solution is named “Boa,” and it is written in C# using SpecFlow for test cases and Boa Constrictor for Web UI and REST API interactions. We use BDD practices like Three Amigos, Example Mapping, and Good Gherkin to develop behaviors and cover them with automated tests. As of January 2021, Boa has over 1400 unique tests that target multiple test bank configurations. We run Boa tests continuously (for every code change), nightly (across all test banks), and “release-ly” (every two weeks before production deployments) at a rate of ~15K test iterations per week. Each test takes roughly half a minute to complete, and we run tests in parallel with up to 32 threads using Selenium Grid.

Introducing SpecFlow+ LivingDoc

SpecFlow+ LivingDoc is living documentation for features. SpecFlow started developing the tool a few years ago, but in recent months under Tricentis, they have significantly ramped up its development with the standalone generator and numerous feature enhancements. To learn about LivingDoc, watch this short introduction video:

When I saw the new SpecFlow+ LivingDoc reports, I couldn’t wait to try them myself. I love SpecFlow, and I’ve used it daily for the past few years. I knew it would bring value to my team at PrecisionLender.

Why Adopt SpecFlow+ LivingDoc?

My team and I wanted to bring SpecFlow+ LivingDoc into our testing workflow for a few reasons. First and foremost, we wanted to share our features with every team member, whether they were in business, development, or testing roles. I originally chose SpecFlow to be the core test framework for our Boa tests because I wanted to write all tests in plain-language Gherkin. That way, product owners and managers could read and understand our tests. We could foster better discussions about product behaviors, test coverage, and story planning. However, even though tests could be understood by anyone, we didn’t have an effective way to share them. Feature files for tests must be stored together with automation code in a repository. Folks must use Visual Studio or a version control tool like Git to view them. That’s fine for developers, but it’s inaccessible for folks who don’t code. SpecFlow+ LivingDoc breaks down that barrier. It combines all scenarios from all feature files into one consolidated HTML report. Folks could use a search bar to find the tests they need instead of plunging through directories of feature files. The report could be generated by Continuous Integration pipelines, published to a shared dashboard, or emailed directly to stakeholders. Pipelines could also update LivingDoc reports any time features change. SpecFlow+ LivingDoc would enable us to actually share our features instead of merely saying that we could.

SpecFlow+ LivingDoc Living Documentation for PrecisionLender Features

Second, we liked the concise test reporting that SpecFlow+ LivingDoc offered. The SpecFlow+ Runner report, which our team already used, provides comprehensive information about test execution: full log messages, duration times, and a complete breakdown of pass-or-fail results by feature and scenario. That information is incredibly helpful when determining why tests fail, but it is too much information when reporting failures to managers. LivingDoc provides just the right amount of information for reporting high-level status: the tests, the results per test, and the pass-or-fail totals. Folks can see test status at a glance. The visuals look nice, too.

SpecFlow+ LivingDoc Analytics for PL App Boa Tests

Third, we wanted to discover any unused step definitions in our C# automation code. The Boa test solution is a very large automation project. As of January 2020, it has over 1400 unique tests and over 1100 unique step definitions, and those numbers will increase as we continue to add new tests. Maintaining any project of this size is a challenge. Sometimes, when making changes to scenarios, old step definitions may no longer be needed, but testers may not think to remove them from the code. These unused step definitions then become “dead code” that bloats the repository. It’s easy to lose track of them. SpecFlow+ LivingDoc offers a special option to report unused step definitions on the Analytics tab. That way, the report can catch dead steps whenever they appear. When I generated the LivingDoc report for the Boa tests, I discovered over a hundred unused steps!

SpecFlow+ LivingDoc Unused Step Definitions

Fourth and finally, my team and I needed a test report that we could share with customers. At PrecisionLender, our customers are banks – and banks are very averse to risk. Some of our customers ask for our test reports so they can take confidence in the quality of our web app. When sharing any information with customers, we need to be careful about what we do share and what we don’t share. Internally, our Boa tests target multiple different system configurations, and we limit the test results we share with customers to the tests for the features they use. For example, if a bank doesn’t factor deposits into their pricing calculations, then that bank’s test report should not include any tests for deposits. The reports should also be high-level instead of granular: we want to share the tests, their scenarios, and their pass-or-fail results, but nothing more. SpecFlow+ LivingDoc fits this need perfectly. It provides Gherkin scenarios with their steps in a filterable tree, and it visually shows results for each test as well as in total. With just a little bit of data modification (as shown later in this article), the report can include exactly the intended scenarios. Our team could use LivingDoc instead of generating our own custom report for customers. LivingDoc would look better than any report we would try to make, too!

Setting Up SpecFlow+ LivingDoc

At PrecisionLender, we currently use JetBrains TeamCity to schedule and launch Boa tests. Some tests launch immediately after app deployments, while others are triggered based on the time of day. When a test pipeline is launched, it follows these steps:

  1. Check out the code repository.
  2. Build the Boa test automation solution.
  3. For each applicable bank configuration, run appropriate Boa tests.

We wanted to add SpecFlow+ LivingDoc in two places: after the build completes and after tests run for each configuration. The LivingDoc generated for the build step would not include test results. It would show all scenarios in all features, and it would also include the unused step definitions. This report would be useful for showing folks our tests and our coverage. The LivingDoc generated for each test run, however, would include test results. Since we run tests against multiple configurations, each run would need its own LivingDoc report. Not all tests run on each configuration, too. Generating LivingDoc reports at each pipeline step serve different needs.

Adding SpecFlow+ LivingDoc to our testing pipelines required only a few things to set up. The first step was to add the SpecFlow.Plus.LivingDocPlugin NuGet package to the .NET test project. Adding this NuGet package makes SpecFlow automatically save test results to a file named TestExecution.json every time tests run. The docs say you can customize this output path using specflow.json, too.

Required SpecFlow NuGet packages, including SpecFlow.Plus.LivingDoc
An example snippet of TestExecution.json

The next step was to install the LivingDoc CLI tool on our TeamCity agents. The CLI tool is a dotnet command line tool, so you need the .NET Core SDK 3.1 or higher. Also, note that you cannot install this package as a NuGet dependency for your .NET test project. (I tried to do that in the hopes of simplifying my build configuration, but NuGet blocks it.) You must install it to your machine’s command line. The installation command looks like this:

dotnet tool install --global SpecFlow.Plus.LivingDoc.CLI

After installing the LivingDoc CLI tool, the final step was to invoke it after each build and each test run. There are three sources from which to generate LivingDoc reports:

  1. Using a folder of feature files
  2. Using a SpecFlow test assembly (.dll)
  3. Using a feature data JSON file previously generated by the LivingDoc CLI tool

For generating LivingDoc after the build, I used this command in PowerShell to include unused steps but exclude test results:

livingdoc test-assembly "$TestAssemblyPath" --binding-assemblies "$TestAssemblyPath" --output-type HTML --output "$LivingDocDir\PLAppLivingDoc.html"

Then, for generating LivingDoc after test runs, I used this command in PowerShell that included TestExecution.json:

livingdoc test-assembly "$TestAssemblyPath" --test-execution-json "$TestExecutionPath" --output-type HTML --output "$HtmlReportPath" --title "PL App Boa Tests"

All the “$” variables are paths configured in our TeamCity projects. I chose to generate reports using the test assembly because I discovered that results wouldn’t appear in the report if I generated them from the feature folder.

Here’s what SpecFlow+ LivingDoc looks like when published as a TeamCity report:

SpecFlow+ LivingDoc report in TeamCity for the build (without test results)

Our team can view reports from TeamCity, or they can download them to view them locally.

Modifying SpecFlow+ LivingDoc Data

As I mentioned previously in this article, my team and I wanted to share SpecFlow+ LivingDoc reports with some of our customers. We just needed to tweak the contents of the report in two ways. First, we needed to remove scenarios that were inapplicable (meaning not executed) for the bank. Second, we needed to remove certain tags that we use internally at PrecisionLender. Scrubbing this data from the reports would give our customers what they need without including information that they shouldn’t see.

Thankfully, SpecFlow+ LivingDoc has a “backdoor” in its design that makes this kind of data modification easy. When generating a LivingDoc report, you can set the --output-type parameter to be “JSON” instead of “HTML” to generate a feature data JSON file. The feature data file contains all the data for the LivingDoc report in JSON format, including scenarios and tags. You can modify the data in this JSON file and then use it to generate an HTML LivingDoc report. Modifying JSON data is much simpler and cleaner than painfully splicing HTML text.

An example snippet of a feature data JSON file

I wrote two PowerShell scripts to modify feature data. Both are available publicly in GitHub at AndyLPK247/SpecFlowPlusLivingDocScripts. You can copy them from the repository to use them for your project, and you can even enhance them with your own changes. Note that the feature data JSON files they use must be generated from test assemblies, not from feature data folders.

The first script is RemoveSkippedScenarios.ps1. It takes in both a feature data JSON file and a test execution JSON file, and it removes all scenarios from the feature data that did not have results in the test execution data. It uses recursive functions to traverse the feature data JSON “tree” of folders, features, and scenarios. Removing unexecuted scenarios means that the LivingDoc report will only include scenarios with test results – none of the scenarios in it should be “skipped.” For my team, this means a LivingDoc report for a particular bank configuration will not include a bunch of skipped tests for other banks. Even though we currently have over 1400 unique tests, any given bank configuration may run only 1000 of those tests. The extra 400 skipped tests would be noise at best and a data privacy violation at worst.

The second script is RemoveTags.ps1. It takes in a list of tags and a feature data JSON file, and it removes all appearances of those tags from every feature, scenario, and example table. Like the script for removing skipped scenarios, it uses recursive functions to traverse the feature data JSON “tree.” The tags must be given as literal names, but the script could easily be adjusted to handle wildcard patterns or regular expressions.

With these new scripts, our test pipelines now look like this:

  1. Check out the code repository.
  2. Build the Boa test automation solution.
  3. Generate the SpecFlow+ LivingDoc report with unused steps but without test results.
  4. For each applicable bank configuration:
    1. Run appropriate Boa tests and get the test execution JSON file.
    2. Generate the feature data JSON file.
    3. Remove unexecuted scenarios from the feature data.
    4. Remove PrecisionLender-specific tags from the feature data.
    5. Generate the SpecFlow+ LivingDoc report using the modified feature data and the test results.

Below is an example of what the modified LivingDoc report looks like when we run our 12 smoke tests:

SpecFlow+ LivingDoc report using modified feature data after running only 12 smoke tests

(Note: At the time of writing this article, the most recent version of SpecFlow+ LivingDoc now includes a filter for test results in addition to its other filters. Using the test result filter, you can remove unexecuted scenarios from view. This feature is very helpful and could be used for our internal testing, but it would not meet our needs of removing sensitive data from reports for our customers.)

Conclusion

Ever since acquiring SpecFlow from TechTalk in January 2020, Tricentis has done great things to improve SpecFlow’s features and strengthen its community. SpecFlow+ LivingDoc is one of the many fruits of that effort. My team and I at PrecisionLender love these slick new reports, and we are already getting significant value out of them.

If you like SpecFlow+ LivingDoc, then I encourage you to check out some of SpecFlow’s other products. Everything SpecFlow offers is now free to use forever – you just need to register a free SpecFlow account. SpecFlow+ Runner is by far the best way to run SpecFlow tests (and, believe me, I’ve used the other runners for NUnit, xUnit.net, and MsTest). SpecMap is great for mapping and planning stories with Azure Boards. SpecFlow’s Online Gherkin Editor is also one of the best and simplest ways to write Gherkin without needing a full IDE.

Finally, if you use SpecFlow for test automation, give Boa Constrictor a try. Boa Constrictor is a .NET implementation of the Screenplay Pattern that my team and I developed at PrecisionLender. It helps you make better interactions for better automation, and it’s a significant step up from the Page Object Model. It’s now an open source project – all you need to do is install the Boa.Constrictor NuGet package! If you’re interested, be sure to check out the SpecFlow livestream in which Andi Willich and I teamed up to convert an existing SpecFlow project from page objects and drivers to Boa Constrictor’s Screenplay calls. SpecFlow and Boa Constrictor work together beautifully.

Using Domain-Specific Languages for Security Testing

I love programming languages. They have fascinated me ever since I first learned to program my TI-83 Plus calculator in ninth grade, many years ago. When I studied computer science in college, I learned how parsers, interpreters, and compilers work. During my internships at IBM, I worked on a language named Enterprise Generation Language as both a tester and a developer. At NetApp, I even developed my own language named DS for test automation. Languages are so much fun to learn, build, and extend.

Today, even though I do not actively work on compilers, I still do some pretty interesting things with languages and testing. I strongly advocate for Behavior-Driven Development and its domain-specific language (DSL) Gherkin. In fact, as I wrote in my article Behavior-Driven Blasphemy, I support using Gherkin-based BDD test frameworks for test automation even if a team is not also doing BDD’s collaborative activities. Why? Gherkin is the world’s first major off-the-shelf DSL for test automation, and it doesn’t require the average tester to know the complexities of compiler theory. DSLs like Gherkin can make tests easier to read, faster to write, and more reliable to run. They provide a healthy separation of concerns between test cases and test code. After working on successful large-scale test automation projects with C# and SpecFlow, I don’t think I could go back to traditional test frameworks.

I’m not the only one who thinks this way. Here’s a tweet from Dinis Cruz, CTO and CISO at Glasswall, after he read one of my articles:

Dinis then tweeted at me to invite me to speak about using DSLs for testing at the Open Security Summit in 2021:

Now, I’m not a “security guy” at all, but I do know a thing or two about DSLs and testing. So, I gladly accepted the invitation to speak! I delivered my talk, “Using DSLs for Security Testing” virtually on Thursday, January 14, 2021 at 10am US Eastern. I also uploaded my slides to GitHub at AndyLPK247/using-dsls-for-security-testing. Check out the YouTube recording here:

This talk was not meant to be a technical demo or tutorial. Instead, it was meant to be a “think big” proposal. The main question I raised was, “How can we use DSLs for security testing?” I used my own story to illustrate the value languages deliver, particularly for testing. My call to action breaks that question down into three parts:

  1. Can DSLs make security testing easier to do and thereby more widely practiced?
  2. Is Gherkin good enough for security testing, or do we need to make a DSL specific to security?
  3. Would it be possible to write a set of “standard” or “universal” security tests using a DSL that anyone could either run directly or use as a template?

My goal for this talk was to spark a conversation about DSLs and security testing. Immediately after my talk, Luis Saiz shared two projects he’s working on regarding DSLs and security: SUSTO and Mist. Dinis also invited me back for a session at the Open Source Summit Mini Summit in February to have a follow-up roundtable discussion for my talk. I can’t wait to explore this idea further. It’s an exciting new space for me.

If this topic sparks your interest, be sure to watch my talk recording, and then join us live in February 2021 for the next Open Source Summit event. Virtual sessions are free to join. Many thanks again to Dinis and the whole team behind Open Source Summit for inviting me to speak and organizing the events.