SpecFlow is an excellent Behavior-Driven Development test framework for .NET. Recently, SpecFlow released a new reporting tool called SpecFlow+ LivingDoc, which generates living documentation for features. It combines all scenarios from all SpecFlow feature files into one central HTML report. The report looks crisp and professional. It is filterable and can optionally show test results. Teams can generate updated reports as part of their Continuous Integration pipelines. The best part is that SpecFlow+ LivingDoc, along with all other features, is completely free to use – all you need to do is register for a free SpecFlow account. There is no reason for any SpecFlow project to not also use LivingDoc.

SpecFlow provides rich documentation on all of SpecFlow+ LivingDoc’s benefits, features, and configurations. In this article, I won’t simply repeat what the official docs already state. Instead, I’m going to share how my team and I at PrecisionLender, a Q2 Company, adopted SpecFlow+ LivingDoc into our test automation solution. I’ll start by giving a brief overview of how we test the PrecisionLender web app. Then, I’ll share why we wanted to make LivingDoc part of our quality workflow. Next, I’ll walk through how we added the new report to our testing pipelines. Finally, as an advanced technique, I’ll show how we modified some of the LivingDoc data files to customize our reports. My goal for this article is to demonstrate the value SpecFlow+ LivingDoc adds to BDD collaboration and automation practices.

PrecisionLender’s Test Automation

PrecisionLender is a web application that empowers commercial bankers with in-the-moment insights that help them structure and price commercial deals. Andi®, PrecisionLender’s intelligent virtual analyst, delivers these hyper-focused recommendations in real time allowing relationship managers to make data-driven decisions while pricing their commercial deals.

The PrecisionLender app is quite complex. It has several rich features to help bankers price any possible nuance for loan opportunities. Some banks also have unique configurations and additional features that make testing challenging.

On top of thorough unit testing, we run suites of end-to-end tests against the PrecisionLender web app. Our test automation solution is named “Boa,” and it is written in C# using SpecFlow for test cases and Boa Constrictor for Web UI and REST API interactions. We use BDD practices like Three Amigos, Example Mapping, and Good Gherkin to develop behaviors and cover them with automated tests. As of January 2021, Boa has over 1400 unique tests that target multiple test bank configurations. We run Boa tests continuously (for every code change), nightly (across all test banks), and “release-ly” (every two weeks before production deployments) at a rate of ~15K test iterations per week. Each test takes roughly half a minute to complete, and we run tests in parallel with up to 32 threads using Selenium Grid.

Introducing SpecFlow+ LivingDoc

SpecFlow+ LivingDoc is living documentation for features. SpecFlow started developing the tool a few years ago, but in recent months under Tricentis, they have significantly ramped up its development with the standalone generator and numerous feature enhancements. To learn about LivingDoc, watch this short introduction video:

When I saw the new SpecFlow+ LivingDoc reports, I couldn’t wait to try them myself. I love SpecFlow, and I’ve used it daily for the past few years. I knew it would bring value to my team at PrecisionLender.

Why Adopt SpecFlow+ LivingDoc?

My team and I wanted to bring SpecFlow+ LivingDoc into our testing workflow for a few reasons. First and foremost, we wanted to share our features with every team member, whether they were in business, development, or testing roles. I originally chose SpecFlow to be the core test framework for our Boa tests because I wanted to write all tests in plain-language Gherkin. That way, product owners and managers could read and understand our tests. We could foster better discussions about product behaviors, test coverage, and story planning. However, even though tests could be understood by anyone, we didn’t have an effective way to share them. Feature files for tests must be stored together with automation code in a repository. Folks must use Visual Studio or a version control tool like Git to view them. That’s fine for developers, but it’s inaccessible for folks who don’t code. SpecFlow+ LivingDoc breaks down that barrier. It combines all scenarios from all feature files into one consolidated HTML report. Folks could use a search bar to find the tests they need instead of plunging through directories of feature files. The report could be generated by Continuous Integration pipelines, published to a shared dashboard, or emailed directly to stakeholders. Pipelines could also update LivingDoc reports any time features change. SpecFlow+ LivingDoc would enable us to actually share our features instead of merely saying that we could.

Second, we liked the concise test reporting that SpecFlow+ LivingDoc offered. The SpecFlow+ Runner report, which our team already used, provides comprehensive information about test execution: full log messages, duration times, and a complete breakdown of pass-or-fail results by feature and scenario. That information is incredibly helpful when determining why tests fail, but it is too much information when reporting failures to managers. LivingDoc provides just the right amount of information for reporting high-level status: the tests, the results per test, and the pass-or-fail totals. Folks can see test status at a glance. The visuals look nice, too.

Third, we wanted to discover any unused step definitions in our C# automation code. The Boa test solution is a very large automation project. As of January 2020, it has over 1400 unique tests and over 1100 unique step definitions, and those numbers will increase as we continue to add new tests. Maintaining any project of this size is a challenge. Sometimes, when making changes to scenarios, old step definitions may no longer be needed, but testers may not think to remove them from the code. These unused step definitions then become “dead code” that bloats the repository. It’s easy to lose track of them. SpecFlow+ LivingDoc offers a special option to report unused step definitions on the Analytics tab. That way, the report can catch dead steps whenever they appear. When I generated the LivingDoc report for the Boa tests, I discovered over a hundred unused steps!

Fourth and finally, my team and I needed a test report that we could share with customers. At PrecisionLender, our customers are banks – and banks are very averse to risk. Some of our customers ask for our test reports so they can take confidence in the quality of our web app. When sharing any information with customers, we need to be careful about what we do share and what we don’t share. Internally, our Boa tests target multiple different system configurations, and we limit the test results we share with customers to the tests for the features they use. For example, if a bank doesn’t factor deposits into their pricing calculations, then that bank’s test report should not include any tests for deposits. The reports should also be high-level instead of granular: we want to share the tests, their scenarios, and their pass-or-fail results, but nothing more. SpecFlow+ LivingDoc fits this need perfectly. It provides Gherkin scenarios with their steps in a filterable tree, and it visually shows results for each test as well as in total. With just a little bit of data modification (as shown later in this article), the report can include exactly the intended scenarios. Our team could use LivingDoc instead of generating our own custom report for customers. LivingDoc would look better than any report we would try to make, too!

Setting Up SpecFlow+ LivingDoc

At PrecisionLender, we currently use JetBrains TeamCity to schedule and launch Boa tests. Some tests launch immediately after app deployments, while others are triggered based on the time of day. When a test pipeline is launched, it follows these steps:

1. Check out the code repository.
2. Build the Boa test automation solution.
3. For each applicable bank configuration, run appropriate Boa tests.

We wanted to add SpecFlow+ LivingDoc in two places: after the build completes and after tests run for each configuration. The LivingDoc generated for the build step would not include test results. It would show all scenarios in all features, and it would also include the unused step definitions. This report would be useful for showing folks our tests and our coverage. The LivingDoc generated for each test run, however, would include test results. Since we run tests against multiple configurations, each run would need its own LivingDoc report. Not all tests run on each configuration, too. Generating LivingDoc reports at each pipeline step serve different needs.

Adding SpecFlow+ LivingDoc to our testing pipelines required only a few things to set up. The first step was to add the SpecFlow.Plus.LivingDocPlugin NuGet package to the .NET test project. Adding this NuGet package makes SpecFlow automatically save test results to a file named TestExecution.json every time tests run. The docs say you can customize this output path using specflow.json, too.

The next step was to install the LivingDoc CLI tool on our TeamCity agents. The CLI tool is a dotnet command line tool, so you need the .NET Core SDK 3.1 or higher. Also, note that you cannot install this package as a NuGet dependency for your .NET test project. (I tried to do that in the hopes of simplifying my build configuration, but NuGet blocks it.) You must install it to your machine’s command line. The installation command looks like this:

dotnet tool install --global SpecFlow.Plus.LivingDoc.CLI

After installing the LivingDoc CLI tool, the final step was to invoke it after each build and each test run. There are three sources from which to generate LivingDoc reports:

1. Using a folder of feature files
2. Using a SpecFlow test assembly (.dll)
3. Using a feature data JSON file previously generated by the LivingDoc CLI tool

For generating LivingDoc after the build, I used this command in PowerShell to include unused steps but exclude test results:

livingdoc test-assembly "$TestAssemblyPath" --binding-assemblies "$TestAssemblyPath" --output-type HTML --output "$LivingDocDir\PLAppLivingDoc.html"

Then, for generating LivingDoc after test runs, I used this command in PowerShell that included TestExecution.json:

livingdoc test-assembly "$TestAssemblyPath" --test-execution-json "$TestExecutionPath" --output-type HTML --output "$HtmlReportPath" --title "PL App Boa Tests"

All the “$” variables are paths configured in our TeamCity projects. I chose to generate reports using the test assembly because I discovered that results wouldn’t appear in the report if I generated them from the feature folder.

Here’s what SpecFlow+ LivingDoc looks like when published as a TeamCity report:

Our team can view reports from TeamCity, or they can download them to view them locally.

Modifying SpecFlow+ LivingDoc Data

As I mentioned previously in this article, my team and I wanted to share SpecFlow+ LivingDoc reports with some of our customers. We just needed to tweak the contents of the report in two ways. First, we needed to remove scenarios that were inapplicable (meaning not executed) for the bank. Second, we needed to remove certain tags that we use internally at PrecisionLender. Scrubbing this data from the reports would give our customers what they need without including information that they shouldn’t see.

Thankfully, SpecFlow+ LivingDoc has a “backdoor” in its design that makes this kind of data modification easy. When generating a LivingDoc report, you can set the --output-type parameter to be “JSON” instead of “HTML” to generate a feature data JSON file. The feature data file contains all the data for the LivingDoc report in JSON format, including scenarios and tags. You can modify the data in this JSON file and then use it to generate an HTML LivingDoc report. Modifying JSON data is much simpler and cleaner than painfully splicing HTML text.

I wrote two PowerShell scripts to modify feature data. Both are available publicly in GitHub at AndyLPK247/SpecFlowPlusLivingDocScripts. You can copy them from the repository to use them for your project, and you can even enhance them with your own changes. Note that the feature data JSON files they use must be generated from test assemblies, not from feature data folders.

The first script is RemoveSkippedScenarios.ps1. It takes in both a feature data JSON file and a test execution JSON file, and it removes all scenarios from the feature data that did not have results in the test execution data. It uses recursive functions to traverse the feature data JSON “tree” of folders, features, and scenarios. Removing unexecuted scenarios means that the LivingDoc report will only include scenarios with test results – none of the scenarios in it should be “skipped.” For my team, this means a LivingDoc report for a particular bank configuration will not include a bunch of skipped tests for other banks. Even though we currently have over 1400 unique tests, any given bank configuration may run only 1000 of those tests. The extra 400 skipped tests would be noise at best and a data privacy violation at worst.

The second script is RemoveTags.ps1. It takes in a list of tags and a feature data JSON file, and it removes all appearances of those tags from every feature, scenario, and example table. Like the script for removing skipped scenarios, it uses recursive functions to traverse the feature data JSON “tree.” The tags must be given as literal names, but the script could easily be adjusted to handle wildcard patterns or regular expressions.

With these new scripts, our test pipelines now look like this:

1. Check out the code repository.
2. Build the Boa test automation solution.
3. Generate the SpecFlow+ LivingDoc report with unused steps but without test results.
4. For each applicable bank configuration:
   1. Run appropriate Boa tests and get the test execution JSON file.
   2. Generate the feature data JSON file.
   3. Remove unexecuted scenarios from the feature data.
   4. Remove PrecisionLender-specific tags from the feature data.
   5. Generate the SpecFlow+ LivingDoc report using the modified feature data and the test results.

Below is an example of what the modified LivingDoc report looks like when we run our 12 smoke tests:

(Note: At the time of writing this article, the most recent version of SpecFlow+ LivingDoc now includes a filter for test results in addition to its other filters. Using the test result filter, you can remove unexecuted scenarios from view. This feature is very helpful and could be used for our internal testing, but it would not meet our needs of removing sensitive data from reports for our customers.)

Conclusion

Ever since acquiring SpecFlow from TechTalk in January 2020, Tricentis has done great things to improve SpecFlow’s features and strengthen its community. SpecFlow+ LivingDoc is one of the many fruits of that effort. My team and I at PrecisionLender love these slick new reports, and we are already getting significant value out of them.

If you like SpecFlow+ LivingDoc, then I encourage you to check out some of SpecFlow’s other products. Everything SpecFlow offers is now free to use forever – you just need to register a free SpecFlow account. SpecFlow+ Runner is by far the best way to run SpecFlow tests (and, believe me, I’ve used the other runners for NUnit, xUnit.net, and MsTest). SpecMap is great for mapping and planning stories with Azure Boards. SpecFlow’s Online Gherkin Editor is also one of the best and simplest ways to write Gherkin without needing a full IDE.

Finally, if you use SpecFlow for test automation, give Boa Constrictor a try. Boa Constrictor is a .NET implementation of the Screenplay Pattern that my team and I developed at PrecisionLender. It helps you make better interactions for better automation, and it’s a significant step up from the Page Object Model. It’s now an open source project – all you need to do is install the Boa.Constrictor NuGet package! If you’re interested, be sure to check out the SpecFlow livestream in which Andi Willich and I teamed up to convert an existing SpecFlow project from page objects and drivers to Boa Constrictor’s Screenplay calls. SpecFlow and Boa Constrictor work together beautifully.