Open Testing: Opening tests like opening source

This article is based on a talk I gave on Open Testing at a few conferences: STARWEST 2021, TAU: The Homecoming, TSQA 2022, QA or the Highway 2022, and Conf42: SRE 2022.

I’m super excited to introduce a somewhat new idea to you and to our industry: Open Testing: What if we open our tests like we open our source? I’m not merely talking about creating open source test frameworks. I’m talking about opening the tests themselves. What if it became normal to share test cases and automated procedures? What if it became normal for companies to publicly share their test results? And what are the degrees of openness in testing for which we should strive as an industry?

I think that we – whether we are testers, developers, managers, or any other role in software – can greatly improve the quality of our work if we adopt principles of openness into our testing practices. To help me explain, I’d like to share how I learned about the main benefits of open source software, and then we can cross those benefits over into testing work.

So, let’s go way back in time to when I first encountered open source software.

My first encounter with open source code

I first started programming when I was in high school. At 13 years old, I was an incoming freshman at Parkville High School in their magnet school for math, science, and computer science in good old Baltimore, Maryland. (Fun fact: Parkville’s mascots were the Knights, which is my last name!) All students in the magnet program needed to have a TI-83 Plus graphing calculator. Now, mind you, this was back in the day before smart phones existed. Flip phones were the cool trend! The TI-83 Plus was cutting-edge handheld technology at that time. It was so advanced that when I first got it, it took me 5 minutes to figure out how to turn it off!

The TI-83 Plus

I quickly learned that the TI-83 Plus was just a mini-computer in disguise. Did you know that this thing has a full programming language built into it? TI-BASIC! Within the first two weeks of my freshman Intro to Computer Science class, our teacher taught us how to program math formulas: Slope. Circle circumference and area. The quadratic formula. You name it, I programmed it, even if it wasn’t a homework assignment. It felt awesome! It was more fun to me than playing video games, and believe me, I was a huge Nintendo fan.

There were two extra features of the TI-83 Plus that made it ideal for programming. First, it had a link cable for sharing programs. Two people could connect their calculators and copy programs from one to the other. Needless to say, with all my formulas, I became quite popular around test time. Second, anyone could open any program file on the calculator and read its code. The TI-BASIC source code could not be hidden. By design, it was “open source.”

This is how I learned my very first lesson about open source software: Open source helps me learn. Whenever I would copy programs from others, including games, I would open the program and read the code to see how it worked. Sometimes, I would make changes to improve it. More importantly, though, many times, I would learn something new that would help me write better programs. This is how I taught myself to code. All on this tiny screen. All through ripping open other people’s code and learning it. All because the code was open to me.

From the moment I wrote my first calculator program, I knew I wanted to become a software engineer. I had that spark.

My first open source library

Let’s fast-forward to college. I entered the Computer Science program at Rochester Institute of Technology – Go Tigers! By my freshman year in college, I had learned Java, C++, a little Python, and, of all things, COBOL. All the code in all my projects until that point had been written entirely by me. Sometimes, I would look at examples in books as a guide, but I’d never use other people’s code. In fact, if a professor caught you using copied code, then you’d fail that assignment and risk being expelled from the school.

Then, in my first software engineering course, we learned how to write unit tests using a library called JUnit. We downloaded JUnit from somewhere online – this was before Maven became big – and hooked it into our Java path. Then, we started writing test classes with test case methods, and somehow, it all ran magically in ways I couldn’t figure out at the time.

I was astounded that I could use software that I didn’t write myself in a project. Permission from a professor was one thing, but the fact that someone out there in the world was giving away good code for free just blew my mind. I saw the value in unit tests, and I immediately saw the value in a simple, free test framework like JUnit.

That’s when I learned my second lesson about open source software: Open source helps me become a better developer. I could have written my own test framework, but that would have taken me a lot of time. JUnit was ready to go and free to use. Plus, since several individuals had already spent years developing JUnit, it would have more features and fewer bugs than anything I could develop on my own for a college project. Using a package like JUnit helped me write and run my unit tests without needing to become an expert in test automation frameworks. I could build cool things without needing to build every single component.

That revelation felt empowering. Within a few years of taking that software engineering course, sites for hosting open source projects like GitHub became huge. Programming language package indexes like Maven, NuGet, PyPI, and NPM became development mainstays. The running joke within Python became that you could import anything! This was way better than swapping calculator games with link cables.

My first chance to give back

When I graduated college, I was zealous for open source software. I believed in it. I was an ardent supporter. But, I was mostly a consumer. As a Software Engineer in Test, I used many major test tools and frameworks: JUnit, TestNG, Cucumber, NUnit, xUnit.net, SpecFlow, pytest, Jasmine, Mocha, Selenium WebDriver, RestSharp, Rest Assured – the list goes on and on. As a Python developer, I used many modules and frameworks in the Python ecosystem like Django, Flask, and requests.

Then, I got the chance to give back: I launched an open source project called Boa Constrictor. Boa Constrictor is a .NET implementation of the Screenplay Pattern. It helps you make better interactions for better automation. Out of the box, it provides Web UI interactions using Selenium WebDriver and Rest API interactions using RestSharp, but you can use it to implement any interactions you want.

My company and I released Boa Constrictor publicly in October 2020. You can check out the boa-constrictor repository on GitHub. Originally, my team and I at Q2 developed all the code. We released it as an open source project hoping that it could help others in the industry. But then, something cool happened: folks in the industry helped us! We started receiving pull requests for new features. In fact, we even started using some new interactions developed by community members internally in our company’s test automation project. We also proudly participated in Hacktoberfest in 2020 and 2021.

Boa Constrictor: The .NET Screenplay Pattern

That’s when I learned my third lesson about open source software: Open source helps me become a better maintainer. Large projects need all the help they can get. Even a team of core maintainers can’t always handle all the work. However, when a project is open source, anyone who uses it can help out. Each little contribution can add value for the whole user base. Maintaining software then becomes easier, and the project can become more impactful.

Struggling with poor quality

As a Software Engineer in Test, I found myself caught between two worlds. In one world, I was a developer at heart who loved to write code to solve problems. In the other world, I was a software quality professional who tested software and advocated for improvements. These worlds came together primarily through test automation and continuous integration. Now that I’m a developer advocate, I still occupy this intersectionality with a greater responsibility for helping others.

However, throughout my entire career, I keep hitting one major problem: Software quality has a problem with quality. Let that sink in: software quality has a big problem with quality. I’ve worked on teams with titles ranging from “Software Quality Assurance” to “Test Engineering & Architecture,” and even an “Automation Center of Excellence.” Despite the titular focus on quality, every team has suffered from aspects of poor quality in workmanship.

Here are a few poignant examples:

  • Manual test case repositories are full of tests with redundant steps.
  • Test automation projects are riddled with duplicate code.
  • Setup and cleanup steps are copy-pasted endlessly, whether needed or not.
  • Automation code uses poor practices, such as global variables instead of dependency injection.
  • A 90% success rate is treated as a “good” day with “limited” flakiness.
  • Many tests cover silly, pointless, or unimportant things instead of valuable, meaningful behaviors.

How can we call ourselves quality professionals when our own work suffers from poor quality? Why are these kinds of problems so pervasive? I think they build up over time. Copy-pasting one procedure feels innocuous. One rogue variable won’t be noticed. One flaky test is no big deal. Once this starts happening, teams insularly keep repeating these practices until they make a mess. I don’t think giving teams more time to work on these problems will solve them, either, because more time does not interrupt inertia – it merely prolongs it.

The developer in me desperately wants to solve these problems. But how? I can do it in my own projects, but because my tests are sealed behind company doors, I can’t use it to show others how to do it at scale. Many of the articles and courses we have on how-to-do-X are full of toy examples, too.

Changing our quality culture

So, how do we get teams to break bad habits? I think our industry needs a culture change. If we could be more open with testing like we are open with source code, then perhaps we could bring many of the benefits we see from open source into testing:

  1. Helping people learn testing
  2. Helping people become better testers
  3. Helping people become better test maintainers

If we cultivate a culture of openness, then we could lead better practices by example. Furthermore, if we become transparent about our quality, it could bolster our users’ confidence in our products while simultaneously keeping us motivated to keep quality high.

There are multiple ways to start pursuing this idea of open testing. Not every possibility may be applicable for every circumstance, but my goal is to get y’all thinking about it. Hopefully, these ideas can inspire better practices for better quality.

Openness through internal collaboration

For a starting point of reference, let’s consider the least open context for testing. Imagine a team where testing work is entirely siloed by role. In this type of team, there is a harsh line between developers and testers. Only the testers ever see test cases, access test repositories, or touch automation. Test cases and test plans are essentially “closed” to non-testers due to access, readability, or even apathy. The only output from testers are failure percentages and bug reports. Results are based more on trust than on evidence.

This kind of team sounds pretty bleak. I hope this isn’t the kind of team you’re on, but maybe it is. Let’s see how openness can make things better.

The first step towards open testing is internal openness. Let’s break down some siloes. Testers don’t exclusively own quality. Not everyone needs to be a tester by title, but everyone on the team should become quality-conscious. In fact, any software development team has three main roles: Business, Development, and Testing. Business looks for what problems to solve, Development addresses how to implement solutions, and Testing provides feedback on the solution. These three roles together are known as “The Three Amigos” or “The Three Hats.”

Each role offers a valuable perspective with unique expertise. When the Three Amigos stay apart, features under development don’t have the benefit of multiple perspectives. They might have serious design flaws, they might be unreasonable to implement, or they might be difficult to test. Misunderstandings could also cause developers to build the wrong things or testers to write useless tests. However, when the Three Amigos get together, they can jointly contribute to the design of product features. Everyone can get on the same page. The team can build quality into the product from the start. They could do activities like Question Storming and Example Mapping to help them define behaviors.

The Three Amigos

As part of this collaboration, not everyone may end up writing tests, but everyone will be thinking about quality. Testing then becomes easier because expected behaviors are well-defined and well-understood. Testers get deeper insight into what is important to cover. When testers share results and open bugs, other team members are more receptive because the feedback is more meaningful and more valuable.

We practiced Three Amigos collaboration at my previous company, Q2. My friend Steve was a developer who saw the value in Example Mapping. Many times, he’d pick up poorly-defined user stories with conflicting information or missing acceptance criteria. Sometimes, he’d burn a whole sprint just trying to figure things out! Once he learned about Example Mapping, he started setting up half-hour sessions with the other two Amigos (one of whom was me) to better understand user stories from the start. He got into it. Thanks to proactive collaboration, he could develop the stories more smoothly. One time, I remember we stopped working on a story because we couldn’t justify its business value, which saved Steve two weeks of pointless work. The story didn’t end there: Steve became a Software Engineer in Test! He shifted left so hard that he shifted into a whole new role.

Openness through living specs

Another step towards open testing is living documentation through specification by example. Collaboration like we saw with the Three Amigos is great, but the value it provides can be fleeting if it is not written down. Teams need artifacts to record designs, examples, and eventually test cases.

One reason why I love Example Mapping is because it facilitates a team to spell out stories, rules, examples, and questions onto color-coded cards that they can keep for future refinement.

  1. Stories become work items.
  2. Rules become acceptance criteria.
  3. Examples become test cases.
  4. Questions become spikes or future stories.

During Example Mapping, folks typically write cards quickly. An example card describes a behavior to test, but it might not carefully design the scenario. It needs further refinement. Defining behaviors using a clear, concise format like Given-When-Then makes behaviors easy to understand and easy to test.

For example, let’s say we wanted to test a web search engine. The example could be to search for a phrase like”panda”. We could write this example as the following scenario:

  1. Given the search engine page is displayed
  2. When the user searches for the phrase “panda”
  3. Then the results page shows a list of links for “panda”

This special Given-When-Then format is known as the Gherkin language. Gherkin comes from Behavior-Driven Development tools like Cucumber, but it can be helpful for any type of testing. Gherkin defines testable behaviors in a concise way that follows the Arrange-Act-Assert pattern. You set things up, you interact with the feature, and you verify the outcomes.

Furthermore, Gherkin encourages Specification by Example. This scenario provides clear instructions on how to perform a search. It has real data, which is the search phrase “panda,” and clear results. Using real-world examples in specifications like this helps all Three Amigos understand the precise behavior.

Turning Example Mapping cards into Gherkin behavior specs

Behavior specifications are multifaceted artifacts:

  1. They are requirements that define how a feature should behave.
  2. They are acceptance criteria that must be met for a deliverable to be complete.
  3. They are test cases with clear instructions.
  4. They could become automated scripts with the right kind of test framework.
  5. They are living documentation for the product.

Living documentation is open and powerful. Anyone on the team or outside the team can read it to learn about the product. Refining ideas into example cards into behavior specs becomes a pipeline that delivers living doc as a byproduct of the software development lifecycle.

SpecFlow is one of the best frameworks that supports this type of openness with Specification by Example and Living Documentation. SpecFlow is a free and open-source test automation framework for .NET. In SpecFlow, you write your test cases as Gherkin scenarios, and you automate each Given-When-Then step using C# methods.

One of SpecFlow’s niftiest features, however, is SpecFlow+ LivingDoc. Most test frameworks focus exclusively on automation code. When a test is automated, then only a programmer can read it and understand it. Gherkin makes this easier because steps are written in plain language, but Gherkin scenarios are nevertheless stored in a code repository that’s inaccessible to many team members. SpecFlow+ LivingDoc breaks that pattern. It turns Gherkin scenarios into a searchable doc site accessible to all Three Amigos. It makes test cases and test automation much more open. LivingDoc also provides test results for each scenario. Green check marks indicate passing tests, while red X’s indicate failures.

Historically, testers use reports like this to provide feedback in-house to their managers and developers. Results indicate what works and what needs to be fixed. However, test results can be useful to more people than just internal team members. What if test results were shared with users and customers? I’m going to repeat that statement, because it might seem shocking: What if users and customers could see test results? 

Think about it. Open test results have very positive effects. Transparency with users builds trust. If users can see that things are tested and working, then they will gain confidence in the quality of the product. If they could peer into the living documentation, then they could learn how to use the product even better. On the flip side, transparency holds development teams accountable to keeping quality high, both in the product and in the testing. Open test results offer these benefits only if the results can be trusted. If tests are useless or failures are rampant, then public test results could actually hurt the ones developing the product.

A SpecFlow+ LivingDoc report

This type of radical transparency would require an enormous culture shift. It may not be appropriate for every company to create public dashboards with their test results, but it could be a strategic differentiator when used wisely. For example, when I worked at Q2, we shared LivingDoc reports with specific PrecisionLender customers after every two-week release. It built trust. Plus, since the LivingDoc report includes only high-level behavior specs with simple results, even a vice president could read it! We could share tests without sharing automation code. That was powerful.

Openness through open source

Let’s keep extending open testing outward. In addition to sharing test results and living documentation, folks can also share tools, frameworks, and other parts of their tests. This is where open testing truly is open source.

We already covered a bunch of open source projects for test automation. As an industry, we are truly blessed with so many incredible projects. Every single one of them represents a team of testers who not only solved a problem but decided to share their solution with the world. Each solution is abstract enough to apply to many circumstances but concrete enough to provide a helpful implementation. Collectively, the projects on this page have probably been downloaded more than a billion times, and that’s no joke. And if you want, you could read the open source code for any of them.

Popular open source test automation projects

Cool new projects appear all the time, too. One of my favorite projects that started in the past few years is Playwright, an awesome browser automation tool from Microsoft. Playwright makes end-to-end web testing easy, reliable, and fast. It provides cross-browser and cross-language support like Selenium, a concise syntax like Cypress, and a bunch of advanced features like automatic waiting, tracing, and code generation. Plus, Playwright is magnitudes faster than other automation tools. It took things that made Selenium, Cypress, and Puppeteer great, and it took them to the next level.

Openness through shared test suites

So far, all the ways of approaching open testing are things we could do today. Many of us are probably already doing these things, even if we didn’t think of them under the phrase “open testing.” But where can these ideas go in the future?

My mind goes back to one of the big problems with testing that I mentioned earlier: duplication. Opening up collaboration fixes some bad habits, and sharing components eliminates some duplication in the plumbing of test automation, but so many of our tests across the industry repeat the same kinds of steps and follow the same types of patterns.

For example, think about any time you’ve ordered something from an online store. It could be Amazon, Walmart, Target – whatever. Every single online store has a virtual shopping cart. Whenever you want to buy something, you add it to your cart. Then, when you’re done shopping, you proceed to pay for all the items in your cart. If you decide you don’t want something anymore, you remove it from the cart. Easy-peasy.

As I describe this type of shopping cart, I don’t need to show you screenshots from the store website to explain it. Y’all have done so much online shopping that you intuitively know how it works, regardless of the store. Heck, I recently ordered a bunch of parts for an old Volkswagen Beetle from a site named JBugs, and the shopping cart was the same.

If so many applications have the same parts, then why do we keep duplicating the same tests in different places? Think about it. Think about how many times different teams have written nearly identical shopping cart tests. Ouch. Think about how much time was wasted on that duplication of effort.

I think this is something where Artificial Intelligence and Machine Learning could help. What if we could develop machine learning models to learn common behaviors for apps and services? The learning agents would look for things like standard icons and typical workflows. We could essentially create test suites for things like login, search, shopping, and payment that could run successfully on most apps. These kinds of tests probably couldn’t cover everything in any given application, but they could cover basic, common behaviors. Maybe that could cover a quarter of all behaviors worth testing. Maybe a third? Maybe half? Every little bit helps!

AI and ML can help us achieve true Autonomous Testing

Now, imagine sharing those generic test suites publicly. In the same way developers have open source projects to help expedite their coding, and in the same way data scientists have open data sets to use for modeling, testers could have open test suites that they could pick up and run as applicable. Not test tools – but actual runnable tests that could run against any application. If these kinds of test suites prove to be valuable, then prominent ones could become universally-accepted bars of quality for software apps. For example, in the future, companies could download and execute tests that run on any system for the apps they’re developing in addition to the tests they develop in-house. I think that could be a really cool opportunity.

This type of testing – Autonomous Testing – is the future. Software developer and testers will use AI-backed tools to better learn, explore, and exercise app behaviors. These tools will make it easier than ever to automate scriptable tests.

How to start pursuing openness

As we have covered, open testing could take many forms:

  1. It could be openness in collaboration to build better quality from the start.
  2. It could be openness in specification by example and living documentation.
  3. It could be openness in sharing tests and their results with customers and users.
  4. It could be openness in sharing tools, frameworks, and platforms.
  5. It could be openness in building shared test sets for common application behaviors.

Some of these ideas might seem far-fetched or aspirational, but quite honestly, I think each of them could add lots of value to testing practices. I think every tester and every team should look at this list and ask themselves, “Could we try some of these things?” Perhaps your team could take baby steps with better collaboration or better specification. Perhaps your team has a cool project you built in-house that you could release as an open source project, like my old team and I did with Boa Constrictor. Perhaps there’s a startup idea in using machine learning for autonomous testing. Perhaps there are other ways to achieve open testing that aren’t listed here. Who knows? It could be cool!

We should also consider the flip side. Are there certain aspects of testing that should remain closed? My mind goes to security. Could fully open testing inadvertently reveal security vulnerabilities? Could lack of coverage in some areas welcome expedited exploitation? I don’t know, but I think we should consider possibilities like these.

If you want to pursue open testing, here are three questions to get you started:

  1. How is your testing today?
    1. In what ways is it already open?
    2. In what ways is it closed?
  2. How could your testing improve with incremental openness?
    1. We’re talking baby steps here – small improvements that you could easily achieve today.
    2. It could be as small as trying Example Mapping or joining a mob programming session.
  3. How could your testing improve with radical openness?
    1. Shoot the moon! Dream big! Get creative!
    2. In the world of software, anything is possible.

Conclusion

We should also remember that open testing isn’t a goal unto itself. It’s a means to an end, and that end is higher quality: quality in our practices, quality in our artifacts, and ultimately quality in the software we create. We shouldn’t seek openness in testing just because I’m spouting lots of buzzwords in this article. At the same time, we also shouldn’t brush off these ideas as too radical or idealistic. What we should do is seek ways for perpetual improvement. Remember that this whole idea of open testing came from the tried-and-true benefits of open source code.

DjangoCon 2019 Break

24 Lessons from Working in Software

Let’s face it: working in software is tough. I’ve worked in software for over a decade, and I’ve learned a lot of things during my journey – oftentimes the hard way.

On April 7, 2022, I tried something new: every hour for 24 hours, I tweeted a lesson I learned during my software career. I covered topics like breaking into the industry, working on projects, and seeking advancement. I used the hashtag #PandyEveryHour to track my tweets. I started at 8am (US Eastern) on April 7, and I concluded at 8am on April 8. It was an awesome experience: lots of folks liked, retweeted, and commented. Some of my insights struck nerves.

In case you missed it while my tweets went live, I compiled all the tweets below in order. Give them a read! If you like any of them in particular, be sure to “like” them on Twitter so I know!

Breaking into the software industry

Coding

Working with others

Improving your skills

Career advancement

Conclusion

So, which lessons resonated the most with you? Which did you disagree with? Let me know!

The main photo for this article was taken at DjangoCon 2019 during one of the breaks between sessions.

Want to practice test automation? Try these demo sites!

One of the biggest struggles in learning how to develop top-notch automation is practice. Testing is as much an art as it is a science. It takes time to discern when to add explicit waits, how to craft robust locators, and why to verify one element over another. It also requires apps with specific elements or endpoints to try certain operations. Unfortunately, although resources for learning how to automate tests (like Test Automation University) are abundant these days, public demo web sites for practicing remain elusive. I’ve struggled to find ones that I like, and others frequently ask me for recommendations.

Which demo sites are good?

Below is a list of demo sites I’ve found either by searching online or through recommendations from friends. Many of these sites appear on other “Top N Demo Sites for Testing” articles as well. My list aims not only to provide links to popular demo sites but also to provide recommendations on how to use them.

Types of sites:

  • Web UI site – looks like a real web site
  • Web UI elements – tutorial pages showcasing web element types
  • Mobile UI site – looks like a real mobile site
  • API site – provides public APIs for testing
  • DIY – “do it yourself”; you must set up and run the demo site yourself
Demo SiteTypeDescription
ParaBankWeb UI & API siteAn online banking site from Parasoft with login and REST/SOAP APIs. You can also access the database if you run the project locally using the source code.
Restful BookerWeb UI & API siteAn online site for bed & breakfast bookings from Mark Winteringham. The frontend is a React app (source), and the backend is a REST API (source).
Automation Practice WebsiteWeb UI siteA basic online store with optional login from SeleniumFramework.com. Great for example web UI tests.
DemoblazeWeb UI siteA basic online store with optional login from BlazeMeter. Great for example web UI tests.
Swag LabsWeb UI siteA basic online store with required login from Sauce Labs. Great for example web UI tests.
Applitools demo siteWeb UI siteA small site with login page and home page from Applitools. Compare/contrast with a second version for visual testing.
Automation BookstoreWeb UI siteA one-page site for dynamically searching book titles. Good for testing responsive design in short demos.
JPetStore DemoWeb UI siteA pet store site from MyBatis built on top of MyBatis 3, Spring 3, and Stripes (source).
GlobalSQA Banking ProjectWeb UI siteA basic banking app with dropdown login and simple pages from GlobalSQA.
Gatling Computers DatabaseWeb UI siteA one-page site from Gatling that provides a paginated list of computer models with the ability to filter and add new computers.
CandyMapperWeb UI siteA Halloween-themed site from Paul Grossman that shows scary bugs. It also comes with a second release that has the fixes.
OWASP Juice ShopDIY Web UI siteA test site written entirely in JavaScript using Node.js, Express and Angular. Specifically built for testing security vulnerabilities.
Cypress Real-World AppDIY Web UI siteA fake payment app from Cypress meant for demonstrating real-world Cypress testing. Could be used for other purposes.
RealWorld example appsDIY Web UI siteOne demo app implemented in multiple languages and frameworks. Not developed for testing but could nevertheless be used.
the-internetWeb UI elementsA site from Dave Haeffner and Elemental Selenium with several concise examples of web elements and interactions.
Selenium Test PagesWeb UI elementsA site with several pages that have slightly deeper examples than the-internet.
LetCodeWeb UI elementsA set of very clean pages along with video tutorials explaining how to automate interactions.
DemoQAWeb UI elementsA practice site from ToolsQA that includes pages for elements, forms, frames, interactions, and even a small bookstore.
Ultimate QA Automation PracticeWeb UI elementsA set of rich practice pages from Ultimate QA.
UI Test Automation PlaygroundWeb UI elementsA set of educational pages with interactable elements from the Rapise team.
SelectorsHub Practice PageWeb UI elementsA practice page from SelectorsHub for interacting with different kinds of web elements.
WebDriverUniversity.comWeb UI elementsAnother set of educational pages with interactable elements.
Sauce Labs Native Sample ApplicationDIY Mobile UI siteA mobile app from Sauce Labs for Android and iOS that is very similar to the Swag Labs web site.
JSONPlaceholderAPI siteA public REST API for generating fake data from a set of predefined resources. Follow the guide to learn how to make requests.
Swagger PetstoreAPI siteA public REST API from Swagger for testing authentication and CRUD operations for pet store data.
Public APIsAPI siteA long list of public APIs that can be used for testing.
Device Registry ServiceDIY API siteA Flask app I developed for teaching how to test REST APIs. Includes the app and automated API tests, which can both be run locally.
Best Buy API PlaygroundDIY API siteA JavaScript-based REST API demo service from Best Buy.

Which ones do you recommend most?

There are lots of demo sites above. Here are the ones I’d personally recommend for different needs:

Of course, the others are still good. That’s why they made the list!

Why bother with “fake” sites?

You might be wondering, “Why do we need demo sites? Why don’t we just use real sites?” Well, demo or “fake” web sites meet a few needs that real sites cannot:

  • Demo sites offer consistency. They are implemented one way and then do not change. Folks can trust that their tests will always work on them.
  • Demo sites are often simpler than real sites. They feel less intimidating for newcomers.
  • Demo sites can be designed for instruction. If they are part of a tutorial, the author can add features to the site to demonstrate concepts.
  • Demo sites are safer for publications like articles, tutorials, and books. Written content is static, so any sites referenced by examples should also be static. Real sites could change.
  • Real sites might require end user agreements that forbid automated requests. Some may even throttle or block requests if they suspect the requests come from a “bot.”
  • Real sites might also bear legal or copyright implications, especially if a company is using the sites for their own content.

What limitations do demo sites have?

Unfortunately, demo sites do have limitations:

  • Demo sites may be too simplified. They may lack large workflows or real-world data. Folks might become frustrated when they discover inactive elements that appear to be real.
  • Demo sites may not be built to scale. High request volume or parallel testing scale might cripple them.
  • Demo sites may seem like they have poor quality, whether or not they actually have poor quality. Sometimes, they are built quickly for testing purposes and therefore don’t have the attention to detail as real sites.
  • Demo sites with significant company branding may be inappropriate to use. For example, if A and B are competitors, then A shouldn’t use B’s demo site for their product tutorials.

Other great lists

Here are other great articles that list demo sites for testing. I referred to them while compiling this list:

Do you have other demo sites to recommend? Put them in the comments below!

Welcome to Raleigh

Thoughts on Remote Work

At my new job as Developer Advocate at Applitools, I now permanently work remotely. Working from home is new to me. At my previous company PrecisionLender (later acquired by Q2), I worked in an office that was a seven-minute drive from my house. My office was a real office: an enclosed room with a door, walls, a large desk, and two huge whiteboards. Working from the office was a joy. We had free snacks and sodas. My colleagues and I all got along great, and we’d go out to lunch almost every day. After COVID-19 hit the USA in March 2020, I was heartbroken to abandon the office. I was so stubborn about eventually returning to the office that I refused to set up a decent workstation at home. For a year and a half, I worked exclusively on my laptop either from a desk in my living room or from the couch – no monitors, no mouse, and no separate keyboard. I mean, come on, the pandemic would be over in a few short months, right?

However, my attitudes about remote work are changing. At PrecisionLender, I was spoiled with the best kind of office situation someone could have. Many others in the software field are not so fortunate. Those who live in the big cities like New York and San Francisco pay thousands of dollars a month to split tiny apartments and spend an hour or more on one-way commutes. Not to mention, many of them move away from family and friends to take those opportunities.

Office layout trends in recent years have also been dismal. The open layout – no walls with clusters of desks sloppily smashed together – is the norm. Perpetual audio and visual distractions are a nightmare for anyone trying to concentrate. Widescreen monitors and noise-cancelling headphones only help a little. I experienced that frustration at MaxPoint and LexisNexis. Cubicles, while passé these days, are only a small step up from open floors. While the “walls” block out visual distractions, you can still hear everything that goes on around you. Plus, the enormous blocks of cubicles on a floor feel brutally confining. I frequently called the IBM 500 complex in RTP “Cubicle City” when I worked there.

Remote work can be a godsend for many hardworking folks who just want to do a good job while doing right by themselves. Folks, especially in software, can literally work anywhere. COVID-19 shutdowns have proven it. My wife and I even took advantage of it: we bought a vacation condo in Seattle, and I worked remotely from there for several weeks. I could automate tests during the day, grab beers with close friends, and catch a Mariners game on the weekend. Even with a 3-hour time zone difference, I was right in step with my team the whole time.

The rooftop of my Seattle condo building
My Seattle condo building’s rooftop

Recently, I saw a Twitter thread by Chris Herd in which he made several predictions on remote work in 2022 and beyond:

This thread really resonated with me. While I still have some concerns about remote work, I do believe it will be a net good for not only my country and people but also for the world. I’d like to highlight some of the points he made that really stuck out to me.

I agree completely with this, and I’m excited to see it happen. The US has so many awesome places, from sea to shining sea. There’s no reason why someone should need to move from their hometown to a big city just to work for a specific company (unless, of course, they want that adventure). Places like Raleigh-Durham, Nashville, Denver, Miami, Pittsburgh, Salt Lake City, Cleveland, and countless other smaller American cities are fantastic places to live. Each has a unique vibe and local specialties. Allowing folks to live where they want with good employment opportunities will rejuvenate local and regional economies. It will bring broader prosperity to all.

This is happening already. The Triangle of NC, where I live, is booming. My wife and I rent local listings on Airbnb, and all through the pandemic, the #1 type of guest we’ve hosted by far is someone interested in relocating to the Triangle.

I hadn’t thought about asynchronous work until Chris mentioned it, but he’s right. Life is 24/7, and people live all around the globe. Remote work can liberate us from a harsh 9-5 and instead embrace more flexible hours. For example, this past week, I had to work around the clock for a few days prepping a big webinar. However, I wasn’t working for 16 hours straight each day. I woke up a bit late, did some light administrative activities before lunch, hit it hard all afternoon, stopped for dinner and time with my puppy, and then burned the midnight oil after dark. That’s what worked for me.

Yes! I’m living it myself. My wife and I just adopted a French bulldog puppy, and I also started restoring a vintage 1970 Volkswagen Beetle.

VW Beetle Engine Swap
I swapped the engine out of my vintage 1970 Volkswagen Beetle
Suki the Frenchie
Suki the French bulldog is the cutest puppy in the world!

I sure hope so! Break down those entry barriers. Do away with the gatekeepers. End coastal elitism.

I sure hope this one becomes true, but I’m skeptical.

This one is so ironic. Based on many articles I’ve read, upper management seems to despise work-from-home because they think their workers will work less. However, all the good, hardworking folks I know are actually working more than ever. Not only are folks more efficient, but they tend to keep working just a little extra because there are fewer barriers between work stuff and personal stuff. In my new job, I know I’m certainly working more hours right now than I probably should, and I’m hoping to settle into a more sustainable routine as I acclimate to my new role. Nevertheless, we absolutely need to protect ourselves against unintentional burnout.

One big concern I have with remote work is the loss of in-person contact. Humans need to be around each other, and screens are a poor substitute. I know I’ve felt lonelier working from home than when I worked in an office. That’s why I absolutely love this idea about regular remote retreats. Companies can still bring folks together, and they can hire professionals who know how to maximize that time. I go to software conferences frequently for similar experiences.

I think this goes hand-in-hand with async work. However, unless safeguards are put in place to prevent overworking, then life-work balance could quickly become work-more-work imbalance.

Honestly, this one was never much of a problem for me. The only “paddings” I’ve had in my workdays are long, boring meetings where I participated little. Thankfully, those were confined mostly to my days at IBM and NetApp earlier in my career.

YES! Even though my commutes have always been relatively short, I still appreciate the time I get back in my day. Here are some things I’ve done with my extra, more flexible time:

  • Sleep in a little later
  • Play with my puppy when I need a break
  • Share lunch nearby with friends
  • Run a quick errand in my Beetle
  • Grab an impromptu bubble tea with my wife
  • Take a walk around my neighborhood while thinking about work-related challenges

Documentation is the unspoken superpower of remote teams. I strongly value written communication for its clarity, searchability, and permanence.

Chris had lots of other good tweets in that thread, so be sure to read them. Overall, his tone was bullishly optimistic on remote work – not only in its benefits, but also in its inevitability. Personally, I hope remote work is here to stay, and that it’s not just a phase brought on by the COVID-19 pandemic. It’s a game-changer for individuals and industries alike. However, I’m still somewhat fearful that big companies will reign in wide scale returns to the office after the pandemic. Giants have already sunk millions to billions in commercial real estate for their offices that they can’t get back. I’m also worried about salaries in different geographies. Employers already pay folks who live outside of big cities much less for the same work due to cost-of-living differences. I don’t think that will be fair as people leave big cities in droves for smaller cities or the countryside. Already, the cost of living is skyrocketing in the Triangle, yet our average tech salaries are far below those of Silicon Valley. We’ll see what happens.

Overall, I am optimistic about remote work just like Chris.

So, what do y’all think? Please leave comments below. I’d love to learn different perspectives!

Playwright Workshop for TAU: The Homecoming

Want to learn Playwright with Python? Take this workshop!

Playwright is an awesome new browser automation library. With Playwright, you can automate web UI interactions for testing or for web scraping with a concise, uniform API in one of four languages: Python, C#, Java, and JavaScript. Playwright is also completely open source and backed by Microsoft. It’s a powerful alternative to Selenium WebDriver.

On December 1, 2021, I delivered a workshop on Playwright for TAU: The Homecoming. In my workshop, I taught how to build a test automation project in Python using Playwright with pytest, Python’s most popular test framework. We automated a test case together for performing a DuckDuckGo web search.

If you missed the workshop, no worries: You can still take the workshop as a self-guided tutorial! The workshop instructions and example code are located in this GitHub repository:

https://github.com/AutomationPanda/tau-playwright-workshop

To take the workshop as a self-guided tutorial, read the repository’s README, and then follow the instructions in the Markdown guides under the workshop folder. The workshop has five main parts:

  1. Getting started
    1. What is Playwright?
    2. Our web search test
    3. Test project setup
  2. First steps with Playwright
    1. Browsers, contexts, and pages
    2. Navigating to a web page
    3. Performing a search
  3. Writing assertions
    1. Checking the search field
    2. Checking the result links
    3. Checking the title
  4. Refactoring using page objects
    1. The search page
    2. The result page
    3. Page object fixtures
  5. Nifty Playwright tricks
    1. Testing different browsers
    2. Capturing screenshots and videos
    3. Running tests in parallel

If you get stuck or have any questions, please open issues against the GitHub repository, and I’ll try to help. Happy coding!

A Simple High-Quality Mic Setup for Software Pros

I’ve always been frustrated with poor quality audio recordings. Microphones built into laptops are very convenient, but they usually yield tinny sound lacking the depth of real voices. When I asked my audiophile friends for advice, they recommended studio-level equipment that was beyond my comprehension and my budget. As a software guy, I just wanted an audio setup that captured high-quality audio while still being convenient for everyday usage. I’d need it for remote meetings as well as for recording talks and tutorials. I was willing to pay for good equipment as long as I could use it well. Unfortunately, my biggest frustration was ignorance. I didn’t know anything about recording.

Here’s what I finally found to work well for me:

  1. A Blue Yeti microphone
  2. A Blue Compass boom arm
  3. A Blue Radius III shockmount
  4. A foam microphone windscreen

When I did my research, the Blue Yeti microphone was at the top of everyone’s recommendation list. The nicest thing about a Blue Yeti mic is that it connects via USB. You simply plug it right into your laptop and select audio input and output channels. The mic doesn’t need an external power source, either.

Initially, I bought only the Blue Yeti mic and the foam windscreen. Instead of using the boom arm, I used the tabletop stand that came with the mic for all my recordings. This was a big mistake. The tabletop stand picked up a lot of local noise, like typing on a keyboard. The audio quality it picked up also sounded like it had a bit of an echo, which may have been due to sound bouncing off the tabletop or my hardwood floors.

The boom arm and shockmount made a huge improvement in recording quality. Plus, with the boom arm mounted firmly to my desk, I can easily move it towards me for recording or out of the way otherwise. It feels quite sturdy. I chose to use all Blue products so that I could be certain that they’d work together. You can save quite a bit of money if you buy the microphone, boom arm, and shockmount together as a bundle on Amazon (~$200), instead of a la carte like I did (~$250).

My audio recording setup: a Blue Yeti microphone mounted onto a Blue Radius III Shockmount dangling from a Blue Compass boom arm. Pardon the mess on my desk – it’s a work in progress!

I plan to get a docking station for my laptop so that I can plug the microphone’s USB cable into the dock, simplifying desk’s cable management.

Even with this new setup, I still felt like I didn’t understand how to use my Blue Yeti microphone to its full potential. Thankfully, YouTube came to the rescue! This video greatly helped me understand things like tuning the mic’s gain and positioning the mic while speaking:

In summary, here’s what I like about the setup:

  • It yields very good (maybe professional?) audio recording quality.
  • It is simple enough for anyone to set up and use.
  • It is convenient to use for remote meetings, video recordings, etc.
  • It feels quite sturdy.
  • It is relatively affordable (compared to other audio equipment).

Please note: I am not an expert in audio equipment, and this article is not sponsored by any company. I simply hope that someone can benefit from the things I learned (and possibly save a few bucks) if they want to improve their own recording game!

Boa Constrictor’s Awesome Hacktoberfest 2021

Boa Constrictor is the .NET Screenplay Pattern. It helps you make better interactions for better automation! Its primary use case is Web UI and REST API test automation, but it can be used to automate any kind of interactions. The Screenplay Pattern is much more scalable for development and execution than the Page Object Model.

The Boa Constrictor maintainers and I strongly support open source software. That’s why we participated in Hacktoberfest 2021. In fact, this was the second Hacktoberfest we did. We launched Boa Constrictor as an open source project a year ago during Hacktoberfest 2020! We love sharing our code with the community and inspiring others to get involved. To encourage participation this year, we added the “hacktoberfest” label to open issues, and we offered cool stickers to anyone who contributed.

Boa Constrictor sticker
Boa Constrictor: The .NET Screenplay Pattern
Sticker Medallion

Hacktoberfest 2021 was a tremendous success for Boa Constrictor. Even though the project is small, we received several contributions. Here’s a summary of all the new stuff we added to Boa Constrictor:

  • Updated WebDriver interactions to use Selenium WebDriver 4.0
  • Implemented asynchronous programming for Tasks and Questions
  • Extended the Wait Task to wait for multiple Questions using AND and OR logic
  • Standardized ToString methods for all WebDriver interactions
  • Automated unit tests for WebDriver Questions
  • Wrote new user guides for test framework integrations and interaction patterns
  • Made small refinements to the doc site
  • Created GitHub templates for issues and pull requests
  • Replaced the symbols NuGet package with embedded debugging
  • Added the README to the NuGet package
  • Added Shields to the README
  • Restructured projects for docs, logos, and talk

During Hacktoberfest 2021, we made a series of four releases because we believe in lean development that puts new features in the hands of developers ASAP. The final capstone release was version 2.0.0: a culmination of all Hacktoberfest work! Here’s a view of the Boa Constrictor NuGet package with its new README (Shields included):

The Boa Constrictor NuGet package with the new README and Shields
The Boa Constrictor NuGet package with the new README and Shields

If you like project stats, then here’s a breakdown of the contributions by numbers:

  • 11 total contributors (5 submitting more than one pull request)
  • 41 pull requests closed
  • 151 commits made
  • Over 10K new lines of code

GitHub’s Code Frequency graph for Boa Constrictor shown below illustrates how much activity the project had during Hacktoberfest 2021. Notice the huge green and red spikes on the right side of the chart corresponding to the month of October 2021. That’s a lot of activity!

Hacktoberfest Contributions
The GitHub Code Frequency Graph for Boa Constrictor

Furthermore, every member of my Test Engineering & Architecture (TEA) team at Q2 completed four pull requests for Hacktoberfest, thus earning our prizes and our bragging rights. For the three others on the team, this was their first Hacktoberfest, and Boa Constrictor was their first open source project. We all joined together to make Boa Constrictor better for everyone. I’m very proud of each of them individually and of our team as a whole.

Personally, I gained more experience as an open source project maintainer. I brainstormed ideas with my team, assigned work to volunteers, and provided reviews for pull requests. I also had to handle slightly awkward situations, like politely turning down pull requests that could not be accepted. Thankfully, the project had very little spam, but we did have many potential contributors request to work on issues but then essentially disappear after being assigned. That made me appreciate the folks who did complete their pull requests even more.

Overall, Hacktoberfest 2021 was a great success for Boa Constrictor. We added several new features, docs, and quality-of-life improvements to the project. We also got people excited about open source contributions. Many thanks to Digital Ocean, Appwrite, Intel, and DeepSource for sponsoring Hacktoberfest 2021. Also, special thanks to Digital Ocean for featuring Boa Constrictor in their Hacktoberfest kickoff event. Keep on hacking!

Boa Constrictor is doing Hacktoberfest 2021!

Boa Constrictor is the .NET Screenplay Pattern. It helps you make better interactions for better automation! Its primary use case is Web UI and REST API test automation, but it can be used to automate any kind of interactions. The Screenplay Pattern is much more scalable for development and execution than the Page Object Model.

My team and I at Q2 developed Boa Constrictor for testing the PrecisionLender web app. Originally, we developed it internally as part of our C# test automation solution named “Boa”, but we later released it as an open source project on GitHub so that others could use it. In fact, we released it publicly in October 2020 during last year’s Hacktoberfest!

We are delighted to announce that Boa Constrictor will participate in Hacktoberfest 2021. Open source software is vital for our industry, and we strongly support efforts like Hacktoberfest to encourage folks to contribute to open source projects. Many thanks to Digital Ocean, Appwrite, Intel, and DeepSource for sponsoring Hacktoberfest again this year.

So, how can you contribute to Boa Constrictor? Take these four easy steps:

  1. Start by learning about the project.
  2. Read our guide to contributing code.
  3. Clone the GitHub repository.
  4. Look for unassigned open issues labeled “hacktoberfest”.
    1. Or, open an issue to propose a new idea!
  5. Add a comment to the issue saying that you’d like to do it.

To encourage contributions, I will give free Boa Constrictor stickers to anyone who makes a valid pull request to the project during Hacktoberfest 2021! (I’ll share a link where you can privately share your mailing address. I’ll mail stickers anywhere in the world – not just inside the United States.) The sticker is a 2″ medallion that looks like this:

Boa Constrictor sticker
The Boa Constrictor Sticker

Remember, you have until October 31 to make four qualifying pull requests for Hacktoberfest. We’d love for you to make at least one of those pull requests for Boa Constrictor.

How Q2 uses BDD with SpecFlow for testing PrecisionLender

This case study was written by Andrew Knight, Lead Software Engineer in Test for Q2’s PrecisionLender product, in collaboration with Q2 and Tricentis. It explains the PrecisionLender team’s continuous testing journey and how SpecFlow served as a cornerstone for success.

What is PrecisionLender?

PrecisionLender is a web application that empowers commercial bankers with in-the-moment insights that help them structure and price commercial deals. Andi®, PrecisionLender’s intelligent virtual analyst, delivers these hyper-focused recommendations in real-time, allowing relationship managers to make data-driven decisions while pricing their commercial deals. PrecisionLender is owned and developed by Q2, a financial experience software company dedicated to providing digital banking and lending solutions to banks, credit unions, alternative finance, and fintech companies in the U.S. and internationally.

The PrecisionLender Opportunity Screen
(Picture taken from the PrecisionLender Support Center)

The starting point

The PrecisionLender team had a robust Continuous Integration (CI) delivery pipeline with strong unit test coverage, but they lacked end-to-end feature coverage. Developers would fill this gap by manually inspecting their changes in a shared development environment. However, as the PrecisionLender app grew, manual checks could not cover all possible integrations. The team knew they needed continuous automated testing to provide a safety net for development to remain lean and efficient. In April 2018, they hired Andrew Knight as their first Software Engineer in Test (SET) – a new role for the company – to lead the effort.

Automating tests with SpecFlow

The PrecisionLender team developed the Boa test solution – a project for automating end-to-end tests at scale. Boa would become PrecisionLender’s internal platform for test automation development. The name “Boa” is a loose acronym for “Behavior-Oriented Automation.”

The team chose SpecFlow to be the core framework for Boa tests. Since the PrecisionLender app’s backend is developed using .NET, SpecFlow was a natural fit. SpecFlow’s Gherkin syntax made tests readable and understandable, even to product owners and product support specialists who do not code.

The SpecFlow framework integrates with tools like Selenium WebDriver for testing Web UIs and RestSharp for testing REST APIs to exercise vital pathways for thorough app coverage. SpecFlow’s dependency injection mechanisms are solid yet simple, and the online docs are thorough. Plus, SpecFlow is an open-source project, so anyone can look at its code to learn how things work, open requests for new features, and even offer code contributions.

An example Boa test, written in Gherkin using SpecFlow.

Executing tests with SpecFlow+ Runner

Writing good tests was only part of the challenge. The PrecisionLender team needed to execute Boa tests continuously to provide fast feedback on changes to the app. The team chose to run Boa tests using SpecFlow+ Runner, which is tailored for SpecFlow tests. The team uses SpecFlow+ Runner to launch tests in parallel in TeamCity any time a developer deploys a code change to internal pre-production environments. The entire test suite also runs every night against multiple product configurations. SpecFlow+ Runner produces a helpful test report with everything needed to triage test failures: pass-and-fail tallies overall and per feature, a visual execution timeline, and full system logs. If engineers need to investigate certain failures more closely, they can use SpecFlow tags and SpecFlow+ Runner profiles to selectively filter tests for reruns. SpecFlow+ Runner’s multiple features help the team expedite test execution and investigation.

The SpecFlow+ Runner report for a dozen smoke tests.

Sharing features with SpecFlow+ LivingDoc

Good test cases are more than just verification procedures – they are behavior specifications. They define how features should work. Instead of keeping testing work siloed by role, the PrecisionLender team wanted to share Boa tests as behavior specs with all stakeholders to foster greater collaboration and understanding around features. The team also wanted to share Boa tests with specific customers without sharing the entire automation code.

SpecFlow+ LivingDoc enabled the PrecisionLender team to turn Gherkin feature files into living documentation. Whereas the SpecFlow+ Runner report focuses on automation execution, the SpecFlow+ LivingDoc report focuses on behavior specification apart from coding and automation details. LivingDoc displays Gherkin scenarios in a readable, searchable way that both internal folks and customers can consume. It can also optionally include high-level pass-and-fail results for each scenario, providing just enough information to be helpful and not overwhelming. LivingDoc has also helped PrecisionLender’s engineers identify and eliminate unused step definitions within the automation code. PrecisionLender benefits greatly from complementary reports from SpecFlow+ Runner and SpecFlow+ LivingDoc.

The SpecFlow+ LivingDoc report for a dozen smoke tests with their pass-and-fail results.

Improving interactions with Boa Constrictor

The Boa test solution initially used the Page Object Model to model interactions with the PrecisionLender app. However, as the PrecisionLender team automated more and more Boa tests, it became apparent that page objects did not scale well. Many page object classes had duplicative methods, making automation code messy. Some methods also did not include appropriate waiting mechanisms, introducing flaky failures.

PrecisionLender’s SETs developed Boa Constrictor, a .NET implementation of the Screenplay Pattern, to make better interactions for better automation. In Screenplay, actors use abilities to perform interactions. For example, an ability could be using Selenium WebDriver, and an interaction could be clicking an element. The Screenplay Pattern can be seen as a refactoring of the Page Object Model that minimizes duplicate code through a better separation of concerns. Individual interactions can be hardened for robustness, eliminating flaky hotspots. The Boa test solution now exclusively uses Boa Constrictor for interactions.

In October 2020, Q2 released Boa Constrictor as an open-source project so that anyone can use it. It is fully compatible with SpecFlow and other .NET test frameworks, and it provides rich interactions for Selenium WebDriver and RestSharp out of the box.

Boa Constrictor, the .NET Screenplay Pattern.

Scaling massively with Selenium Grid

When the PrecisionLender team first started automating Boa tests, they ran tests one at a time. That soon became too slow since the average Boa test took 20 to 50 seconds to complete. The team then started running up to 3 tests in parallel on one machine, but that also was not fast enough. They turned to Selenium Grid, a tool for running WebDriver sessions remotely across multiple machines.

PrecisionLender built a set of internal Selenium Grid instances using Microsoft Azure virtual machines to run Boa tests at high scale. As of July 2021, PrecisionLender has over 1800 unique Boa tests that run across four distinct product configurations. Whenever TeamCity detects a code change, it triggers a “continuous” Boa test suite with over 1000 tests running 50 parallel tests using Google Chrome on Selenium Grid. It completes execution in about 10 minutes. TeamCity launches the full test suite every night against all product configurations with 64-100 parallel tests on Selenium Grid. Continuous Integration currently runs up to 10K Boa tests daily against the PrecisionLender app with SpecFlow+ Runner and Selenium Grid.

The Boa test solution architecture, including Continuous Integration through TeamCity and parallel testing with SpecFlow+ Runner and Selenium Grid.

Shifting left with BDD

Better testing and automation practices eventually inspired better development practices. Product owners would create user stories, but developers would struggle to understand requirements and business purposes fully. PrecisionLender’s SETs started bringing together the Three Amigos – business, development, and testing roles – to discuss product behaviors proactively while creating user stories. They introduced Behavior-Driven Development (BDD) activities like Example Mapping to explore behaviors together. Then, well-defined stories could be easily connected to SpecFlow tests written in Gherkin following Specification by Example (SBE). Teams repeatedly saved time by thinking before coding and specifying before testing. They built higher quality into features from the beginning, and they stopped before working on half-baked stories with unjustified value propositions. Developers who participated in these behavior-driven practices were also more likely to automate Boa tests on their own. Furthermore, one of PrecisionLender’s developers loved BDD practices so much that he joined the team of SETs! Through Gherkin, SpecFlow provided a foundation that enabled quality work to shift left.

Challenges along the way

Achieving true continuous testing had its challenges along the way. Intermittent failure was the most significant issue PrecisionLender faced at scale. With so many tests, environments, and infrastructural pieces, arbitrary failures were statistically unavoidable. The PrecisionLender team took a two-pronged approach to handle intermittent failures: (1) eliminate race conditions in automation using good interactions with Boa Constrictor, and (2) use SpecFlow+ Runner to automatically retry failed tests to determine if failures were consistent or intermittent. These two approaches reduced the frequency of flaky failures and helped engineers quickly resolve any remaining issues. As a result, Boa tests enjoy well above a 99% success rate, and most failures are due to actual bugs.

PrecisionLender app performance at scale was a second big challenge. Running up to 100 tests in parallel turned functional tests into de facto load tests. Testing at scale repeatedly uncovered performance bottlenecks in the app. Performance issues caused widespread test failures that were difficult to diagnose because they appeared intermittently. Still, the visual timeline and timestamps in the SpecFlow+ Runner report helped the team identify periods of failure that could be crosschecked against backend logs, metrics, and database queries. Developers resolved many performance issues and significantly boost the app’s response times and load capacity.

Training team members to develop solid test automation was the third challenge. At the start of the journey, test automation, Gherkin, and BDD were all new to PrecisionLender. The PrecisionLender SETs took active steps to train others on how to develop good tests and good automation through group workshops, Three Amigos meetings, and one-on-one mentoring sessions. They shared resources like the Automation Panda blog for how to write good tests and good Gherkin. The investment in education paid off: many developers have joined the SETs in writing readable, reliable Boa tests that run continuously.

Benefits to the business

Developing a continuous testing solution brought many incredible benefits to PrecisionLender. First, the quality of the PrecisionLender app improved because continuous testing provided fast feedback on failures that developers could quickly fix. Instead of relying on manual spot checks, the team could trust the comprehensive safety net of Boa tests to catch bugs. Many issues would be caught within an hour of a developer making a code commit, and the longest feedback cycle would be only one business day for the full nightly test suites to run. Boa tests catch failures before customers ever experience them. The continuous nature of testing enables PrecisionLender to publish new releases every two weeks.

Second, the high reliability of the Boa test solution means that the PrecisionLender team can trust test results. When a test passes, the behavior is working. When a test fails, there is a real bug. Reliability also means that engineers spend less time on automation maintenance and more time on more valuable activities, like developing new features and adding new tests. Quality is present in both the product code and the test code.

Third, continuous testing boosts customer confidence in PrecisionLender. Customers trust the software quality because they know that PrecisionLender thoroughly tests every release. The PrecisionLender team also shares SpecFlow+ LivingDoc reports with specific clients to prove quality.

A bright future

PrecisionLender’s continuous testing journey is not over. Since the PrecisionLender team hired its first SET, it has hired three more, in addition to a testing manager, to grow quality improvement efforts. Multiple development teams have written their own Boa tests, and they plan to write more tests independently. SpecFlow’s tools have been indispensable in helping the PrecisionLender team achieve successful quality assurance. As PrecisionLender welcomes more customers, the Boa solution will be ready to scale with more tests, more configurations, and more executions.