Author: Andy Knight

I'm a software engineer who specializes in test automation.

Want to practice test automation? Try these demo sites!

One of the biggest struggles in learning how to develop top-notch automation is practice. Testing is as much an art as it is a science. It takes time to discern when to add explicit waits, how to craft robust locators, and why to verify one element over another. It also requires apps with specific elements or endpoints to try certain operations. Unfortunately, although resources for learning how to automate tests (like Test Automation University) are abundant these days, public demo web sites for practicing remain elusive. I’ve struggled to find ones that I like, and others frequently ask me for recommendations.

Which demo sites are good?

Below is a list of demo sites I’ve found either by searching online or through recommendations from friends. Many of these sites appear on other “Top N Demo Sites for Testing” articles as well. My list aims not only to provide links to popular demo sites but also to provide recommendations on how to use them.

Types of sites:

  • Web UI site – looks like a real web site
  • Web UI elements – tutorial pages showcasing web element types
  • Mobile UI site – looks like a real mobile site
  • API site – provides public APIs for testing
  • DIY – “do it yourself”; you must set up and run the demo site yourself
Demo SiteTypeDescription
ParaBankWeb UI & API siteAn online banking site from Parasoft with login and REST/SOAP APIs. You can also access the database if you run the project locally using the source code.
Restful BookerWeb UI & API siteAn online site for bed & breakfast bookings from Mark Winteringham. The frontend is a React app (source), and the backend is a REST API (source).
Automation Practice WebsiteWeb UI siteA basic online store with optional login from SeleniumFramework.com. Great for example web UI tests.
Luma – Magento eCommerceWeb UI siteAnother online store with several pages.
DemoblazeWeb UI siteA basic online store with optional login from BlazeMeter. Great for example web UI tests.
Swag LabsWeb UI siteA basic online store with required login from Sauce Labs. Great for example web UI tests.
Applitools demo siteWeb UI siteA small site with login page and home page from Applitools. Compare/contrast with a second version for visual testing.
Automation BookstoreWeb UI siteA one-page site for dynamically searching book titles. Good for testing responsive design in short demos.
JPetStore DemoWeb UI siteA pet store site from MyBatis built on top of MyBatis 3, Spring 3, and Stripes (source).
GlobalSQA Banking ProjectWeb UI siteA basic banking app with dropdown login and simple pages from GlobalSQA.
Gatling Computers DatabaseWeb UI siteA one-page site from Gatling that provides a paginated list of computer models with the ability to filter and add new computers.
CandyMapperWeb UI siteA Halloween-themed site from Paul Grossman that shows scary bugs. It also comes with a second release that has the fixes.
Bulldoggy: The Reminders AppDIY Web UI siteA small web app I built myself in Python using FastAPI and HTMX. It has login and uses a TinyDB database.
OWASP Juice ShopDIY Web UI siteA test site written entirely in JavaScript using Node.js, Express and Angular. Specifically built for testing security vulnerabilities.
Cypress Real-World AppDIY Web UI siteA fake payment app from Cypress meant for demonstrating real-world Cypress testing. Could be used for other purposes.
RealWorld example appsDIY Web UI siteOne demo app implemented in multiple languages and frameworks. Not developed for testing but could nevertheless be used.
the-internetWeb UI elementsA site from Dave Haeffner and Elemental Selenium with several concise examples of web elements and interactions.
Selenium Test PagesWeb UI elementsA site with several pages that have slightly deeper examples than the-internet.
LetCodeWeb UI elementsA set of very clean pages along with video tutorials explaining how to automate interactions.
DemoQAWeb UI elementsA practice site from ToolsQA that includes pages for elements, forms, frames, interactions, and even a small bookstore.
Ultimate QA Automation PracticeWeb UI elementsA set of rich practice pages from Ultimate QA.
UI Test Automation PlaygroundWeb UI elementsA set of educational pages with interactable elements from the Rapise team.
SelectorsHub Practice PageWeb UI elementsA practice page from SelectorsHub for interacting with different kinds of web elements.
WebDriverUniversity.comWeb UI elementsAnother set of educational pages with interactable elements.
Sauce Labs Native Sample ApplicationDIY Mobile UI siteA mobile app from Sauce Labs for Android and iOS that is very similar to the Swag Labs web site.
JSONPlaceholderAPI siteA public REST API for generating fake data from a set of predefined resources. Follow the guide to learn how to make requests.
Swagger PetstoreAPI siteA public REST API from Swagger for testing authentication and CRUD operations for pet store data.
Public APIsAPI siteA long list of public APIs that can be used for testing.
Device Registry ServiceDIY API siteA Flask app I developed for teaching how to test REST APIs. Includes the app and automated API tests, which can both be run locally.
Best Buy API PlaygroundDIY API siteA JavaScript-based REST API demo service from Best Buy.

Which ones do you recommend most?

There are lots of demo sites above. Here are the ones I’d personally recommend for different needs:

Of course, the others are still good. That’s why they made the list!

Why bother with “fake” sites?

You might be wondering, “Why do we need demo sites? Why don’t we just use real sites?” Well, demo or “fake” web sites meet a few needs that real sites cannot:

  • Demo sites offer consistency. They are implemented one way and then do not change. Folks can trust that their tests will always work on them.
  • Demo sites are often simpler than real sites. They feel less intimidating for newcomers.
  • Demo sites can be designed for instruction. If they are part of a tutorial, the author can add features to the site to demonstrate concepts.
  • Demo sites are safer for publications like articles, tutorials, and books. Written content is static, so any sites referenced by examples should also be static. Real sites could change.
  • Real sites might require end user agreements that forbid automated requests. Some may even throttle or block requests if they suspect the requests come from a “bot.”
  • Real sites might also bear legal or copyright implications, especially if a company is using the sites for their own content.

What limitations do demo sites have?

Unfortunately, demo sites do have limitations:

  • Demo sites may be too simplified. They may lack large workflows or real-world data. Folks might become frustrated when they discover inactive elements that appear to be real.
  • Demo sites may not be built to scale. High request volume or parallel testing scale might cripple them.
  • Demo sites may seem like they have poor quality, whether or not they actually have poor quality. Sometimes, they are built quickly for testing purposes and therefore don’t have the attention to detail as real sites.
  • Demo sites with significant company branding may be inappropriate to use. For example, if A and B are competitors, then A shouldn’t use B’s demo site for their product tutorials.

Other great lists

Here are other great articles that list demo sites for testing. I referred to them while compiling this list:

Do you have other demo sites to recommend? Put them in the comments below!

Welcome to Raleigh

Thoughts on Remote Work

At my new job as Developer Advocate at Applitools, I now permanently work remotely. Working from home is new to me. At my previous company PrecisionLender (later acquired by Q2), I worked in an office that was a seven-minute drive from my house. My office was a real office: an enclosed room with a door, walls, a large desk, and two huge whiteboards. Working from the office was a joy. We had free snacks and sodas. My colleagues and I all got along great, and we’d go out to lunch almost every day. After COVID-19 hit the USA in March 2020, I was heartbroken to abandon the office. I was so stubborn about eventually returning to the office that I refused to set up a decent workstation at home. For a year and a half, I worked exclusively on my laptop either from a desk in my living room or from the couch – no monitors, no mouse, and no separate keyboard. I mean, come on, the pandemic would be over in a few short months, right?

However, my attitudes about remote work are changing. At PrecisionLender, I was spoiled with the best kind of office situation someone could have. Many others in the software field are not so fortunate. Those who live in the big cities like New York and San Francisco pay thousands of dollars a month to split tiny apartments and spend an hour or more on one-way commutes. Not to mention, many of them move away from family and friends to take those opportunities.

Office layout trends in recent years have also been dismal. The open layout – no walls with clusters of desks sloppily smashed together – is the norm. Perpetual audio and visual distractions are a nightmare for anyone trying to concentrate. Widescreen monitors and noise-cancelling headphones only help a little. I experienced that frustration at MaxPoint and LexisNexis. Cubicles, while passé these days, are only a small step up from open floors. While the “walls” block out visual distractions, you can still hear everything that goes on around you. Plus, the enormous blocks of cubicles on a floor feel brutally confining. I frequently called the IBM 500 complex in RTP “Cubicle City” when I worked there.

Remote work can be a godsend for many hardworking folks who just want to do a good job while doing right by themselves. Folks, especially in software, can literally work anywhere. COVID-19 shutdowns have proven it. My wife and I even took advantage of it: we bought a vacation condo in Seattle, and I worked remotely from there for several weeks. I could automate tests during the day, grab beers with close friends, and catch a Mariners game on the weekend. Even with a 3-hour time zone difference, I was right in step with my team the whole time.

The rooftop of my Seattle condo building
My Seattle condo building’s rooftop

Recently, I saw a Twitter thread by Chris Herd in which he made several predictions on remote work in 2022 and beyond:

This thread really resonated with me. While I still have some concerns about remote work, I do believe it will be a net good for not only my country and people but also for the world. I’d like to highlight some of the points he made that really stuck out to me.

I agree completely with this, and I’m excited to see it happen. The US has so many awesome places, from sea to shining sea. There’s no reason why someone should need to move from their hometown to a big city just to work for a specific company (unless, of course, they want that adventure). Places like Raleigh-Durham, Nashville, Denver, Miami, Pittsburgh, Salt Lake City, Cleveland, and countless other smaller American cities are fantastic places to live. Each has a unique vibe and local specialties. Allowing folks to live where they want with good employment opportunities will rejuvenate local and regional economies. It will bring broader prosperity to all.

This is happening already. The Triangle of NC, where I live, is booming. My wife and I rent local listings on Airbnb, and all through the pandemic, the #1 type of guest we’ve hosted by far is someone interested in relocating to the Triangle.

I hadn’t thought about asynchronous work until Chris mentioned it, but he’s right. Life is 24/7, and people live all around the globe. Remote work can liberate us from a harsh 9-5 and instead embrace more flexible hours. For example, this past week, I had to work around the clock for a few days prepping a big webinar. However, I wasn’t working for 16 hours straight each day. I woke up a bit late, did some light administrative activities before lunch, hit it hard all afternoon, stopped for dinner and time with my puppy, and then burned the midnight oil after dark. That’s what worked for me.

Yes! I’m living it myself. My wife and I just adopted a French bulldog puppy, and I also started restoring a vintage 1970 Volkswagen Beetle.

VW Beetle Engine Swap
I swapped the engine out of my vintage 1970 Volkswagen Beetle
Suki the Frenchie
Suki the French bulldog is the cutest puppy in the world!

I sure hope so! Break down those entry barriers. Do away with the gatekeepers. End coastal elitism.

I sure hope this one becomes true, but I’m skeptical.

This one is so ironic. Based on many articles I’ve read, upper management seems to despise work-from-home because they think their workers will work less. However, all the good, hardworking folks I know are actually working more than ever. Not only are folks more efficient, but they tend to keep working just a little extra because there are fewer barriers between work stuff and personal stuff. In my new job, I know I’m certainly working more hours right now than I probably should, and I’m hoping to settle into a more sustainable routine as I acclimate to my new role. Nevertheless, we absolutely need to protect ourselves against unintentional burnout.

One big concern I have with remote work is the loss of in-person contact. Humans need to be around each other, and screens are a poor substitute. I know I’ve felt lonelier working from home than when I worked in an office. That’s why I absolutely love this idea about regular remote retreats. Companies can still bring folks together, and they can hire professionals who know how to maximize that time. I go to software conferences frequently for similar experiences.

I think this goes hand-in-hand with async work. However, unless safeguards are put in place to prevent overworking, then life-work balance could quickly become work-more-work imbalance.

Honestly, this one was never much of a problem for me. The only “paddings” I’ve had in my workdays are long, boring meetings where I participated little. Thankfully, those were confined mostly to my days at IBM and NetApp earlier in my career.

YES! Even though my commutes have always been relatively short, I still appreciate the time I get back in my day. Here are some things I’ve done with my extra, more flexible time:

  • Sleep in a little later
  • Play with my puppy when I need a break
  • Share lunch nearby with friends
  • Run a quick errand in my Beetle
  • Grab an impromptu bubble tea with my wife
  • Take a walk around my neighborhood while thinking about work-related challenges

Documentation is the unspoken superpower of remote teams. I strongly value written communication for its clarity, searchability, and permanence.

Chris had lots of other good tweets in that thread, so be sure to read them. Overall, his tone was bullishly optimistic on remote work – not only in its benefits, but also in its inevitability. Personally, I hope remote work is here to stay, and that it’s not just a phase brought on by the COVID-19 pandemic. It’s a game-changer for individuals and industries alike. However, I’m still somewhat fearful that big companies will reign in wide scale returns to the office after the pandemic. Giants have already sunk millions to billions in commercial real estate for their offices that they can’t get back. I’m also worried about salaries in different geographies. Employers already pay folks who live outside of big cities much less for the same work due to cost-of-living differences. I don’t think that will be fair as people leave big cities in droves for smaller cities or the countryside. Already, the cost of living is skyrocketing in the Triangle, yet our average tech salaries are far below those of Silicon Valley. We’ll see what happens.

Overall, I am optimistic about remote work just like Chris.

So, what do y’all think? Please leave comments below. I’d love to learn different perspectives!

Playwright Workshop for TAU: The Homecoming

Want to learn Playwright with Python? Take this workshop!

Playwright is an awesome new browser automation library. With Playwright, you can automate web UI interactions for testing or for web scraping with a concise, uniform API in one of four languages: Python, C#, Java, and JavaScript. Playwright is also completely open source and backed by Microsoft. It’s a powerful alternative to Selenium WebDriver.

On December 1, 2021, I delivered a workshop on Playwright for TAU: The Homecoming. In my workshop, I taught how to build a test automation project in Python using Playwright with pytest, Python’s most popular test framework. We automated a test case together for performing a DuckDuckGo web search.

If you missed the workshop, no worries: You can still take the workshop as a self-guided tutorial! The workshop instructions and example code are located in this GitHub repository:

https://github.com/AutomationPanda/tau-playwright-workshop

To take the workshop as a self-guided tutorial, read the repository’s README, and then follow the instructions in the Markdown guides under the workshop folder. The workshop has five main parts:

  1. Getting started
    1. What is Playwright?
    2. Our web search test
    3. Test project setup
  2. First steps with Playwright
    1. Browsers, contexts, and pages
    2. Navigating to a web page
    3. Performing a search
  3. Writing assertions
    1. Checking the search field
    2. Checking the result links
    3. Checking the title
  4. Refactoring using page objects
    1. The search page
    2. The result page
    3. Page object fixtures
  5. Nifty Playwright tricks
    1. Testing different browsers
    2. Capturing screenshots and videos
    3. Running tests in parallel

If you get stuck or have any questions, please open issues against the GitHub repository, and I’ll try to help. Happy coding!

A Simple High-Quality Mic Setup for Software Pros

I’ve always been frustrated with poor quality audio recordings. Microphones built into laptops are very convenient, but they usually yield tinny sound lacking the depth of real voices. When I asked my audiophile friends for advice, they recommended studio-level equipment that was beyond my comprehension and my budget. As a software guy, I just wanted an audio setup that captured high-quality audio while still being convenient for everyday usage. I’d need it for remote meetings as well as for recording talks and tutorials. I was willing to pay for good equipment as long as I could use it well. Unfortunately, my biggest frustration was ignorance. I didn’t know anything about recording.

Here’s what I finally found to work well for me:

  1. A Blue Yeti microphone
  2. A Blue Compass boom arm
  3. A Blue Radius III shockmount
  4. A foam microphone windscreen

When I did my research, the Blue Yeti microphone was at the top of everyone’s recommendation list. The nicest thing about a Blue Yeti mic is that it connects via USB. You simply plug it right into your laptop and select audio input and output channels. The mic doesn’t need an external power source, either.

Initially, I bought only the Blue Yeti mic and the foam windscreen. Instead of using the boom arm, I used the tabletop stand that came with the mic for all my recordings. This was a big mistake. The tabletop stand picked up a lot of local noise, like typing on a keyboard. The audio quality it picked up also sounded like it had a bit of an echo, which may have been due to sound bouncing off the tabletop or my hardwood floors.

The boom arm and shockmount made a huge improvement in recording quality. Plus, with the boom arm mounted firmly to my desk, I can easily move it towards me for recording or out of the way otherwise. It feels quite sturdy. I chose to use all Blue products so that I could be certain that they’d work together. You can save quite a bit of money if you buy the microphone, boom arm, and shockmount together as a bundle on Amazon (~$200), instead of a la carte like I did (~$250).

My audio recording setup: a Blue Yeti microphone mounted onto a Blue Radius III Shockmount dangling from a Blue Compass boom arm. Pardon the mess on my desk – it’s a work in progress!

I plan to get a docking station for my laptop so that I can plug the microphone’s USB cable into the dock, simplifying desk’s cable management.

Even with this new setup, I still felt like I didn’t understand how to use my Blue Yeti microphone to its full potential. Thankfully, YouTube came to the rescue! This video greatly helped me understand things like tuning the mic’s gain and positioning the mic while speaking:

In summary, here’s what I like about the setup:

  • It yields very good (maybe professional?) audio recording quality.
  • It is simple enough for anyone to set up and use.
  • It is convenient to use for remote meetings, video recordings, etc.
  • It feels quite sturdy.
  • It is relatively affordable (compared to other audio equipment).

Please note: I am not an expert in audio equipment, and this article is not sponsored by any company. I simply hope that someone can benefit from the things I learned (and possibly save a few bucks) if they want to improve their own recording game!

Boa Constrictor’s Awesome Hacktoberfest 2021

Boa Constrictor is the .NET Screenplay Pattern. It helps you make better interactions for better automation! Its primary use case is Web UI and REST API test automation, but it can be used to automate any kind of interactions. The Screenplay Pattern is much more scalable for development and execution than the Page Object Model.

The Boa Constrictor maintainers and I strongly support open source software. That’s why we participated in Hacktoberfest 2021. In fact, this was the second Hacktoberfest we did. We launched Boa Constrictor as an open source project a year ago during Hacktoberfest 2020! We love sharing our code with the community and inspiring others to get involved. To encourage participation this year, we added the “hacktoberfest” label to open issues, and we offered cool stickers to anyone who contributed.

Boa Constrictor sticker
Boa Constrictor: The .NET Screenplay Pattern
Sticker Medallion

Hacktoberfest 2021 was a tremendous success for Boa Constrictor. Even though the project is small, we received several contributions. Here’s a summary of all the new stuff we added to Boa Constrictor:

  • Updated WebDriver interactions to use Selenium WebDriver 4.0
  • Implemented asynchronous programming for Tasks and Questions
  • Extended the Wait Task to wait for multiple Questions using AND and OR logic
  • Standardized ToString methods for all WebDriver interactions
  • Automated unit tests for WebDriver Questions
  • Wrote new user guides for test framework integrations and interaction patterns
  • Made small refinements to the doc site
  • Created GitHub templates for issues and pull requests
  • Replaced the symbols NuGet package with embedded debugging
  • Added the README to the NuGet package
  • Added Shields to the README
  • Restructured projects for docs, logos, and talk

During Hacktoberfest 2021, we made a series of four releases because we believe in lean development that puts new features in the hands of developers ASAP. The final capstone release was version 2.0.0: a culmination of all Hacktoberfest work! Here’s a view of the Boa Constrictor NuGet package with its new README (Shields included):

The Boa Constrictor NuGet package with the new README and Shields
The Boa Constrictor NuGet package with the new README and Shields

If you like project stats, then here’s a breakdown of the contributions by numbers:

  • 11 total contributors (5 submitting more than one pull request)
  • 41 pull requests closed
  • 151 commits made
  • Over 10K new lines of code

GitHub’s Code Frequency graph for Boa Constrictor shown below illustrates how much activity the project had during Hacktoberfest 2021. Notice the huge green and red spikes on the right side of the chart corresponding to the month of October 2021. That’s a lot of activity!

Hacktoberfest Contributions
The GitHub Code Frequency Graph for Boa Constrictor

Furthermore, every member of my Test Engineering & Architecture (TEA) team at Q2 completed four pull requests for Hacktoberfest, thus earning our prizes and our bragging rights. For the three others on the team, this was their first Hacktoberfest, and Boa Constrictor was their first open source project. We all joined together to make Boa Constrictor better for everyone. I’m very proud of each of them individually and of our team as a whole.

Personally, I gained more experience as an open source project maintainer. I brainstormed ideas with my team, assigned work to volunteers, and provided reviews for pull requests. I also had to handle slightly awkward situations, like politely turning down pull requests that could not be accepted. Thankfully, the project had very little spam, but we did have many potential contributors request to work on issues but then essentially disappear after being assigned. That made me appreciate the folks who did complete their pull requests even more.

Overall, Hacktoberfest 2021 was a great success for Boa Constrictor. We added several new features, docs, and quality-of-life improvements to the project. We also got people excited about open source contributions. Many thanks to Digital Ocean, Appwrite, Intel, and DeepSource for sponsoring Hacktoberfest 2021. Also, special thanks to Digital Ocean for featuring Boa Constrictor in their Hacktoberfest kickoff event. Keep on hacking!

Boa Constrictor is doing Hacktoberfest 2021!

Boa Constrictor is the .NET Screenplay Pattern. It helps you make better interactions for better automation! Its primary use case is Web UI and REST API test automation, but it can be used to automate any kind of interactions. The Screenplay Pattern is much more scalable for development and execution than the Page Object Model.

My team and I at Q2 developed Boa Constrictor for testing the PrecisionLender web app. Originally, we developed it internally as part of our C# test automation solution named “Boa”, but we later released it as an open source project on GitHub so that others could use it. In fact, we released it publicly in October 2020 during last year’s Hacktoberfest!

We are delighted to announce that Boa Constrictor will participate in Hacktoberfest 2021. Open source software is vital for our industry, and we strongly support efforts like Hacktoberfest to encourage folks to contribute to open source projects. Many thanks to Digital Ocean, Appwrite, Intel, and DeepSource for sponsoring Hacktoberfest again this year.

So, how can you contribute to Boa Constrictor? Take these four easy steps:

  1. Start by learning about the project.
  2. Read our guide to contributing code.
  3. Clone the GitHub repository.
  4. Look for unassigned open issues labeled “hacktoberfest”.
    1. Or, open an issue to propose a new idea!
  5. Add a comment to the issue saying that you’d like to do it.

To encourage contributions, I will give free Boa Constrictor stickers to anyone who makes a valid pull request to the project during Hacktoberfest 2021! (I’ll share a link where you can privately share your mailing address. I’ll mail stickers anywhere in the world – not just inside the United States.) The sticker is a 2″ medallion that looks like this:

Boa Constrictor sticker
The Boa Constrictor Sticker

Remember, you have until October 31 to make four qualifying pull requests for Hacktoberfest. We’d love for you to make at least one of those pull requests for Boa Constrictor.

How Q2 uses BDD with SpecFlow for testing PrecisionLender

This case study was written by Andrew Knight, Lead Software Engineer in Test for Q2’s PrecisionLender product, in collaboration with Q2 and Tricentis. It explains the PrecisionLender team’s continuous testing journey and how SpecFlow served as a cornerstone for success.

What is PrecisionLender?

PrecisionLender is a web application that empowers commercial bankers with in-the-moment insights that help them structure and price commercial deals. Andi®, PrecisionLender’s intelligent virtual analyst, delivers these hyper-focused recommendations in real-time, allowing relationship managers to make data-driven decisions while pricing their commercial deals. PrecisionLender is owned and developed by Q2, a financial experience software company dedicated to providing digital banking and lending solutions to banks, credit unions, alternative finance, and fintech companies in the U.S. and internationally.

The PrecisionLender Opportunity Screen
(Picture taken from the PrecisionLender Support Center)

The starting point

The PrecisionLender team had a robust Continuous Integration (CI) delivery pipeline with strong unit test coverage, but they lacked end-to-end feature coverage. Developers would fill this gap by manually inspecting their changes in a shared development environment. However, as the PrecisionLender app grew, manual checks could not cover all possible integrations. The team knew they needed continuous automated testing to provide a safety net for development to remain lean and efficient. In April 2018, they hired Andrew Knight as their first Software Engineer in Test (SET) – a new role for the company – to lead the effort.

Automating tests with SpecFlow

The PrecisionLender team developed the Boa test solution – a project for automating end-to-end tests at scale. Boa would become PrecisionLender’s internal platform for test automation development. The name “Boa” is a loose acronym for “Behavior-Oriented Automation.”

The team chose SpecFlow to be the core framework for Boa tests. Since the PrecisionLender app’s backend is developed using .NET, SpecFlow was a natural fit. SpecFlow’s Gherkin syntax made tests readable and understandable, even to product owners and product support specialists who do not code.

The SpecFlow framework integrates with tools like Selenium WebDriver for testing Web UIs and RestSharp for testing REST APIs to exercise vital pathways for thorough app coverage. SpecFlow’s dependency injection mechanisms are solid yet simple, and the online docs are thorough. Plus, SpecFlow is an open-source project, so anyone can look at its code to learn how things work, open requests for new features, and even offer code contributions.

An example Boa test, written in Gherkin using SpecFlow.

Executing tests with SpecFlow+ Runner

Writing good tests was only part of the challenge. The PrecisionLender team needed to execute Boa tests continuously to provide fast feedback on changes to the app. The team chose to run Boa tests using SpecFlow+ Runner, which is tailored for SpecFlow tests. The team uses SpecFlow+ Runner to launch tests in parallel in TeamCity any time a developer deploys a code change to internal pre-production environments. The entire test suite also runs every night against multiple product configurations. SpecFlow+ Runner produces a helpful test report with everything needed to triage test failures: pass-and-fail tallies overall and per feature, a visual execution timeline, and full system logs. If engineers need to investigate certain failures more closely, they can use SpecFlow tags and SpecFlow+ Runner profiles to selectively filter tests for reruns. SpecFlow+ Runner’s multiple features help the team expedite test execution and investigation.

The SpecFlow+ Runner report for a dozen smoke tests.

Sharing features with SpecFlow+ LivingDoc

Good test cases are more than just verification procedures – they are behavior specifications. They define how features should work. Instead of keeping testing work siloed by role, the PrecisionLender team wanted to share Boa tests as behavior specs with all stakeholders to foster greater collaboration and understanding around features. The team also wanted to share Boa tests with specific customers without sharing the entire automation code.

SpecFlow+ LivingDoc enabled the PrecisionLender team to turn Gherkin feature files into living documentation. Whereas the SpecFlow+ Runner report focuses on automation execution, the SpecFlow+ LivingDoc report focuses on behavior specification apart from coding and automation details. LivingDoc displays Gherkin scenarios in a readable, searchable way that both internal folks and customers can consume. It can also optionally include high-level pass-and-fail results for each scenario, providing just enough information to be helpful and not overwhelming. LivingDoc has also helped PrecisionLender’s engineers identify and eliminate unused step definitions within the automation code. PrecisionLender benefits greatly from complementary reports from SpecFlow+ Runner and SpecFlow+ LivingDoc.

The SpecFlow+ LivingDoc report for a dozen smoke tests with their pass-and-fail results.

Improving interactions with Boa Constrictor

The Boa test solution initially used the Page Object Model to model interactions with the PrecisionLender app. However, as the PrecisionLender team automated more and more Boa tests, it became apparent that page objects did not scale well. Many page object classes had duplicative methods, making automation code messy. Some methods also did not include appropriate waiting mechanisms, introducing flaky failures.

PrecisionLender’s SETs developed Boa Constrictor, a .NET implementation of the Screenplay Pattern, to make better interactions for better automation. In Screenplay, actors use abilities to perform interactions. For example, an ability could be using Selenium WebDriver, and an interaction could be clicking an element. The Screenplay Pattern can be seen as a refactoring of the Page Object Model that minimizes duplicate code through a better separation of concerns. Individual interactions can be hardened for robustness, eliminating flaky hotspots. The Boa test solution now exclusively uses Boa Constrictor for interactions.

In October 2020, Q2 released Boa Constrictor as an open-source project so that anyone can use it. It is fully compatible with SpecFlow and other .NET test frameworks, and it provides rich interactions for Selenium WebDriver and RestSharp out of the box.

Boa Constrictor, the .NET Screenplay Pattern.

Scaling massively with Selenium Grid

When the PrecisionLender team first started automating Boa tests, they ran tests one at a time. That soon became too slow since the average Boa test took 20 to 50 seconds to complete. The team then started running up to 3 tests in parallel on one machine, but that also was not fast enough. They turned to Selenium Grid, a tool for running WebDriver sessions remotely across multiple machines.

PrecisionLender built a set of internal Selenium Grid instances using Microsoft Azure virtual machines to run Boa tests at high scale. As of July 2021, PrecisionLender has over 1800 unique Boa tests that run across four distinct product configurations. Whenever TeamCity detects a code change, it triggers a “continuous” Boa test suite with over 1000 tests running 50 parallel tests using Google Chrome on Selenium Grid. It completes execution in about 10 minutes. TeamCity launches the full test suite every night against all product configurations with 64-100 parallel tests on Selenium Grid. Continuous Integration currently runs up to 10K Boa tests daily against the PrecisionLender app with SpecFlow+ Runner and Selenium Grid.

The Boa test solution architecture, including Continuous Integration through TeamCity and parallel testing with SpecFlow+ Runner and Selenium Grid.

Shifting left with BDD

Better testing and automation practices eventually inspired better development practices. Product owners would create user stories, but developers would struggle to understand requirements and business purposes fully. PrecisionLender’s SETs started bringing together the Three Amigos – business, development, and testing roles – to discuss product behaviors proactively while creating user stories. They introduced Behavior-Driven Development (BDD) activities like Example Mapping to explore behaviors together. Then, well-defined stories could be easily connected to SpecFlow tests written in Gherkin following Specification by Example (SBE). Teams repeatedly saved time by thinking before coding and specifying before testing. They built higher quality into features from the beginning, and they stopped before working on half-baked stories with unjustified value propositions. Developers who participated in these behavior-driven practices were also more likely to automate Boa tests on their own. Furthermore, one of PrecisionLender’s developers loved BDD practices so much that he joined the team of SETs! Through Gherkin, SpecFlow provided a foundation that enabled quality work to shift left.

Challenges along the way

Achieving true continuous testing had its challenges along the way. Intermittent failure was the most significant issue PrecisionLender faced at scale. With so many tests, environments, and infrastructural pieces, arbitrary failures were statistically unavoidable. The PrecisionLender team took a two-pronged approach to handle intermittent failures: (1) eliminate race conditions in automation using good interactions with Boa Constrictor, and (2) use SpecFlow+ Runner to automatically retry failed tests to determine if failures were consistent or intermittent. These two approaches reduced the frequency of flaky failures and helped engineers quickly resolve any remaining issues. As a result, Boa tests enjoy well above a 99% success rate, and most failures are due to actual bugs.

PrecisionLender app performance at scale was a second big challenge. Running up to 100 tests in parallel turned functional tests into de facto load tests. Testing at scale repeatedly uncovered performance bottlenecks in the app. Performance issues caused widespread test failures that were difficult to diagnose because they appeared intermittently. Still, the visual timeline and timestamps in the SpecFlow+ Runner report helped the team identify periods of failure that could be crosschecked against backend logs, metrics, and database queries. Developers resolved many performance issues and significantly boost the app’s response times and load capacity.

Training team members to develop solid test automation was the third challenge. At the start of the journey, test automation, Gherkin, and BDD were all new to PrecisionLender. The PrecisionLender SETs took active steps to train others on how to develop good tests and good automation through group workshops, Three Amigos meetings, and one-on-one mentoring sessions. They shared resources like the Automation Panda blog for how to write good tests and good Gherkin. The investment in education paid off: many developers have joined the SETs in writing readable, reliable Boa tests that run continuously.

Benefits to the business

Developing a continuous testing solution brought many incredible benefits to PrecisionLender. First, the quality of the PrecisionLender app improved because continuous testing provided fast feedback on failures that developers could quickly fix. Instead of relying on manual spot checks, the team could trust the comprehensive safety net of Boa tests to catch bugs. Many issues would be caught within an hour of a developer making a code commit, and the longest feedback cycle would be only one business day for the full nightly test suites to run. Boa tests catch failures before customers ever experience them. The continuous nature of testing enables PrecisionLender to publish new releases every two weeks.

Second, the high reliability of the Boa test solution means that the PrecisionLender team can trust test results. When a test passes, the behavior is working. When a test fails, there is a real bug. Reliability also means that engineers spend less time on automation maintenance and more time on more valuable activities, like developing new features and adding new tests. Quality is present in both the product code and the test code.

Third, continuous testing boosts customer confidence in PrecisionLender. Customers trust the software quality because they know that PrecisionLender thoroughly tests every release. The PrecisionLender team also shares SpecFlow+ LivingDoc reports with specific clients to prove quality.

A bright future

PrecisionLender’s continuous testing journey is not over. Since the PrecisionLender team hired its first SET, it has hired three more, in addition to a testing manager, to grow quality improvement efforts. Multiple development teams have written their own Boa tests, and they plan to write more tests independently. SpecFlow’s tools have been indispensable in helping the PrecisionLender team achieve successful quality assurance. As PrecisionLender welcomes more customers, the Boa solution will be ready to scale with more tests, more configurations, and more executions.

My Upside-Down QA Story for Global Testers Day

Happy Global Testers Day! For 2021, QA Touch is celebrating with webinars, games, competitions, blogs, and videos. I participated by sharing an “upside-down” story from years ago when I accidentally wiped out all of NetApp’s continuous integration testing. Please watch my story below. I hope you find it both insightful and entertaining!

dust

Skin Rashes and Software Testing

Many of you know me as the “Automation Panda” through my blog, my Twitter handle, or my online courses. Maybe you’ve attended one of my conference talks. I love connecting with others, but many people don’t get to know me personally. Behind my black-and-white façade, I’m a regular guy. When I’m not programming, I enjoy cooking and video gaming. I’m also currently fixing up a vintage Volkswagen Beetle. However, for nearly two years, I’ve suffered a skin rash that will not go away. I haven’t talked about it much until recently, when it became unbearable.

For a while, things turned bad. Thankfully, things are a little better now. I’d like to share my journey publicly because it helps to humanize folks in tech like me. We’re all people, not machines. Vulnerability is healthy. My journey also reminded me of a few important tenants for testing software. I hope you’ll read my story, and I promise to avoid the gross parts.


Distant Precursors

I’ve been blessed with great health and healthcare my entire life. However, when I was a teenager, I had a weird skin issue: the skin around my right eye socket turned dry, itchy, and red. Imagine dandruff, but on your face – it was flaky like a bad test (ha!). Lotions and creams did nothing to help. After it persisted for several weeks, my parents scheduled an appointment with my pediatrician. They didn’t know what caused it, but they gave me a sample tube of topical steroids to try. The steroids worked great. The skin around my eye cleared up and stayed normal. This issue resurfaced again while I was in college, but it went away on its own after a month or two.

Rise of the Beast

Around October 2019, I noticed this same rash started appearing again around my right eye for the first time in a decade. The exact date is fuzzy in my mind, but I remember it started before TestBash San Francisco 2019. At that time, my response was to ignore it. I’d continue to keep up my regular hygiene (like washing my face), and eventually it would go away like last time.

Unfortunately, the rash got worse. It started spreading to my cheek and my forehead. I started using body lotion on it, but it would burn whenever I’d apply it, and the rash would persist. My wife started trying a bunch of fancy, high-dollar lotions (like Kiehl’s and other brands I didn’t know), but they didn’t help at all. By Spring 2020, my hands and forearms started breaking out with dry, itchy red spots, too. These were worse: if I scratched them, they would bleed. I also remember taking my morning shower one day and thinking to myself, “Gosh, my whole body is super itchy!”

I had enough, so I visited a local dermatologist. He took one look at my face and arms and prescribed topical steroids. He told me to use them for about four weeks and then stop for two weeks before a reevaluation. The steroids partially worked. The itching would subside, but the rash wouldn’t go away completely. When I stopped using the steroids, the rash returned to the same places. I also noticed the rash slowly spreading, too. Eventually, it crept up my upper arms to my neck and shoulders, down my back and torso, and all the way down to my legs.

On a second attempt, the dermatologist prescribed a much stronger topical steroid in addition to a round of oral steroids. My skin healed much better with the stronger medicine, but, inevitably, when I stopped using them, the rash returned. By now, patches of the rash could be found all over my body, and they itched like crazy. I couldn’t wear white shirts anymore because spots would break out and bleed, as if I nicked myself while shaving. I don’t remember the precise timings, but I also asked the dermatologist to remove a series of three moles on my body that became infected badly by the rash.

Fruitless Mitigations

As a good tester, I wanted to know the root cause for my rash. Was I allergic to something? Did I have a deeper medical issue? Steroids merely addressed the symptoms, and they did a mediocre job at best. So, I tried investigating what triggered my rash.

When the dry patch first appeared above my eye, I suspected cold, dry weather. “Maybe the crisp winter air is drying my skin too much.” That’s when I tried using an assortment of creams. When the rash started spreading, that’s when I knew the cause was more than winter weather.

Then, my mind turned to allergies. I knew I was allergic to pet dander, but I never had reactions to anything else before. At the same time the rash started spreading from my eye, I noticed I had a small problem with mangos. Whenever I would bite into a fresh mango, my lips would become severely, painfully chapped for the next few days. I learned from reading articles online that some folks who are allergic to poison ivy (like me) are also sensitive to mangos because both plants contain urushiol in their skin and sap. At that time, my family and I were consuming a boxful of mangos. I immediately cut out mangos and hoped for the best. No luck.

When the rash spread to my whole body, I became serious about finding the root cause. Every effect has a cause, and I wanted to investigate all potential causes in order of likelihood. I already crossed dry weather and mangoes off the list. Since the rash appeared in splotches across my whole body, then I reasoned its trigger could either be external – something coming in contact with all parts of skin – or internal – something not right from the inside pushing out.

What comes in contact with the whole body? Air, water, and cloth. Skin reactions to things in air and water seemed unlikely, so I focused on clothing and linens. That’s when I remembered a vague story from childhood. When my parents taught me how to do laundry, they told me they used a scent-free, hypoallergenic detergent because my dad had a severe skin reaction to regular detergent one time long before I was born. Once I confirmed the story with my parents, I immediately sprung to action. I switched over my detergent and fabric softener. I rewashed all my clothes and linens – all of them. I even thoroughly cleaned out my dryer duct to make sure no chemicals could leech back into the machine. (Boy, that was a heaping pile of dust.) Despite these changes, my rash persisted. I also changed my soaps and shampoos to no avail.

At the same time, I looked internally. I read in a few online articles that skin rashes could be caused by deficiencies. I started taking a daily multivitamin. I also tried supplements for calcium, Vitamin B6, Vitamin D, and collagen. Although I’m sure my body was healthier as a result, none of these supplements made a noticeable difference.

My dermatologist even did a skin punch test. He cut a piece of skin out of my back about 3mm wide through all layers of the skin. The result of the biopsy was “atopic dermatitis.” Not helpful.

For relief, I tried an assortment of creams from Eucerin, CeraVe, Aveeno, and O’Keeffe’s. None of them staved off the persistent itching or reduced the redness. They were practically useless. The only cream that had any impact (other than steroids) was an herbal Chinese medicine. With a cooling burn, it actually stopped the itch and visibly reduced the redness. By Spring 2021, I stopped going to the dermatologist and simply relied on the Chinese cream. I didn’t have a root cause, but I had an inexpensive mitigation that was 差不多 (chà bù duō; “almost even” or “good enough”).

Insufferability

Up until Summer 2021, my rash was mostly an uncomfortable inconvenience. Thankfully, since everything shut down for the COVID pandemic, I didn’t need to make public appearances with unsightly skin. The itchiness was the worst symptom, but nobody would see me scratch at home.

Then, around the end of June 2021, the rash got worse. My whole face turned red, and my splotches became itchier than ever. Worst of all, the Chinese cream no longer had much effect. Timing was lousy, too, since my wife and I were going to spend most of July in Seattle. I needed medical help ASAP. I needed to see either my primary care physician or an allergist. I called both offices to schedule appointments. The wait time for my primary doctor was 1.5 months, while the wait time for an allergist was 3 months! Even if I wouldn’t be in Seattle, I couldn’t see a doctor anyway.

My rash plateaued while in Seattle. It was not great, but it didn’t stop me from enjoying our visit. I was okay during a quick stop in Salt Lake City, too. However, as soon as I returned home to the Triangle, the rash erupted. It became utterly unbearable – to the point where I couldn’t sleep at night. I was exhausted. My skin was raw. I could not focus on any deep work. I hit the point of thorough debilitation.

When I visited my doctor on August 3, she performed a series of blood tests. Those confirmed what the problem was not:

  1. My metabolic panel was okay.
  2. My cholesterol was okay.
  3. My thyroid was okay.
  4. I did not have celiac disease (gluten intolerance).
  5. I did not have hepatitis C.

Nevertheless, these results did not indicate any culprit. My doctor then referred me to an allergist for further testing on August 19.

The two weeks between appointments was hell. I was not allowed to take steroids or antihistamines for one week before the allergy test. My doctor also prescribed me hydroxyzine under the presumption that the root cause was an allergy. Unfortunately, I did not react well to hydroxyzine. It did nothing to relieve the rash or the itching. Even though I took it at night, I would feel off the next day, to the point where I literally could not think critically while trying to do my work. It affected me so badly that I accidentally ran a red light. During the two weeks between appointments, I averaged about 4 hours of sleep per night. I had to take sick days off work, and on days I could work, I had erratic hours. (Thankfully, my team, manager, and company graciously accommodated my needs.) I had no relief. The creams did nothing. I even put myself on an elimination diet in a desperate attempt to avoid potential allergens.

If you saw this tweet, now you know the full story behind it:

A Possible Answer

On August 19, 2021, the allergist finally performed a skin prick allergy test. A skin prick test is one of the fastest, easiest ways to reveal common allergies. The nurse drew dots down both of my forearms. She then lightly scratched my skin next to each dot with a plastic nub that had been dunked in an allergen. After waiting 15 minutes, she measured the diameter of each spot to determine the severity of the allergic reaction, if any. She must have tested about 60 different allergens.

The results yielded immediate answers:

  • I did not have any of the “Big 8” food allergies.
  • I am allergic to cats and dogs, which I knew.
  • I am allergic to certain pollens, which I suspected.
  • I am allergic to certain fungi, which is no surprise.

Then, there was the major revelation: I am allergic to dust mites.

Once I did a little bit of research, this made lots of sense. Dust mites are microscopic bugs that live in plush environments (like mattresses and pillows) and eat dead skin cells. The allergy is not to the mite itself but rather to its waste. They can appear anywhere but are typically most prevalent in bedrooms. My itchiness always seemed strongest at night while in bed. The worst areas on my skin were my face and upper torso, which have the most contact with my bed pillows and covers. Since I sleep in bed every night, the allergic reaction would be recurring and ongoing. No wonder I couldn’t get any relief!

I don’t yet know if eliminating dust mites will completely cure my skin problems, but the skin prick test at least provides hard evidence that I have a demonstrable allergy to dust mites.


Lessons for Software Testing

After nearly two years of suffering, I’m grateful to have a probable root cause for my rash. Nevertheless, I can’t help but feel frustrated that it took so long to find a meaningful answer. As a software tester, I feel like I should have been able to figure it out much sooner. Reflecting on my journey reminds me of important lessons for software testing.

First and foremost, formal testing with wide coverage is better than random checking. When I first got my rash, I tried to address it based on intuition. Maybe I should stop eating mangoes? Maybe I should change my shower soap, or my laundry detergent? Maybe eating probiotics will help? These ideas, while not entirely bad, were based more on conjecture than evidence. Checking them took weeks at a time and yielded unclear results. Compare that to the skin prick test, which took half an hour in total time and immediately yielded definite answers. So many software development teams do their testing more like tossing mangoes than like a skin prick test. They spot-check a few things on the new features they develop and ship them instead of thoroughly covering at-risk behaviors. Spot checks feel acceptable when everything is healthy, but they are not helpful when something systemic goes wrong. Hard evidence is better than wild guesses. Running an established set of tests, even if they seem small or basic, can deliver immense value in short time.

When tests yield results, knowing what is “good” is just as important as knowing what is “bad.” Frequently, software engineers only look at failing tests. If a test passes, who cares? Failures are the things that need attention. Well, passing tests rule out potential root causes. One of the best results from my allergy test is that I’m not allergic to any of the “Big 8” food allergies: eggs, fish, milk, peanuts, shellfish, soy, tree nuts, and wheat. That’s a huge relief, because I like to eat good food. When we as software engineers get stuck trying to figure out why things are broken, it may be helpful to remember what isn’t broken.

Unfortunately, no testing has “complete” coverage, either. My skin prick test covered about 60 common allergens. Thankfully, it revealed my previously-unknown allergy to dust mites, but I recognize that I might be allergic to other things not covered by the test. Even if I mitigate dust mites in my house, I might still have this rash. That worries me a bit. As a software tester, I worry about product behaviors I am unable to cover with tests. That’s why I try to maximize coverage on the riskiest areas with the resources I have. I also try to learn as much about the behaviors under test to minimize unknowns.

Testing is expensive but worthwhile. My skin prick allergy test cost almost $600 out of pocket. To me, that cost is outrageously high, but it was grudgingly worthwhile to get a definitive answer. (I won’t digress into problems with American healthcare costs.) Many software teams shy away from regular, formal testing work because they don’t want to spend the time doing it or pay the dollars for someone else to do it. I would’ve gladly shelled out a grand a year ago if I could have known the root cause to my rash. My main regret is not visiting an allergist sooner.

Finally, test results are useless without corrective action. Now that I know I have a dust mite allergy, I need to mitigate dust mites in my house:

  1. I need to encase my mattress and pillow with hypoallergenic barriers that keep dust mites out.
  2. I need to wash all my bedding in hot water (at least 130° F) (or freeze it for over 24 hours).
  3. I need to deeply clean my bedroom suite to eliminate existing dust.
  4. I need to maintain a stricter cleaning schedule for my house.
  5. I need to upgrade my HVAC air filters.
  6. I need to run an air purifier in my bedroom to eliminate any other airborne allergens.

In the worst case, I can take allergy shots to abate my symptoms.

Simply knowing my allergies doesn’t fix them. The same goes for software testing – testing does not improve quality, it merely indicates problems with quality. We as engineers must improve software behaviors based on feedback from testing, whether that means fixing bugs, improving user experience, or shipping warnings for known issues.


Next Steps

Now that I know I have an allergy to dust mites, I will do everything I can to abate them. I already ordered covers and an air purifier from Amazon. I also installed new HVAC air filters that catch more allergens. For the past few nights, I slept in a different bed, and my skin has noticeably improved. Hopefully, this is the main root cause and I won’t need to do more testing!