DjangoCon 2019 Break

24 Lessons from Working in Software

Let’s face it: working in software is tough. I’ve worked in software for over a decade, and I’ve learned a lot of things during my journey – oftentimes the hard way.

On April 7, 2022, I tried something new: every hour for 24 hours, I tweeted a lesson I learned during my software career. I covered topics like breaking into the industry, working on projects, and seeking advancement. I used the hashtag #PandyEveryHour to track my tweets. I started at 8am (US Eastern) on April 7, and I concluded at 8am on April 8. It was an awesome experience: lots of folks liked, retweeted, and commented. Some of my insights struck nerves.

In case you missed it while my tweets went live, I compiled all the tweets below in order. Give them a read! If you like any of them in particular, be sure to “like” them on Twitter so I know!

Breaking into the software industry

Coding

Working with others

Improving your skills

Career advancement

Conclusion

So, which lessons resonated the most with you? Which did you disagree with? Let me know!

The main photo for this article was taken at DjangoCon 2019 during one of the breaks between sessions.

Want to practice test automation? Try these demo sites!

One of the biggest struggles in learning how to develop top-notch automation is practice. Testing is as much an art as it is a science. It takes time to discern when to add explicit waits, how to craft robust locators, and why to verify one element over another. It also requires apps with specific elements or endpoints to try certain operations. Unfortunately, although resources for learning how to automate tests (like Test Automation University) are abundant these days, public demo web sites for practicing remain elusive. I’ve struggled to find ones that I like, and others frequently ask me for recommendations.

Which demo sites are good?

Below is a list of demo sites I’ve found either by searching online or through recommendations from friends. Many of these sites appear on other “Top N Demo Sites for Testing” articles as well. My list aims not only to provide links to popular demo sites but also to provide recommendations on how to use them.

Types of sites:

  • Web UI site – looks like a real web site
  • Web UI elements – tutorial pages showcasing web element types
  • Mobile UI site – looks like a real mobile site
  • API site – provides public APIs for testing
  • DIY – “do it yourself”; you must set up and run the demo site yourself
Demo SiteTypeDescription
ParaBankWeb UI & API siteAn online banking site from Parasoft with login and REST/SOAP APIs. You can also access the database if you run the project locally using the source code.
Restful BookerWeb UI & API siteAn online site for bed & breakfast bookings from Mark Winteringham. The frontend is a React app (source), and the backend is a REST API (source).
Automation Practice WebsiteWeb UI siteA basic online store with optional login from SeleniumFramework.com. Great for example web UI tests.
Luma – Magento eCommerceWeb UI siteAnother online store with several pages.
DemoblazeWeb UI siteA basic online store with optional login from BlazeMeter. Great for example web UI tests.
Swag LabsWeb UI siteA basic online store with required login from Sauce Labs. Great for example web UI tests.
Applitools demo siteWeb UI siteA small site with login page and home page from Applitools. Compare/contrast with a second version for visual testing.
Automation BookstoreWeb UI siteA one-page site for dynamically searching book titles. Good for testing responsive design in short demos.
JPetStore DemoWeb UI siteA pet store site from MyBatis built on top of MyBatis 3, Spring 3, and Stripes (source).
GlobalSQA Banking ProjectWeb UI siteA basic banking app with dropdown login and simple pages from GlobalSQA.
Gatling Computers DatabaseWeb UI siteA one-page site from Gatling that provides a paginated list of computer models with the ability to filter and add new computers.
CandyMapperWeb UI siteA Halloween-themed site from Paul Grossman that shows scary bugs. It also comes with a second release that has the fixes.
Bulldoggy: The Reminders AppDIY Web UI siteA small web app I built myself in Python using FastAPI and HTMX. It has login and uses a TinyDB database.
OWASP Juice ShopDIY Web UI siteA test site written entirely in JavaScript using Node.js, Express and Angular. Specifically built for testing security vulnerabilities.
Cypress Real-World AppDIY Web UI siteA fake payment app from Cypress meant for demonstrating real-world Cypress testing. Could be used for other purposes.
RealWorld example appsDIY Web UI siteOne demo app implemented in multiple languages and frameworks. Not developed for testing but could nevertheless be used.
the-internetWeb UI elementsA site from Dave Haeffner and Elemental Selenium with several concise examples of web elements and interactions.
Selenium Test PagesWeb UI elementsA site with several pages that have slightly deeper examples than the-internet.
LetCodeWeb UI elementsA set of very clean pages along with video tutorials explaining how to automate interactions.
DemoQAWeb UI elementsA practice site from ToolsQA that includes pages for elements, forms, frames, interactions, and even a small bookstore.
Ultimate QA Automation PracticeWeb UI elementsA set of rich practice pages from Ultimate QA.
UI Test Automation PlaygroundWeb UI elementsA set of educational pages with interactable elements from the Rapise team.
SelectorsHub Practice PageWeb UI elementsA practice page from SelectorsHub for interacting with different kinds of web elements.
WebDriverUniversity.comWeb UI elementsAnother set of educational pages with interactable elements.
Sauce Labs Native Sample ApplicationDIY Mobile UI siteA mobile app from Sauce Labs for Android and iOS that is very similar to the Swag Labs web site.
JSONPlaceholderAPI siteA public REST API for generating fake data from a set of predefined resources. Follow the guide to learn how to make requests.
Swagger PetstoreAPI siteA public REST API from Swagger for testing authentication and CRUD operations for pet store data.
Public APIsAPI siteA long list of public APIs that can be used for testing.
Device Registry ServiceDIY API siteA Flask app I developed for teaching how to test REST APIs. Includes the app and automated API tests, which can both be run locally.
Best Buy API PlaygroundDIY API siteA JavaScript-based REST API demo service from Best Buy.

Which ones do you recommend most?

There are lots of demo sites above. Here are the ones I’d personally recommend for different needs:

Of course, the others are still good. That’s why they made the list!

Why bother with “fake” sites?

You might be wondering, “Why do we need demo sites? Why don’t we just use real sites?” Well, demo or “fake” web sites meet a few needs that real sites cannot:

  • Demo sites offer consistency. They are implemented one way and then do not change. Folks can trust that their tests will always work on them.
  • Demo sites are often simpler than real sites. They feel less intimidating for newcomers.
  • Demo sites can be designed for instruction. If they are part of a tutorial, the author can add features to the site to demonstrate concepts.
  • Demo sites are safer for publications like articles, tutorials, and books. Written content is static, so any sites referenced by examples should also be static. Real sites could change.
  • Real sites might require end user agreements that forbid automated requests. Some may even throttle or block requests if they suspect the requests come from a “bot.”
  • Real sites might also bear legal or copyright implications, especially if a company is using the sites for their own content.

What limitations do demo sites have?

Unfortunately, demo sites do have limitations:

  • Demo sites may be too simplified. They may lack large workflows or real-world data. Folks might become frustrated when they discover inactive elements that appear to be real.
  • Demo sites may not be built to scale. High request volume or parallel testing scale might cripple them.
  • Demo sites may seem like they have poor quality, whether or not they actually have poor quality. Sometimes, they are built quickly for testing purposes and therefore don’t have the attention to detail as real sites.
  • Demo sites with significant company branding may be inappropriate to use. For example, if A and B are competitors, then A shouldn’t use B’s demo site for their product tutorials.

Other great lists

Here are other great articles that list demo sites for testing. I referred to them while compiling this list:

Do you have other demo sites to recommend? Put them in the comments below!

Welcome to Raleigh

Thoughts on Remote Work

At my new job as Developer Advocate at Applitools, I now permanently work remotely. Working from home is new to me. At my previous company PrecisionLender (later acquired by Q2), I worked in an office that was a seven-minute drive from my house. My office was a real office: an enclosed room with a door, walls, a large desk, and two huge whiteboards. Working from the office was a joy. We had free snacks and sodas. My colleagues and I all got along great, and we’d go out to lunch almost every day. After COVID-19 hit the USA in March 2020, I was heartbroken to abandon the office. I was so stubborn about eventually returning to the office that I refused to set up a decent workstation at home. For a year and a half, I worked exclusively on my laptop either from a desk in my living room or from the couch – no monitors, no mouse, and no separate keyboard. I mean, come on, the pandemic would be over in a few short months, right?

However, my attitudes about remote work are changing. At PrecisionLender, I was spoiled with the best kind of office situation someone could have. Many others in the software field are not so fortunate. Those who live in the big cities like New York and San Francisco pay thousands of dollars a month to split tiny apartments and spend an hour or more on one-way commutes. Not to mention, many of them move away from family and friends to take those opportunities.

Office layout trends in recent years have also been dismal. The open layout – no walls with clusters of desks sloppily smashed together – is the norm. Perpetual audio and visual distractions are a nightmare for anyone trying to concentrate. Widescreen monitors and noise-cancelling headphones only help a little. I experienced that frustration at MaxPoint and LexisNexis. Cubicles, while passé these days, are only a small step up from open floors. While the “walls” block out visual distractions, you can still hear everything that goes on around you. Plus, the enormous blocks of cubicles on a floor feel brutally confining. I frequently called the IBM 500 complex in RTP “Cubicle City” when I worked there.

Remote work can be a godsend for many hardworking folks who just want to do a good job while doing right by themselves. Folks, especially in software, can literally work anywhere. COVID-19 shutdowns have proven it. My wife and I even took advantage of it: we bought a vacation condo in Seattle, and I worked remotely from there for several weeks. I could automate tests during the day, grab beers with close friends, and catch a Mariners game on the weekend. Even with a 3-hour time zone difference, I was right in step with my team the whole time.

The rooftop of my Seattle condo building
My Seattle condo building’s rooftop

Recently, I saw a Twitter thread by Chris Herd in which he made several predictions on remote work in 2022 and beyond:

This thread really resonated with me. While I still have some concerns about remote work, I do believe it will be a net good for not only my country and people but also for the world. I’d like to highlight some of the points he made that really stuck out to me.

I agree completely with this, and I’m excited to see it happen. The US has so many awesome places, from sea to shining sea. There’s no reason why someone should need to move from their hometown to a big city just to work for a specific company (unless, of course, they want that adventure). Places like Raleigh-Durham, Nashville, Denver, Miami, Pittsburgh, Salt Lake City, Cleveland, and countless other smaller American cities are fantastic places to live. Each has a unique vibe and local specialties. Allowing folks to live where they want with good employment opportunities will rejuvenate local and regional economies. It will bring broader prosperity to all.

This is happening already. The Triangle of NC, where I live, is booming. My wife and I rent local listings on Airbnb, and all through the pandemic, the #1 type of guest we’ve hosted by far is someone interested in relocating to the Triangle.

I hadn’t thought about asynchronous work until Chris mentioned it, but he’s right. Life is 24/7, and people live all around the globe. Remote work can liberate us from a harsh 9-5 and instead embrace more flexible hours. For example, this past week, I had to work around the clock for a few days prepping a big webinar. However, I wasn’t working for 16 hours straight each day. I woke up a bit late, did some light administrative activities before lunch, hit it hard all afternoon, stopped for dinner and time with my puppy, and then burned the midnight oil after dark. That’s what worked for me.

Yes! I’m living it myself. My wife and I just adopted a French bulldog puppy, and I also started restoring a vintage 1970 Volkswagen Beetle.

VW Beetle Engine Swap
I swapped the engine out of my vintage 1970 Volkswagen Beetle
Suki the Frenchie
Suki the French bulldog is the cutest puppy in the world!

I sure hope so! Break down those entry barriers. Do away with the gatekeepers. End coastal elitism.

I sure hope this one becomes true, but I’m skeptical.

This one is so ironic. Based on many articles I’ve read, upper management seems to despise work-from-home because they think their workers will work less. However, all the good, hardworking folks I know are actually working more than ever. Not only are folks more efficient, but they tend to keep working just a little extra because there are fewer barriers between work stuff and personal stuff. In my new job, I know I’m certainly working more hours right now than I probably should, and I’m hoping to settle into a more sustainable routine as I acclimate to my new role. Nevertheless, we absolutely need to protect ourselves against unintentional burnout.

One big concern I have with remote work is the loss of in-person contact. Humans need to be around each other, and screens are a poor substitute. I know I’ve felt lonelier working from home than when I worked in an office. That’s why I absolutely love this idea about regular remote retreats. Companies can still bring folks together, and they can hire professionals who know how to maximize that time. I go to software conferences frequently for similar experiences.

I think this goes hand-in-hand with async work. However, unless safeguards are put in place to prevent overworking, then life-work balance could quickly become work-more-work imbalance.

Honestly, this one was never much of a problem for me. The only “paddings” I’ve had in my workdays are long, boring meetings where I participated little. Thankfully, those were confined mostly to my days at IBM and NetApp earlier in my career.

YES! Even though my commutes have always been relatively short, I still appreciate the time I get back in my day. Here are some things I’ve done with my extra, more flexible time:

  • Sleep in a little later
  • Play with my puppy when I need a break
  • Share lunch nearby with friends
  • Run a quick errand in my Beetle
  • Grab an impromptu bubble tea with my wife
  • Take a walk around my neighborhood while thinking about work-related challenges

Documentation is the unspoken superpower of remote teams. I strongly value written communication for its clarity, searchability, and permanence.

Chris had lots of other good tweets in that thread, so be sure to read them. Overall, his tone was bullishly optimistic on remote work – not only in its benefits, but also in its inevitability. Personally, I hope remote work is here to stay, and that it’s not just a phase brought on by the COVID-19 pandemic. It’s a game-changer for individuals and industries alike. However, I’m still somewhat fearful that big companies will reign in wide scale returns to the office after the pandemic. Giants have already sunk millions to billions in commercial real estate for their offices that they can’t get back. I’m also worried about salaries in different geographies. Employers already pay folks who live outside of big cities much less for the same work due to cost-of-living differences. I don’t think that will be fair as people leave big cities in droves for smaller cities or the countryside. Already, the cost of living is skyrocketing in the Triangle, yet our average tech salaries are far below those of Silicon Valley. We’ll see what happens.

Overall, I am optimistic about remote work just like Chris.

So, what do y’all think? Please leave comments below. I’d love to learn different perspectives!

Playwright Workshop for TAU: The Homecoming

Want to learn Playwright with Python? Take this workshop!

Playwright is an awesome new browser automation library. With Playwright, you can automate web UI interactions for testing or for web scraping with a concise, uniform API in one of four languages: Python, C#, Java, and JavaScript. Playwright is also completely open source and backed by Microsoft. It’s a powerful alternative to Selenium WebDriver.

On December 1, 2021, I delivered a workshop on Playwright for TAU: The Homecoming. In my workshop, I taught how to build a test automation project in Python using Playwright with pytest, Python’s most popular test framework. We automated a test case together for performing a DuckDuckGo web search.

If you missed the workshop, no worries: You can still take the workshop as a self-guided tutorial! The workshop instructions and example code are located in this GitHub repository:

https://github.com/AutomationPanda/tau-playwright-workshop

To take the workshop as a self-guided tutorial, read the repository’s README, and then follow the instructions in the Markdown guides under the workshop folder. The workshop has five main parts:

  1. Getting started
    1. What is Playwright?
    2. Our web search test
    3. Test project setup
  2. First steps with Playwright
    1. Browsers, contexts, and pages
    2. Navigating to a web page
    3. Performing a search
  3. Writing assertions
    1. Checking the search field
    2. Checking the result links
    3. Checking the title
  4. Refactoring using page objects
    1. The search page
    2. The result page
    3. Page object fixtures
  5. Nifty Playwright tricks
    1. Testing different browsers
    2. Capturing screenshots and videos
    3. Running tests in parallel

If you get stuck or have any questions, please open issues against the GitHub repository, and I’ll try to help. Happy coding!

A Simple High-Quality Mic Setup for Software Pros

I’ve always been frustrated with poor quality audio recordings. Microphones built into laptops are very convenient, but they usually yield tinny sound lacking the depth of real voices. When I asked my audiophile friends for advice, they recommended studio-level equipment that was beyond my comprehension and my budget. As a software guy, I just wanted an audio setup that captured high-quality audio while still being convenient for everyday usage. I’d need it for remote meetings as well as for recording talks and tutorials. I was willing to pay for good equipment as long as I could use it well. Unfortunately, my biggest frustration was ignorance. I didn’t know anything about recording.

Here’s what I finally found to work well for me:

  1. A Blue Yeti microphone
  2. A Blue Compass boom arm
  3. A Blue Radius III shockmount
  4. A foam microphone windscreen

When I did my research, the Blue Yeti microphone was at the top of everyone’s recommendation list. The nicest thing about a Blue Yeti mic is that it connects via USB. You simply plug it right into your laptop and select audio input and output channels. The mic doesn’t need an external power source, either.

Initially, I bought only the Blue Yeti mic and the foam windscreen. Instead of using the boom arm, I used the tabletop stand that came with the mic for all my recordings. This was a big mistake. The tabletop stand picked up a lot of local noise, like typing on a keyboard. The audio quality it picked up also sounded like it had a bit of an echo, which may have been due to sound bouncing off the tabletop or my hardwood floors.

The boom arm and shockmount made a huge improvement in recording quality. Plus, with the boom arm mounted firmly to my desk, I can easily move it towards me for recording or out of the way otherwise. It feels quite sturdy. I chose to use all Blue products so that I could be certain that they’d work together. You can save quite a bit of money if you buy the microphone, boom arm, and shockmount together as a bundle on Amazon (~$200), instead of a la carte like I did (~$250).

My audio recording setup: a Blue Yeti microphone mounted onto a Blue Radius III Shockmount dangling from a Blue Compass boom arm. Pardon the mess on my desk – it’s a work in progress!

I plan to get a docking station for my laptop so that I can plug the microphone’s USB cable into the dock, simplifying desk’s cable management.

Even with this new setup, I still felt like I didn’t understand how to use my Blue Yeti microphone to its full potential. Thankfully, YouTube came to the rescue! This video greatly helped me understand things like tuning the mic’s gain and positioning the mic while speaking:

In summary, here’s what I like about the setup:

  • It yields very good (maybe professional?) audio recording quality.
  • It is simple enough for anyone to set up and use.
  • It is convenient to use for remote meetings, video recordings, etc.
  • It feels quite sturdy.
  • It is relatively affordable (compared to other audio equipment).

Please note: I am not an expert in audio equipment, and this article is not sponsored by any company. I simply hope that someone can benefit from the things I learned (and possibly save a few bucks) if they want to improve their own recording game!

Boa Constrictor’s Awesome Hacktoberfest 2021

Boa Constrictor is the .NET Screenplay Pattern. It helps you make better interactions for better automation! Its primary use case is Web UI and REST API test automation, but it can be used to automate any kind of interactions. The Screenplay Pattern is much more scalable for development and execution than the Page Object Model.

The Boa Constrictor maintainers and I strongly support open source software. That’s why we participated in Hacktoberfest 2021. In fact, this was the second Hacktoberfest we did. We launched Boa Constrictor as an open source project a year ago during Hacktoberfest 2020! We love sharing our code with the community and inspiring others to get involved. To encourage participation this year, we added the “hacktoberfest” label to open issues, and we offered cool stickers to anyone who contributed.

Boa Constrictor sticker
Boa Constrictor: The .NET Screenplay Pattern
Sticker Medallion

Hacktoberfest 2021 was a tremendous success for Boa Constrictor. Even though the project is small, we received several contributions. Here’s a summary of all the new stuff we added to Boa Constrictor:

  • Updated WebDriver interactions to use Selenium WebDriver 4.0
  • Implemented asynchronous programming for Tasks and Questions
  • Extended the Wait Task to wait for multiple Questions using AND and OR logic
  • Standardized ToString methods for all WebDriver interactions
  • Automated unit tests for WebDriver Questions
  • Wrote new user guides for test framework integrations and interaction patterns
  • Made small refinements to the doc site
  • Created GitHub templates for issues and pull requests
  • Replaced the symbols NuGet package with embedded debugging
  • Added the README to the NuGet package
  • Added Shields to the README
  • Restructured projects for docs, logos, and talk

During Hacktoberfest 2021, we made a series of four releases because we believe in lean development that puts new features in the hands of developers ASAP. The final capstone release was version 2.0.0: a culmination of all Hacktoberfest work! Here’s a view of the Boa Constrictor NuGet package with its new README (Shields included):

The Boa Constrictor NuGet package with the new README and Shields
The Boa Constrictor NuGet package with the new README and Shields

If you like project stats, then here’s a breakdown of the contributions by numbers:

  • 11 total contributors (5 submitting more than one pull request)
  • 41 pull requests closed
  • 151 commits made
  • Over 10K new lines of code

GitHub’s Code Frequency graph for Boa Constrictor shown below illustrates how much activity the project had during Hacktoberfest 2021. Notice the huge green and red spikes on the right side of the chart corresponding to the month of October 2021. That’s a lot of activity!

Hacktoberfest Contributions
The GitHub Code Frequency Graph for Boa Constrictor

Furthermore, every member of my Test Engineering & Architecture (TEA) team at Q2 completed four pull requests for Hacktoberfest, thus earning our prizes and our bragging rights. For the three others on the team, this was their first Hacktoberfest, and Boa Constrictor was their first open source project. We all joined together to make Boa Constrictor better for everyone. I’m very proud of each of them individually and of our team as a whole.

Personally, I gained more experience as an open source project maintainer. I brainstormed ideas with my team, assigned work to volunteers, and provided reviews for pull requests. I also had to handle slightly awkward situations, like politely turning down pull requests that could not be accepted. Thankfully, the project had very little spam, but we did have many potential contributors request to work on issues but then essentially disappear after being assigned. That made me appreciate the folks who did complete their pull requests even more.

Overall, Hacktoberfest 2021 was a great success for Boa Constrictor. We added several new features, docs, and quality-of-life improvements to the project. We also got people excited about open source contributions. Many thanks to Digital Ocean, Appwrite, Intel, and DeepSource for sponsoring Hacktoberfest 2021. Also, special thanks to Digital Ocean for featuring Boa Constrictor in their Hacktoberfest kickoff event. Keep on hacking!

Boa Constrictor is doing Hacktoberfest 2021!

Boa Constrictor is the .NET Screenplay Pattern. It helps you make better interactions for better automation! Its primary use case is Web UI and REST API test automation, but it can be used to automate any kind of interactions. The Screenplay Pattern is much more scalable for development and execution than the Page Object Model.

My team and I at Q2 developed Boa Constrictor for testing the PrecisionLender web app. Originally, we developed it internally as part of our C# test automation solution named “Boa”, but we later released it as an open source project on GitHub so that others could use it. In fact, we released it publicly in October 2020 during last year’s Hacktoberfest!

We are delighted to announce that Boa Constrictor will participate in Hacktoberfest 2021. Open source software is vital for our industry, and we strongly support efforts like Hacktoberfest to encourage folks to contribute to open source projects. Many thanks to Digital Ocean, Appwrite, Intel, and DeepSource for sponsoring Hacktoberfest again this year.

So, how can you contribute to Boa Constrictor? Take these four easy steps:

  1. Start by learning about the project.
  2. Read our guide to contributing code.
  3. Clone the GitHub repository.
  4. Look for unassigned open issues labeled “hacktoberfest”.
    1. Or, open an issue to propose a new idea!
  5. Add a comment to the issue saying that you’d like to do it.

To encourage contributions, I will give free Boa Constrictor stickers to anyone who makes a valid pull request to the project during Hacktoberfest 2021! (I’ll share a link where you can privately share your mailing address. I’ll mail stickers anywhere in the world – not just inside the United States.) The sticker is a 2″ medallion that looks like this:

Boa Constrictor sticker
The Boa Constrictor Sticker

Remember, you have until October 31 to make four qualifying pull requests for Hacktoberfest. We’d love for you to make at least one of those pull requests for Boa Constrictor.

How Q2 uses BDD with SpecFlow for testing PrecisionLender

This case study was written by Andrew Knight, Lead Software Engineer in Test for Q2’s PrecisionLender product, in collaboration with Q2 and Tricentis. It explains the PrecisionLender team’s continuous testing journey and how SpecFlow served as a cornerstone for success.

What is PrecisionLender?

PrecisionLender is a web application that empowers commercial bankers with in-the-moment insights that help them structure and price commercial deals. Andi®, PrecisionLender’s intelligent virtual analyst, delivers these hyper-focused recommendations in real-time, allowing relationship managers to make data-driven decisions while pricing their commercial deals. PrecisionLender is owned and developed by Q2, a financial experience software company dedicated to providing digital banking and lending solutions to banks, credit unions, alternative finance, and fintech companies in the U.S. and internationally.

The PrecisionLender Opportunity Screen
(Picture taken from the PrecisionLender Support Center)

The starting point

The PrecisionLender team had a robust Continuous Integration (CI) delivery pipeline with strong unit test coverage, but they lacked end-to-end feature coverage. Developers would fill this gap by manually inspecting their changes in a shared development environment. However, as the PrecisionLender app grew, manual checks could not cover all possible integrations. The team knew they needed continuous automated testing to provide a safety net for development to remain lean and efficient. In April 2018, they hired Andrew Knight as their first Software Engineer in Test (SET) – a new role for the company – to lead the effort.

Automating tests with SpecFlow

The PrecisionLender team developed the Boa test solution – a project for automating end-to-end tests at scale. Boa would become PrecisionLender’s internal platform for test automation development. The name “Boa” is a loose acronym for “Behavior-Oriented Automation.”

The team chose SpecFlow to be the core framework for Boa tests. Since the PrecisionLender app’s backend is developed using .NET, SpecFlow was a natural fit. SpecFlow’s Gherkin syntax made tests readable and understandable, even to product owners and product support specialists who do not code.

The SpecFlow framework integrates with tools like Selenium WebDriver for testing Web UIs and RestSharp for testing REST APIs to exercise vital pathways for thorough app coverage. SpecFlow’s dependency injection mechanisms are solid yet simple, and the online docs are thorough. Plus, SpecFlow is an open-source project, so anyone can look at its code to learn how things work, open requests for new features, and even offer code contributions.

An example Boa test, written in Gherkin using SpecFlow.

Executing tests with SpecFlow+ Runner

Writing good tests was only part of the challenge. The PrecisionLender team needed to execute Boa tests continuously to provide fast feedback on changes to the app. The team chose to run Boa tests using SpecFlow+ Runner, which is tailored for SpecFlow tests. The team uses SpecFlow+ Runner to launch tests in parallel in TeamCity any time a developer deploys a code change to internal pre-production environments. The entire test suite also runs every night against multiple product configurations. SpecFlow+ Runner produces a helpful test report with everything needed to triage test failures: pass-and-fail tallies overall and per feature, a visual execution timeline, and full system logs. If engineers need to investigate certain failures more closely, they can use SpecFlow tags and SpecFlow+ Runner profiles to selectively filter tests for reruns. SpecFlow+ Runner’s multiple features help the team expedite test execution and investigation.

The SpecFlow+ Runner report for a dozen smoke tests.

Sharing features with SpecFlow+ LivingDoc

Good test cases are more than just verification procedures – they are behavior specifications. They define how features should work. Instead of keeping testing work siloed by role, the PrecisionLender team wanted to share Boa tests as behavior specs with all stakeholders to foster greater collaboration and understanding around features. The team also wanted to share Boa tests with specific customers without sharing the entire automation code.

SpecFlow+ LivingDoc enabled the PrecisionLender team to turn Gherkin feature files into living documentation. Whereas the SpecFlow+ Runner report focuses on automation execution, the SpecFlow+ LivingDoc report focuses on behavior specification apart from coding and automation details. LivingDoc displays Gherkin scenarios in a readable, searchable way that both internal folks and customers can consume. It can also optionally include high-level pass-and-fail results for each scenario, providing just enough information to be helpful and not overwhelming. LivingDoc has also helped PrecisionLender’s engineers identify and eliminate unused step definitions within the automation code. PrecisionLender benefits greatly from complementary reports from SpecFlow+ Runner and SpecFlow+ LivingDoc.

The SpecFlow+ LivingDoc report for a dozen smoke tests with their pass-and-fail results.

Improving interactions with Boa Constrictor

The Boa test solution initially used the Page Object Model to model interactions with the PrecisionLender app. However, as the PrecisionLender team automated more and more Boa tests, it became apparent that page objects did not scale well. Many page object classes had duplicative methods, making automation code messy. Some methods also did not include appropriate waiting mechanisms, introducing flaky failures.

PrecisionLender’s SETs developed Boa Constrictor, a .NET implementation of the Screenplay Pattern, to make better interactions for better automation. In Screenplay, actors use abilities to perform interactions. For example, an ability could be using Selenium WebDriver, and an interaction could be clicking an element. The Screenplay Pattern can be seen as a refactoring of the Page Object Model that minimizes duplicate code through a better separation of concerns. Individual interactions can be hardened for robustness, eliminating flaky hotspots. The Boa test solution now exclusively uses Boa Constrictor for interactions.

In October 2020, Q2 released Boa Constrictor as an open-source project so that anyone can use it. It is fully compatible with SpecFlow and other .NET test frameworks, and it provides rich interactions for Selenium WebDriver and RestSharp out of the box.

Boa Constrictor, the .NET Screenplay Pattern.

Scaling massively with Selenium Grid

When the PrecisionLender team first started automating Boa tests, they ran tests one at a time. That soon became too slow since the average Boa test took 20 to 50 seconds to complete. The team then started running up to 3 tests in parallel on one machine, but that also was not fast enough. They turned to Selenium Grid, a tool for running WebDriver sessions remotely across multiple machines.

PrecisionLender built a set of internal Selenium Grid instances using Microsoft Azure virtual machines to run Boa tests at high scale. As of July 2021, PrecisionLender has over 1800 unique Boa tests that run across four distinct product configurations. Whenever TeamCity detects a code change, it triggers a “continuous” Boa test suite with over 1000 tests running 50 parallel tests using Google Chrome on Selenium Grid. It completes execution in about 10 minutes. TeamCity launches the full test suite every night against all product configurations with 64-100 parallel tests on Selenium Grid. Continuous Integration currently runs up to 10K Boa tests daily against the PrecisionLender app with SpecFlow+ Runner and Selenium Grid.

The Boa test solution architecture, including Continuous Integration through TeamCity and parallel testing with SpecFlow+ Runner and Selenium Grid.

Shifting left with BDD

Better testing and automation practices eventually inspired better development practices. Product owners would create user stories, but developers would struggle to understand requirements and business purposes fully. PrecisionLender’s SETs started bringing together the Three Amigos – business, development, and testing roles – to discuss product behaviors proactively while creating user stories. They introduced Behavior-Driven Development (BDD) activities like Example Mapping to explore behaviors together. Then, well-defined stories could be easily connected to SpecFlow tests written in Gherkin following Specification by Example (SBE). Teams repeatedly saved time by thinking before coding and specifying before testing. They built higher quality into features from the beginning, and they stopped before working on half-baked stories with unjustified value propositions. Developers who participated in these behavior-driven practices were also more likely to automate Boa tests on their own. Furthermore, one of PrecisionLender’s developers loved BDD practices so much that he joined the team of SETs! Through Gherkin, SpecFlow provided a foundation that enabled quality work to shift left.

Challenges along the way

Achieving true continuous testing had its challenges along the way. Intermittent failure was the most significant issue PrecisionLender faced at scale. With so many tests, environments, and infrastructural pieces, arbitrary failures were statistically unavoidable. The PrecisionLender team took a two-pronged approach to handle intermittent failures: (1) eliminate race conditions in automation using good interactions with Boa Constrictor, and (2) use SpecFlow+ Runner to automatically retry failed tests to determine if failures were consistent or intermittent. These two approaches reduced the frequency of flaky failures and helped engineers quickly resolve any remaining issues. As a result, Boa tests enjoy well above a 99% success rate, and most failures are due to actual bugs.

PrecisionLender app performance at scale was a second big challenge. Running up to 100 tests in parallel turned functional tests into de facto load tests. Testing at scale repeatedly uncovered performance bottlenecks in the app. Performance issues caused widespread test failures that were difficult to diagnose because they appeared intermittently. Still, the visual timeline and timestamps in the SpecFlow+ Runner report helped the team identify periods of failure that could be crosschecked against backend logs, metrics, and database queries. Developers resolved many performance issues and significantly boost the app’s response times and load capacity.

Training team members to develop solid test automation was the third challenge. At the start of the journey, test automation, Gherkin, and BDD were all new to PrecisionLender. The PrecisionLender SETs took active steps to train others on how to develop good tests and good automation through group workshops, Three Amigos meetings, and one-on-one mentoring sessions. They shared resources like the Automation Panda blog for how to write good tests and good Gherkin. The investment in education paid off: many developers have joined the SETs in writing readable, reliable Boa tests that run continuously.

Benefits to the business

Developing a continuous testing solution brought many incredible benefits to PrecisionLender. First, the quality of the PrecisionLender app improved because continuous testing provided fast feedback on failures that developers could quickly fix. Instead of relying on manual spot checks, the team could trust the comprehensive safety net of Boa tests to catch bugs. Many issues would be caught within an hour of a developer making a code commit, and the longest feedback cycle would be only one business day for the full nightly test suites to run. Boa tests catch failures before customers ever experience them. The continuous nature of testing enables PrecisionLender to publish new releases every two weeks.

Second, the high reliability of the Boa test solution means that the PrecisionLender team can trust test results. When a test passes, the behavior is working. When a test fails, there is a real bug. Reliability also means that engineers spend less time on automation maintenance and more time on more valuable activities, like developing new features and adding new tests. Quality is present in both the product code and the test code.

Third, continuous testing boosts customer confidence in PrecisionLender. Customers trust the software quality because they know that PrecisionLender thoroughly tests every release. The PrecisionLender team also shares SpecFlow+ LivingDoc reports with specific clients to prove quality.

A bright future

PrecisionLender’s continuous testing journey is not over. Since the PrecisionLender team hired its first SET, it has hired three more, in addition to a testing manager, to grow quality improvement efforts. Multiple development teams have written their own Boa tests, and they plan to write more tests independently. SpecFlow’s tools have been indispensable in helping the PrecisionLender team achieve successful quality assurance. As PrecisionLender welcomes more customers, the Boa solution will be ready to scale with more tests, more configurations, and more executions.

My Upside-Down QA Story for Global Testers Day

Happy Global Testers Day! For 2021, QA Touch is celebrating with webinars, games, competitions, blogs, and videos. I participated by sharing an “upside-down” story from years ago when I accidentally wiped out all of NetApp’s continuous integration testing. Please watch my story below. I hope you find it both insightful and entertaining!