quality

The Behavior-Driven Three Amigos

Recently, my manager said to me, “Andy, your BDD documentation looks great, but could you please mention The Three Amigos?” A brief flash of panic came over me – I had never heard of “The Three Amigos” before. My immediate thought was the 1986 film of the same name or the Disney film The Three Caballeros. After a little research, I knew exactly who they were; I just didn’t know they had a name.

Who are “The Three Amigos”?

The Three Amigos” refers to a meeting of the minds of the three primary roles involved in producing software:

  1. Business – Often named the “business analyst” (BA) or “product owner” (PO), the business role provides what problem must be solved. They provide requirements for the solution. Typically, the business role is non-technical.
  2. Development – The developer role provides how the solution to the problem will be implemented. They build the software and must be very technical.
  3. Testing – The testing role, sometimes named “quality assurance” (QA), verifies that the delivered software product works correctly. They also try to find defects. The tester role must be somewhat technical.

During software development, The Three Amigos should meet regularly to discuss how the product will be developed. It is a shift left practice to avoid misunderstandings (like a game of telephone), thus improving quality and avoiding missed deadlines. The discussions should include only the individuals who will actually work on the specific deliverable, not the whole team.

While The Three Amigos seems most popular in Agile, it can be applied to any software development process. Some (here and here) advocate regularly scheduled formal meetings. Others (here and here) interpret it as an attitude instead of a process, in which the roles continuously collaborate. Regardless of implementation, The Three Amigos need to touch base before development begins.

Applying BDD

The Three Amigos fits perfectly into behavior-driven development, especially as part of BDD with Agile. Behavior scenarios are meant to foster collaboration between technical and non-technical roles because they are simple, high-level, and written in plain language. Given-When-Then provides a common format for discussion.

Ideally, when The Three Amigos meet during grooming and planning, they would formalize acceptance criteria as Gherkin features. Those feature files are then used directly by the developer for direction and the tester for automation. They act like a receipt of purchase for the business role – the feature file says, “This is what you ordered.”

Since BDD relies on feature files as artifacts, The Behavior-Driven Three Amigos must be more than just an attitude. The point of the collaboration is to produce feature files early for process efficiency. Less formal meetings could quickly devolve into all-talk-no-action.

Don’t Presume Anything

Don’t presume that the three roles will naturally collaborate on their own. I’ve seen teams in which the testers don’t participate in planning. I’ve also seen organizations in which automation engineers don’t help to write the test cases that they need to automate! Developers often abdicate responsibility for testing considerations, because “that’s QA’s job.” And, specifically for BDD, I’ve noticed that product owners resist writing acceptance criteria in Gherkin because they think it is too technical and beyond their role.

The Three Amigos exists as a named practice because collaboration between roles does not always happen. It is an accountability measure. Remember, the ultimate purpose for The Three Amigos is higher quality in both the product and the process. Nobody wants more meetings on their calendar, but everyone can agree that quality is necessary.

Why is Automation Full of Duplicate Code?

Copypasta
noun

A block of text that is duplicated repeatedly via "copy-paste",
    often causing annoyance or frustration.

One the the biggest problems (if not the biggest problem) I have seen in test automation is copypasta – the unnecessary duplication of code. It happens at all layers of testing. It happens in any type of project. It happens at companies big and small. And the consequences are stark: test development slows down, mistakes become more common, and maintenance becomes a nightmare. Although duplicate code can happen in any software project, it is especially prevalent in test automation. The reasons may or may not surprise you, but the solutions are clear.

4 Reasons Why Duplicate Code Pervades Automation

#1: Test cases are repetitive. For any given product, tests will share many of the same steps. For example, web app tests must all navigate to a start page at first, or API tests might cover a few variations for one call. Testing mechanics such as input parameters, setup/cleanup, logging, and assertions happen frequently. Put all of that together into test suites that have tens, hundreds, thousands, or even more test cases. It’s simply the nature of testing.

#2: Automation frameworks reinforce repetition. Most frameworks structure test cases as a class with methods (like JUnit) or as a collection of functions (like pytest), in which each method or function represents one test. Inherently, this basic structure is a good thing for making tests independent. However, lazy programmers may abuse the structure. Often, they put all test code inside these test methods, instead of extracting repetitive logic into helper methods or design patterns. Then, it becomes easier to simply duplicate an entire test case method and change a few things, rather than to implement a better overall design.

#3: Test code takes a backseat to product code. Business needs drive software development in the industry, and since test code is not part of the product delivered to customers, it is often deemed to be less important. Not as much devotion is given to developing good test code. Many best practices are abandoned for expediency.

#4: Testers often have weaker development skills. This is not a condemnation of testers, nor a universal labeling, but rather a distinction between disciplines: developers are developers because they are good at making software, and testers are testers because they are good at exercising software and finding bugs. Of course, I know plenty of testers who do indeed have strong dev skills. However, I also know solid testers who have limited programming experience. When automation responsibility falls upon testers with limited dev skills, poor development practices happen, and code duplication is typically rampant.

How to Avoid Duplicate Code in Automation

There are a number of ways to slay the copypasta monster. The first line of defense is to check yourself before you wreck yourself. Always question yourself when you copy-paste blocks of code. Why did you do that? What are you changing in the pasted copy? Should you abstract that logic into a method or a class that can be reused? Can you parameterize it? Override your Ctrl-C, Ctrl-V keyboard shortcut if necessary.

Be a good programmer. Develop packages for reusable actions. Things like assertions, logging, setup, and cleanup should be shared by all test cases. In that shared code, keep action calls short. Long method names with too many parameters inhibit usability. Remember that automated test cases should be self-documenting so that they read like test procedures. Whenever possible, make repetitive actions happen automatically. For example, make library methods do internal logging. For another example, I once wrote code to automatically reconnect SSH sessions whenever they dropped. These auto-actions allow test case code to focus less on the low-level mechanics and more on the high-level features under test.

Finally, be a team player. Use the same development practices for test code as for product code. Automation is a product, and its customers are the team. Use coding standards, design patterns, and revision control. Most importantly, reinforce good practices through code review. Use the review process as a constructive way to learn new tricks and even to mentor less experienced team members. Finally, divide testing roles between test formulation, test case automation, and test framework development. “QA” (quality assurance) is a wide discipline, and not everyone is equally skilled. Let people do what they do best. There is strength in diversity and in teamwork.

10 Things You Lose Without Automation

Automation has a lot of potential to improve software development. Unfortunately, though, automation is often seen as a luxury. Deadlines in the real word are unforgiving, and since test code isn’t product code, automation tasks are given lower priority and dunked into the black hole of the backlog. Some might argue that this is okay because it is lean or because a new project is just getting started. Once, I even heard it quipped that the first ones cut during a layoff are the automation folks. And it is true that automation requires a nontrivial resource investment.

However, I want to turn the tables. Instead of thinking about automation in terms of the opportunity, think about automation in terms of the opportunity cost. What happens if you don’t automate your tests from the get-go?  There are 10 major things you lose:

#1: Man Hours

Automated tests will automatically run.  Manual tests must be manually run.  That’s ontological.  If you only run a test one time, then automation has no return-on-investment.  But if you run a test more than once, automation saves a tester from repeating themselves. Plus, it’s easy: push the button and wait for results. Automated tests almost always run faster than manual tests, too.  Considering that time is money and engineer salaries aren’t cheap, man hours are a clear opportunity cost.

#2: Coverage

Automated tests can achieve greater coverage than manual tests, particularly for regression testing. As product development progresses, the sheer number of test cases increases. For example, in Agile, new tests will be created every sprint. Older tests must be run periodically to verify that new features don’t break existing features. If regression tests are manual, then testers must burn hours grinding through the same tests repeatedly.  Often, for expediency, this means that they skip some tests – not in the sense of being lazy, but rather as part of a risk-based approach.  Weaker coverage plus risk of missing bugs are accepted for the sake of shorter testing time.  If those regression tests were automated, then there would be no reason to shrink coverage, because they would be easy to run.

#3: Consistency

People make mistakes. It’s human nature – nobody’s perfect. And manual tests are prone to human error because humans run them. I remember how nervous I felt running manual on-call system checks at MaxPoint for the first time, afraid that I would miss a problem that could bring down a million-dollar bidding system.  Automated scripts run the same way every time.

#4: Protection

Continuous integration (CI) protects code against defects by building and testing every code change in real time. A CI system will automatically trigger tests all the time.Tests not running in CI (like manual tests) are effectively dead. At NetApp, failing code changes would immediately be kicked out of the code line, making automated tests act like a vaccine against bugs. On the other hand, I remember a project at MaxPoint that was riddled with bugs and perpetually delayed. When I asked the developers to see their unit tests, they said they never wrote unit tests because “it wasn’t a requirement.”

#5: Delivery Time

Continuous delivery (CD) is the natural extension of continuous integration, in which software products can automatically be delivered (and potentially even deployed) as the final step in a CI pipeline. This is how big companies like Google, Facebook, and Netflix can deliver so rapidly. No automation means no CD.

#6: Results and Metrics

Non-engineers (managers, product owners, scrum masters, oh my!) love to ask questions about tests.  “Are we red or green?” “How many tests do we have for this feature?” “What’s our coverage?” “How often do we run the tests?” Automated tests simply yield more accurate and more comprehensive results. Automation can also generate test reports, so engineers don’t need to waste time drafting emails or updating wiki pages.

#7: Accountability

Numbers don’t lie. Scripts don’t lie. Engineers typically don’t lie, but… results from manual tests can have a fudge factor, or a mistake in reporting, or any other sort of inconsistency. Inaccurate results may lead to poor business decisions. Automated results tell it like it is.

#8: Creativity

Manual testing can devolve into repetitive, menial labor: just follow steps 1-10 again and again and again. It would be much more effective for manual testers to focus on exploratory testing rather than deterministic testing. While automated tests can cover the fixed, repetitive test scenarios, exploratory testing lets testers find creative ways to uncover defects and judge how well a product actually works. Lack of automation ties up human capital.

#9: Peace of Mind

Are you sure that your product is “good”? Can you run enough tests to make sure? I learned the value of peace of mind while I was still in college. In my compiler theory course, I had to develop a simple programming language and build a compiler for it. Every week, we had to add new language features: arithmetic, strings, arrays, functions, etc. And every week, I wrote a slew of mini-programs to test grammar updates to my new language. By the time the project was complete, I had 1000+ automated test cases running through JUnit with 100% coverage, and the entire suite took a mere few minutes to run. And there were many late nights when the tests caught bugs in my language right away before committing code. There was no way I could have passed that class without my automated tests.

#10: Quality

The ultimate purpose of test automation is product quality. Having automation doesn’t necessarily mean product quality is good, but not having automation severely limits how quality can be pursued. Anecdotally, I’ve seen much better code quality come out of projects that have good test automation than ones without it. If I were a product owner, I know what I would want.