development

In BDD, What Should Be A Feature?

How do I decide what a feature should be? And should I define a feature first before writing behavior specs, or should I start with behaviors and see how they fit together into features?

Features, scenarios, and behaviors are all common BDD terms that should be carefully defined:

A behavior is an operation with inputs, actions, and expected outcomes.
A scenario is the specification of a behavior using formal steps and examples.
A feature is a desired product functionality often involving multiple behaviors.

Don’t try to over-think the definition of “feature.” Some features are small, while other features are large. The main distinction between a feature and a scenario or behavior is that features are what customers expect to receive. Small features may cover only a few or even only one behavior, while large features may cover several.

The Gherkin language has Feature and Scenario sections. In this sense, a Feature is simply a collection of related Scenarios. They align roughly to the more general meanings of the terms.

Don’t over-think features with Agile, either. Some teams define a feature as a collection of user stories. Other teams say that one user story is a feature. In terms of Gherkin, don’t presume that one user story must have exactly one feature file with one Feature section. A user story could have zero-to-many feature files to cover its behaviors. Do whatever is appropriate.

Features should be determined by customer needs. They should solve problems the customers have. For example, perhaps the customer needs a better way to process orders through their online store. That’s where features should start – as business needs. Behaviors should then naturally come as part of grooming and refinement efforts. Thus, in most cases, features should be identified first before individual behaviors.

Nevertheless, there may be times during development that scenario-to-feature realignment should be done. It may be more convenient to create a new feature file for related behaviors. Or, a new feature may be “discovered” out of particularly useful behaviors. This is more the exception than the norm.

Django Settings for Different Environments

The Django settings module is one of the most important files in a Django web project. It contains all of the configuration for the project, both standard and custom. Django settings are really nice because they are written in Python instead of a text file format, meaning they can be set using code instead of literal values.

Settings must often use different values for different environments. The DEBUG setting is a perfect example: It should always be True in a development environment to help debug problems, but it should never be True in a production environment to avoid security holes. Another good example is the DATABASES setting: Development and test environments should not use production data. This article covers two good ways to handle environment-specific settings.

Multiple Settings Modules

The simplest way to handle environment-specific settings is to create a separate settings module for each environment. Different settings values would be assigned in each module. For example, instead of just one mysite.settings module, there could be:

mysite
`-- mysite
    |-- __init__.py
    |-- settings_dev.py
    |-- settings_prod.py
    `-- settings_test.py

For the DEBUG setting, mysite.settings_dev and mysite.settings_test would contain:

DEBUG = True

And mysite.settings_prod would contain:

DEBUG = False

Then, set the DJANGO_SETTINGS_MODULE environment variable to the name of the desired settings module. The default value is mysite.settings, where “mysite” is the name of the project. Make sure to set this variable wherever the Django site is run. Also make sure that the settings module is available in PYTHONPATH.

More details on this approach are given on the Django settings page.

Using Environment Variables

One problem with multiple settings modules is that many settings won’t need to be different between environments. Duplicating these settings then violates the DRY principle (“don’t repeat yourself”). A more advanced approach for handling environment-specific settings is to use custom environment variables as Django inputs. Remember, the settings module is written in Python, so values can be set using calls and conditions. One settings module can be written to handle all environments.

Add a function like this to read environment variables:

# Imports
import os
from django.core.exceptions import ImproperlyConfigured

# Function
def read_env_var(name, default=None):
    if not value:
       raise ImproperlyConfigured("The %s value must be provided as an env variable" % name)
    return value

Then, use it to read environment variables in the settings module:

# Read the secret key directly
# This is a required value
# If the env variable is not found, the site will not launch
SECRET_KEY = read_env_var("SECRET_KEY")

# Read the debug setting
# Default the value to False
# Environment variables are strings, so the value must be converted to a Boolean
DEBUG = read_env_var("DEBUG", "False") == "True"

To avoid a proliferation of required environment variables, one variable could be used to specify the target environment like this:

# Read the target environment
TARGET_ENV = read_env_var("TARGET_ENV")

# Set the debug setting to True only for production
DEBUG = (TARGET_ENV == "prod")

# Set database config for the chosen environment
if TARGET_ENV == "dev":
    DATABASES = { ... }
elif TARGET_ENV == "prod":
    DATABASES = { ... }
elif TARGET_ENV == "test":
    DATABASES = { ... }

Managing environment variables can be pesky. A good way to manage them is using shell scripts. If the Django site will be deployed to Heroku, variables should be saved as config vars.

Conclusion

These are the two primary ways I recommend to handle different settings for different environments in a Django project. Personally, I prefer the second approach of using one settings module with environment variable inputs. Just make sure to reference all settings from the settings module (“from django.conf import settings”) instead of directly referencing environment variables!

Django Projects in PyCharm Community Edition

JetBrains PyCharm is one of the best Python IDEs around. It’s smooth and intuitive – a big step up from Atom or Notepad++ for big projects. PyCharm is available as a standalone IDE or as a plugin for its big sister, IntelliJ IDEA. The free Community Edition provides basic features akin to IntelliJ, while the licensed Professional Edition provides advanced features such as web development and database tools. The Professional Edition isn’t cheap, though: a license for one user may cost up to $199 annually (though discounts and free licenses may be available).

This guide shows how to develop Django web projects using PyCharm Community Edition. Even though Django-specific features are available only in PyCharm Professional Edition, it is still possible to develop Django projects using the free version with help from the command line. Personally, I’ve been using the free version of PyCharm to develop a small web site for a side business of mine. This guide covers setup steps, basic actions, and feature limitations based on my own experiences. Due to the limitations in the free version, I recommend it only for small Django projects or for hobbyists who want to play around.

Prerequisites

This guide focuses specifically on configuring PyCharm Community Edition for Django development. As such, readers should be familiar with Python and the Django web framework. Readers should also be comfortable with the command line for a few actions, specifically for Django admin commands. Experience with JetBrains software like PyCharm and IntelliJ IDEA is helpful but not required.

Python and PyCharm Community Edition must be installed on the development machine. If you are not sure which version of Python to use, I strongly recommend Python 3. Any required Python packages (namely Django) should be installed via pip.

Creating Django Projects and Apps

Django projects and apps require a specific directory layout with some required settings. It is possible to create this content manually through PyCharm, but it is recommended to use the standard Django commands instead, as shown in Part 1 of the official Django tutorial.

> django-admin startproject newproject
> cd newproject
> django-admin startapp newapp

Then, open the new project in PyCharm. The files and directories will be visible in the Project Explorer view.

The project root directory should be at the top of Project Explorer. The .idea folder contains IDE-specific config files that are not relevant for Django.

Creating New Files and Directories

Creating new files and directories is easy. Simply right-click the parent directory in Project Explorer and select the appropriate file type under New. Files may be deleted using right-click options as well or by highlighting the file and typing the Delete or Backspace key.

Files and folders are easy to visually create, copy, move, rename, and delete.

Django projects require a specific directory structure. Make sure to put files in the right places with the correct names. PyCharm Community Edition won’t check for you.

Writing New Code

Double-click any file in Project Explorer to open it in an editor. The Python editor offers all standard IDE features like source highlighting, real-time error checking, code completion, and code navigation. This is the main reason why I use PyCharm over a simpler editor for Python development. PyCharm also has many keyboard shortcuts to make actions easier.

Nice.

Editors for other file types, such as HTML, CSS, or JavaScript, may require additional plugins not included with PyCharm Community Edition. For example, Django templates must be edited in the regular HTML editor because the special editor is available only in the Professional Edition.

Workable, but not as nice.

Running Commands from the Command Line

Django admin commands can be run from the command line. PyCharm automatically refreshes any file changes almost immediately. Typically, I switch to the command line to add new apps, make migrations, and update translations. I also created a few aliases for easier file searching.

> python manage.py makemigrations
> python manage.py migrate
> python manage.py makemessages -l zh
> python manage.py compilemessages
> python manage.py test
> python manage.py collectstatic
> python manage.py runserver

Creating Run Configurations

PyCharm Community Edition does not include the Django manage.py utility feature. Nevertheless, it is possible to create Run Configurations for any Django admin command so that they can be run in the IDE instead of at the command line.

First, make sure that a Project SDK is set. From the File menu, select Project Structure…. Verify that a Project SDK is assigned on the Project tab. If not, then you may need to create a new one – the SDK should be the Python installation directory or a virtual environment. Make sure to save the new Project SDK setting by clicking the OK button.

Don’t leave that Project SDK blank!

Then from the Run menu, select Edit Configurations…. Click the plus button in the upper-left corner to add a Python configuration. Give the config a good name (like “Django: <command>”). Then, set Script to “manage.py” and Script parameters to the name and options for the desired Django admin command (like “runserver”). Set Working directory to the absolute path of the project root directory. Make sure the appropriate Python SDK is selected and the PYTHONPATH settings are checked. Click the OK button to save the config. The command can then be run from Run menu options or from the run buttons in the upper-right corner of the IDE window.

Run configurations should look like this. Anything done at the command line can also be done here.

When commands are run, the Run view appears at the bottom of the IDE window to show console output.

Special run configurations are particularly useful for the “test” and “runserver” commands because they enable rudimentary debugging. You can set breakpoints, run the command with debugging, and step through the Python code. If you need to interact with a web page to exercise the code, PyCharm will take screen focus once a breakpoint is hit. Even though debugging Django templates is not possible in the free version, debugging the Python code can help identify most problems. Be warned that debugging is typically a bit slower than normal execution.

Debugging makes Django development so much easier.

I typically use the command line instead of run configurations for other Django commands just for simplicity.

Version Control Systems

PyCharm has out-of-the-box support for version control systems like Git and Subversion. VCS actions are available under the VCS menu or when right-clicking a file in Project Explorer. PyCharm can directly check out projects from a repository, add new projects to a repository, or automatically identify the version control system being used when opening a project. Any VCS commands entered at the command line will be automatically reflected in PyCharm.

PyCharm’s VCS menu is initially generic. Once you select a VCS for your project, the options will be changed to reflect the chosen VCS. For example, Git will have options for “Fetch”, “Pull”, and “Push”.

Personally, I use Git with either GitHub or Atlassian Bitbucket. I prefer to do most Git actions like graphically through PyCharm, but occasionally I drop to the command line when I need to do more advanced operations (like checking commit IDs or forcing hard resets). PyCharm also has support for .gitignore files.

Python Virtual Environments

Creating virtual environments is a great way to manage Python project dependencies. Virtual environments are especially useful when deploying Django web apps. I strongly recommend setting up a virtual environment for every Python project you develop.

PyCharm can use virtual environments to run the project. If a virtual environment already exists, then it can be set as the Project SDK under Project Structure as described above. Select New… > Python SDK > Add Local, and set the path. Otherwise, new virtual environments can be created directly through PyCharm. Follow the same process to add a virtual environment, but under Python SDK, select Create VirtualEnv instead of Add Local. Give the new virtual environment an appropriate name and path. Typically, I put my virtual environments either all in one common place or one level up from my project root directory.

Creating a new virtual environment is pretty painless.

Databases

Out of the box, PyCharm Community Edition won’t give you database tools. You’re stuck with third-party plugins, the command line, or external database tools. This isn’t terrible, though. Since Django abstracts data into the Model layer, most developers rarely need to directly interact with the underlying database. Nevertheless, the open-source Database Navigator plugin provides support in PyCharm for the major databases (Oracle, MySQL, SQLite, PostgreSQL).

Limitations

The sections above show that PyCharm Community Edition can handle Django projects just like any other Python projects. This is a blessing and a curse, because advanced features are available only in the Professional Edition:

Django template support
Inter-project navigation (view to template)
Better code completion
Model dependency graphs
manage.py utility console
Database tools

The two features that matter most to me are the template support and the better code completion. With templates, I sometimes make typos or forget closing tags. With code completion, not all options are available because Django does some interesting things with model fields and dynamically-added attributes. However, all these missing features are “nice-to-have” but not “need-to-have” for me.

Conclusion

I hope you found this guide useful! Feel free to enter suggestions for better usage in the comments section below. You may also want to look at alternative IDEs, such as PyDev.

Easier Grep for Django Projects

Grep is a wonderful UNIX command line tool that searches for text in plain-text files. It can search one file or many, and its search phrase may be a regular expression. Grep is an essential tool for anyone who uses UNIX-based systems.

Grep is also useful when programming. Let’s face it: Sometimes, it’s easier to grep when searching for text rather than using fancy search tools or IDE features. I find this to be especially true for languages with a dynamic or duck type system like Python because IDEs cannot always correctly resolve links. Grep is fast, easy, and thorough. I use grep a lot when developing Django web projects because Django development relies heavily upon the command line.

The main challenge with grep is filtering the right files and text. In a large project, false positives will bloat grep’s output, making it harder to identify the desired lines. Specifically in a Django project, files for migrations, language translations, and static content may need to be excluded from searches.

I created a few helpful aliases for grepping Django projects:

alias grep_dj='grep -r --exclude="*.pyc" --exclude="*.mo" --exclude="*.bak"'
alias grep_djx='grep_dj --exclude="*.po" --exclude="*/migrations/*"'

The alias “grep_dj” does a recursive directory search, excluding compiled files (.pyc for Python and .mo for language) and backup files (.bak, which I often use for development database backups). The alias “grep_djx” further excludes language messages files (.po) and migrations.

To use these aliases, simply run the alias commands above at the command line. They may also be added to profile files so that they are created for every shell session. Then, invoke them like this:

> cd /path/to/django/project
> grep_djx "search phrase" *

Other grep options may be added, such as case-ignore:

> grep_djx -i "another phrase" *

These aliases are meant purely for the convenience of project-wide text searching. If you need to pinpoint specific files, then it may be better to use the raw grep command. You can also tweak these aliases to include or exclude other files – use mine simply as an example.

Copying File Paths from Finder in macOS

My personal laptop is a 13” Macbook Pro. Since I do a lot of software development work on my Mac, I often need to copy file paths. Unfortunately, it’s not easy to get file paths directly from Finder. Newer versions of macOS no longer include the path in the “Get Info” window. It is possible to get file paths from Terminal, either using “cd” and “ls” commands or by dragging files from Finder, but using Terminal is not always convenient.

Recently, I discovered how to make it easy. Using Automator, I added a “Copy Path” action to the right-click (or “secondary-click”) menu that will copy the absolute file path to the clipboard! This makes it very easy to get file paths directly from Finder. I learned this method by reading an OS X Daily article, and since it was so useful, I decided to share it here.

My Mac

The following steps were run on my personal Mac, which has the following specs:

It’s older than today’s kindergartners, but it still works reasonably well. I upgraded to 16GB memory and SSD storage.

The Steps

Launch Automator. (It’s in the Applications folder.)

Create a new service by navigating to File -> New and selecting Service from the dialog box.

Under Actions, search for “Copy to Clipboard”, and drag it to the right side of the panel. Set “Service receives selected” to “files or folders” and “in” to “Finder”.

Save the service with a name like “Copy Path”. Close Automator and open Finder. Right-click (or “secondary-click”) on any file – you should see the name of the new service as an available action. When you select it, the absolute file path is copied to the clipboard! You can then paste it (Cmd-V) into any text area.

This new action has been very helpful to me while programming. I hope you also find it helpful!

Who Should Lead BDD?

Behavior-driven development offers great benefits: better communication, easier test automation, and higher code quality. There are many ways for a team to start doing BDD, and naturally, someone needs to stand up and lead the effort. In my experience, adopting BDD is its own process. An evangelist converts team leaders, training sessions are given, and Gherkinized acceptance criteria start being automated. However, not everyone will embrace the changes, especially those across different role types. And big changes take time. Rome wasn’t built in a day, and neither will be a mature, effective BDD process.

This post covers three possible ways to lead BDD adoption, each from one of the Three Amigos roles. Any role can lead the charge, but each will have its unique struggles. These possibilities are advisory but not necessarily prescriptive. If you want to move your team into BDD, use these three approaches as guidelines for crafting a plan that best meets your needs. And, of course, the advice in Winning Support for BDD pertains to all approaches. Furthermore, as you read these approaches, put yourselves in the shoes of roles other than your own, so you can better understand the struggles each role faces.

Note: The approaches below presume that the underlying software development process is Agile Scrum. Nevertheless, they may be tweaked and applied to other processes, like Waterfall or Kanban.

The Starting Point

The starting point for all three approaches below is a “traditional” Agile sprint – one that is not (yet) behavior-driven. Product owners write user stories, developers implement the solutions, and testers test the deliverables. The diagram below shows the the main flow of sprint work in this type of sprint, and it will serve as the basis for illustrating BDD adoption:

The overall flow of a “traditional,” non-behavior-driven Agile sprint. Ceremonies like planning, review, and retrospective should still happen, but the are left out of this diagram to put emphasis on parts affected by BDD.

QA-Led BDD

Circumstances

The most common approach I’ve seen is QA-led BDD adoption, because testers arguably have the most to gain. It is most applicable when the Three Amigos roles (biz, dev, and test) are well-defined and separate. The impetus for QA to lead BDD adoption could be that developers deliver code too late to adequately test and automate within a sprint, or it could be that the QA team is struggling to scale their test automation development. There may also be resistance to BDD from biz and dev roles.

Steps

The sensible path for QA is to start all the way to the right and progressively shift left. This means that the starting point would be test automation. Start by building a solid automation code base. Pick a well-supported BDD framework like Cucumber, SpecFlow, or behave, and start adding scenarios and step definitions. Select scenarios for core product features rather than the latest sprint stories, so that the code base will be populated with the most basic, useful steps. Once the automation code reaches a “critical mass” for step reusability, QA can then proactively classify new test scenarios as automated or manual. Automated tests become easier and easier to write, giving QA more time to be exploratory with manual testing. Ideally, all manual testing would become exploratory.

Then, it’s time to start shifting left. At this point, all Gherkin steps would be in the automation code only, so set up a tool like Pickles to expose the steps to all team members as living documentation. QA should then schedule Three Amigos meetings with biz and dev to proactively discuss user story expectations. In those meetings, QA should start demonstrating how to write acceptance criteria in Gherkin, which then expedites testing. A big win would be if a QA engineer could write a new scenario using only pre-existing, pre-automated steps and then run it successfully on the spot.

Once biz and dev folks are convinced of BDD’s benefits, encourage them to participate in writing Gherkin. When they get comfortable, encourage product owners to write acceptance criteria in Gherkin when they write user stories, and hold Three Amigos meetings before sprint planning as part of grooming. Convince them that for them to help write Gherkin scenarios is a process efficiency for the whole team.

This slideshow requires JavaScript.

Struggles

Shifting left is never easy, especially when team members are hardened into their roles. That’s why QA must write both really good test automation and really good Gherkin scenarios. Success should speak for itself once QA delivers good automation fast. Furthermore, QA must be clear that BDD is not merely a test tool, it’s a process that requires a paradigm shift. Otherwise, BDD could be easily pigeonholed to be a “QA thing.”

Dev-Led BDD

Circumstances

There are a few reasons that could push developers to lead BDD. On some Agile teams, there’s no distinction between dev and QA roles: all team members are software engineers responsible for both developing and testing the software. Or, developers may not be satisfied with the testing effort. Maybe too many bugs are escaping the sprint, or maybe automation isn’t getting done in time. Or, perhaps the product owner is not happy with the deliverables and putting pressure on the team to do better. Whatever the circumstance, developers are more than capable of winning with BDD.

Steps

The best way for developers to start is to set up Three Amigos meetings, to stop the game of telephone between biz and test. In those meetings, start translating acceptance criteria into Gherkin. Then, start helping out with test automation – that may mean anything from offering advice to QA to building the framework from scratch. Then, start pushing left and right to get biz and test on board with BDD.

This slideshow requires JavaScript.

Struggles

It may be difficult for developers to work on test automation because they may lack either the expertise or the time to devote to good test automation. Automation is a specialized discipline, and it takes time and diligence to build up expertise to do it right. I’ve seen very skilled developers haughtily build very shabby automation frameworks.

Developers must also be careful to not be too technical, or else biz and test roles may reject BDD for being too complicated or beyond their abilities. Furthermore, some teams may be resistant to developing test automation. For example, automation work may be “starved” for points because it is underestimated or similarly starved for time because it is deemed lower in priority to other work.

Biz-Led BDD

Circumstances

BDD is designed to bring technical and business roles together into healthier collaboration, and biz folks can certainly lead BDD adoption as successfully as more technical folks can. Major reasons for biz to take the lead could be if development is perpetually running behind schedule, if deliverables don’t meet the original requirements, or if software bugs are rampant.

Steps

For biz roles, “shift left” could be better called “pull left.” Start by writing solid user stories and Gherkin acceptance criteria. Focus on good Gherkin that is readable and reusable. Then, introduce BDD as a refinement to the Agile process, highlighting its benefits. Initiate Three Amigos meetings to make sure that you are communicating the right things to dev and test. Once collaboration is going well, suggest BDD automation as a way to expedite dev and test work. If acceptance criteria are all Gherkinized, then developing BDD automation would be a natural extension.

This slideshow requires JavaScript.

Struggles

In my experience, biz roles (specifically product owners) tend to be the most hesitant about BDD. They often see writing Gherkin as a burdensome requirement rather than a way to help their team. Or, they may fear that BDD is “too technical” for them. It may also be difficult for them to pitch BDD automation to the team. To be successful, biz roles need to step outside their comfort zone to win supporters from dev and test.

Process paradigm shifts can be hard, especially on teams that are already overwhelmed with work. Some people just don’t like change. Process and automation change can also be a big challenge if QA is outsourced (which is common).

Side-By-Side Comparison

Here’s the TL;DR:

Role	Circumstances	Steps	Struggles
QA	Code is delivered too late to test and automate Automation development is not scaling	Build a solid BDD automation framework Demonstrate automation success Set up Three Amigos meetings during the sprint Start writing Gherkin scenarios with biz and dev as part of grooming	Showing that BDD is a whole development process, not just a QA thing Getting the team to truly shift left
Dev	No separation of dev and QA roles Too many bugs are escaping the sprint Pressure from biz to do better	Initiate better collaboration through Three Amigos and Gherkin Push right by helping QA with testing and automation Push left by helping biz write better acceptance criteria	Humbly learning good automation practices Dedicating time for automation and more meetings
Biz	Missed deadlines Deliverables not matching expectations Too many bugs	Write acceptance criteria in good Gherkin Set up Three Amigos meetings to review Gherkin Pitch BDD automation	Learning semi-technical things Pushing all the way to automation

Conclusion

These are just three general approaches intended to show how BDD is for everyone. If you have other approaches, please describe them in the comment section below! Whatever the approach, make sure to demonstrate that BDD helps everyone, or else people may feel forced into corners and reject BDD for bad reasons. And remember, software quality is not just QA’s responsibility; it is everyone’s responsibility.

10 Gotchas for Automation Code Reviews

Lately, I’ve been doing lots of code reviews. I probably spend about an hour every work day handling reviews for my team, both as a reviewer and an author. All of the reviews exclusively cover end-to-end test automation: new tests, old fixes, config changes, and framework updates. I adamantly believe that test automation code should undergo the same scrutiny of review as the product code it tests, because test automation is a product. Thus, all of the same best practices (like the guides here and here) should be applied. Furthermore, I also look for problems that, anecdotally, seem to appear more frequently in test automation than in other software domains. Below is a countdown of my “Top 10 Gotchas”. They are the big things I emphasize in test automation code reviews, in addition to the standard review checklist items.

#10: No Proof of Success

“Trust, but verify,” as Ronald Reagan would say. Tests need to run successfully in order to pass review, and proof of success (such as a log or a screen shot) must be attached to the review. In the best case, this means something green (or perhaps blue for Jenkins). However, if the product under test is not ready or has a bug, this could also mean a successful failure with proof that the critical new sections of the code were exercised. Tests should also be run in the appropriate environments, to avoid the “it-ran-fine-on-my-machine” excuse later.

#9: Typos and Bad Formatting

My previous post, Should I Reject a Code Review for Typos?, belabored this point. Typos and bad formatting reflect carelessness, cause frustration, and damage reputation. They are especially bad for Behavior-Driven Development frameworks.

#8: Hard-Coded Values

Hard-coded values often indicate hasty development. Sometimes, they aren’t a big problem, but they can cripple an automation code base’s flexibility. I always ask the following questions when I see a hard-coded value:

Should this be a shared constant?
Should this be a parameterized value for the method/function/step using it?
Should this be passed into the test as an external input (such as from a config file or the command line)?

#7: Incorrect Test Coverage

It is surprisingly common to see an automated test that doesn’t actually cover the intended test steps. A step from the test procedure may be missing, or an assertion may yield a false positive. Sometimes, assertions may not even be performed! When reviewing tests, keep the original test procedure handy, and watch out for missing coverage.

#6: Inadequate Documentation

Documentation is vital for good testing and good maintenance. When a test fails, the doc it provides (both in the logs it prints and in its very own code) significantly assist triage. Automated test cases should read like test procedures. This is one reason why self-documenting behavior-driven test frameworks are so popular. Even without BDD, test automation should be flush with comments and self-documenting identifiers. If I cannot understand a test by skimming its code in a code review, then I ask questions, and when the author provides answers, I ask them to add their answers as comments to the code.

#5: Poor Code Placement

Automation projects tend to grow fast. Along with new tests, new shared code like page objects and data models are added all the time. Maintaining a good, organized structure is necessary for project scalability and teamwork. Test cases should be organized by feature area. Common code should be abstracted from test cases and put into shared libraries. Framework-level code for things like inputs and logging should be separated from test-level code. If code is put in the wrong place, it could be difficult to find or reuse. It could also create a dependency nightmare. For example, non-web tests should not have a dependency on Selenium WebDriver. Make sure new code is put in the right place.

#4: Bad Config Changes

Even the most seemingly innocuous configuration tweak can have huge impacts:

A username change can cause tests to abort setup.
A bad URL can direct a test to the wrong site.
Committing local config files to version control can cause other teammates’ local projects to fail to build.
Changing test input values may invalidate test runs.
One time, I brought down a whole continuous integration pipeline by removing one dependency.

As a general rule, submit any config changes in a separate code review from other changes, and provide a thorough explanation to the reviewers for why the change is needed. Any time I see unusual config changes, I always call them out.

#3: Framework Hacks

A framework is meant to help engineers automate tests. However, sometimes the framework may also be a hindrance. Rather than improve the framework design, many engineers will try to hack around the framework. Sometimes, the framework may already provide the desired feature! I’ve seen this very commonly with dependency injection – people just don’t know how to use it. Hacks should be avoided because test automation projects need a strong overall design strategy.

#2: Brittleness

Test automation must be robust enough to handle bumps in the road. However, test logic is not always written to handle slightly unexpected cases. Here are a few examples of brittleness to watch out for in review:

Do test cases have adequate cleanup routines, even when they crash?
Are all exceptions handled properly, even unexpected ones?
Is Selenium WebDriver always disposed?
Will SSH connections be automatically reconnected if dropped?
Are XPaths too loose or too strict?
Is a REST API response code of 201 just as good as 200?

#1: Redundancy

Redundancy is the #1 problem for test automation. I wrote a whole article about it: Why is Automation Full of Duplicate Code? Many testing operations are inherently repetitive. Engineers sometimes just copy-paste code blocks, rather than seek existing methods or add new helpers, to save development time. Plus, it can be difficult to find reusable parts that meet immediate needs in a large code base. Nevertheless, good code reviews should catch code redundancy and suggest better solutions.

Please let me know in the comments section if there are any other specific things you look for when reviewing test automation code!