Why Python is Great for Test Automation

Python is an incredible programming language. As Dan Callahan said in his PyCon 2018 keynote, “Python is the second best language for anything, and that’s an amazing aspiration.” For test automation, however, I believe it is one of the best choices. Here are ten reasons why:

#1: The Zen of Python

The Zen of Python, as codified in PEP 20, is an ideal guideline for test automation. Test code should be a natural bridge between plain-language test steps and the programming calls to automate them. Tests should be readable and descriptive because they describe the features under test. They should be explicit in what they cover. Simple steps are better than complicated ones. Test code should add minimal extra verbiage to the tests themselves. Python, in its concise elegance, is a powerful bridge from test case to test code.

(Want a shortcut to the Zen of Python? Run “import this” at the Python interpreter.)

#2: pytest

pytest is one of the best test frameworks currently available in any language, not just for Python. It can handle any functional tests: unit, integration, and end-to-end. Test cases are written simply as functions (meaning no side effects as long as globals are avoided) and can take parametrized inputs. Fixtures are a generic, reusable way to handle setup and cleanup operations. Basic “assert” statements have automatic introspection so failure messages print meaningful values. Tests can be filtered when executed. Plugins extent pytest to do code coverage, run tests in parallel, use Gherkin scenarios, and integrate with other frameworks like Django and Flask. Other Python test frameworks are great, but pytest is by far the best-in-show. (Pythonic frameworks always win in Python.)

#3: Packages

For all the woes about the CheeseShop, Python has a rich library of useful packages for testing: pytest, unittest, doctest, tox, logging, paramiko, requests, Selenium WebDriver, Splinter, Hypothesis, and others are available as off-the-shelf ingredients for custom automation recipes. They’re just a “pip install” away. No reinventing wheels here!

#4: Multi-Paradigm

Python is object-oriented and functional. It lets programmers decide if functions or classes are better for the needs at hand. This is a major boon for test automation because (a) stateless functions avoid side effects and (b) simple syntax for those functions make them readable. pytest itself uses functions for test cases instead of shoehorning them into classes (à la JUnit).

#5: Typing Your Way

Python’s out-of-the-box dynamic duck typing is great for test automation because most feature tests (“above unit”) don’t need to be picky about types. However, when static types are needed, projects like mypy, Pyre, and MonkeyType come to the rescue. Python provides typing both ways!

#6: IDEs

Good IDE support goes a long way to make a language and its frameworks easy to use. For Python testing, JetBrains PyCharm supports visual testing with pytest, unittest, and doctest out of the box, and its Professional Edition includes support for BDD frameworks (like pytest-bdd, behave, and lettuce) and Web development. For a lighter offering, Visual Studio Code is taking the world by storm. Its Python extensions support all the good stuff: snippets, linting, environments, debugging, testing, and a command line terminal right in the window. Atom, Sublime, PyDev, and Notepad++ also get the job done.

#7: Command Line Workflow

Python and the command line are like peanut butter and jelly – a match made in heaven. The entire test automation workflow can be driven from the command line. Pipenv can manage packages and environments. Every test framework has a console runner to discover and launch tests. There’s no need to “build” test code first because Python is an interpreted language, further simplifying execution. Rich command line support makes testing easy to manage manually, with tools, or as part of build scripts / CI pipelines.

As a bonus, automation modules can be called from the Python REPL interpreter or, even better, a Jupyter notebook. What does this mean? Automation-assisted exploratory testing! Imagine using Python calls to automatically steer a Web app to a point that requires a manual check. Calls can be swapped out, rerun, skipped, or changed on the fly. Python makes it possible.

#8: Ease of Entry

Python has always been friendly to beginners thanks to its Zen, whether those beginners are programming newbies or expert engineers. This gives Python a big advantage as an automation language choice because tests need to be done quickly and easily. Nobody wants to waste time when the features are in hand and just need to be verified. Plus, many manual software testers (often without programming experience) are now starting to do automation work (by choice or by force) and benefit from Python’s low learning curve.

#9: Strength for Scalability

Even though Python is great for beginners, it’s also no toy language. Python has industrial-grade strength because its design always favors one right way to get a job done. Development can scale thanks to meaningful syntax, good structure, modularity, and a rich ecosystem of tools and packages. Command line versatility enables it to fit into any tool or workflow. The fact that Python may be slower than other languages is not an issue for feature tests because system delays (such as response times for Web pages and REST calls) are orders of magnitude slower than language-level performance hits.

#10: Popularity

Python is one of the most popular programming languages in the world today. It is consistently ranked near the top on TIOBE, Stack Overflow, and GitHub (as well as GitHut). It is a beloved choice for Web developers, infrastructure engineers, data scientists, and test automationeers alike. The Python community also powers it forward. There is no shortage of Python developers, nor is there any dearth of support online. Python is not going away anytime soon. (Python 3, that is.)

Other Languages?

The purpose of this article is to highlight what makes Python great for test automation based on its own merits. Although I strongly believe that Python is one of the best automation languages, other choices like Java, C#, and Ruby are also viable. Check out my article The Best Programming Language for Test Automation for a comparison.

 

This article was posted with the author’s permission on both Automation Panda and PyBites.

Cypress.io and the Future of Web Testing

What is Cypress.io?

Cypress.io is an up-and-coming Web test automation framework. It is open source and written entirely in JavaScript. Unlike Selenium WebDriver tests that work outside the browser, Cypress works directly inside the browser. It enables developers to write front-end tests entirely in JavaScript, directly accessing everything within the browser. As a result, tests run much more quickly and reliably than Selenium-based tests.

Some nifty features include:

  • A rich yet simple API for interactions with automatic waiting
  • Mocha, Chai, and Sinon bundled in
  • A sleek dashboard with automatic reloads for Test-Driven Development
  • Easy debugging
  • Network traffic control for validation and mocking
  • Automatic screenshots and videos

Cypress was clearly developed for developers. It enables rapid test development with rapid feedback. The Cypress Test Runner is free, while the Cypress Dashboard Service (for better reporting and CI) will require a paid license.

How Do I Start Using Cypress?

I won’t post examples or instructions for using Cypress here. Please refer to the Cypress documentation for getting started and the tutorial video below. Make sure your machine is set up for JavaScript development.

Will Cypress Replace WebDriver?

TL;DR: No.

Cypress has its niche. It is ideal for small teams whose stacks are exclusively JavaScript and whose developers are responsible for all testing. However, WebDriver still has key advantages.

  1. While Selenium WebDriver supports nearly all major browsers, Cypress currently supports only one browser: Google Chrome. That’s a major limitation. Web apps do not work the same across browsers. Many industries (especially banking and finance) put strict controls on browser types and versions, too.
  2. Cypress is JavaScript only. Its website proudly touts its JavaScript purity like a badge of honor. However, that has downsides. First, all testing must happen inside the bubble of the browser, which makes parallel testing and system interactions much more difficult. Second, testers must essentially be developers, which may not work well for all teams. Third, other programming languages that may offer advantages for testing (like Python) cannot be used. Selenium WebDriver, on the other hand, has multiple language bindings and lets tests live outside the browser.
  3. Within the JavaScript ecosystem, Cypress is not the only all-in-one end-to-end framework. Protractor is more mature, more customizable, and easier to parallelize. It wraps Selenium WebDriver calls for simplification and safety in a similar way to how Cypress’s API is easy to use.
  4. The WebDriver standard is a W3C Recommendation. What does this mean? All major browsers have a vested interest in implementing the standard. Selenium is simply the most popular implementation of the standard. It’s not going away. Cypress, however, is just a cool project backed with commercial intent.

Further reading:

What Does Cypress Mean for the Future?

There are a few big takeaways.

  1. JavaScript is taking over the world. It was the most popular language on GitHub in 2017. JavaScript-only stacks like MEAN and MERN are increasingly popular. The demand for a complete JavaScript-only test framework like Cypress is further evidence.
  2. “Bundled” test frameworks are becoming popular. Historically, a test framework simply provided test structure, basic fixtures, and maybe an assertion library (like JUnit). Then, extra test packages became popular (like Selenium WebDriver, REST APIs, mocking, logging, etc.). Now, new frameworks like Cypress and Protractor aim to provide pre-canned recipes of all these pieces to simplify the setup.
  3. Many new test frameworks will likely be developer-centric. There is a trend in the software industry (especially with Agile) of eliminating traditional tester roles and putting testing work onto developers. The role of the “Software Engineer in Test” – a developer who builds test systems – is also on the rise. Test automation tools and frameworks will need to provide good developer experience (DX) to survive. Cypress is poised to ride that wave.
  4. WebDriver is not perfect. Cypress was developed in large part to address WebDriver’s shortcomings, namely the slowness, difficulty, and unreliability (though unreliability is often a result of poor implementation). Many developers don’t like to use Selenium WebDriver, and so there will be a constant itch to make something better. Cypress isn’t there yet, but it might get there one day.

A Cucumber Kimchi Recipe

My wife and I love to eat kimchi. Recently, I started making it at home instead of buying the big (expensive) jars at our local Asian market. I just made oi-sobagi (오이소박이), a cucumber kimchi, for the first time. Given that “Cucumber” is a test automation buzzword, I figured I’d share the recipe here. I hope you enjoy it as much as my Chinese cucumber recipe!

The recipe I used came from Maangchi, “YouTube’s Korean Julia Child.” She has many other delicious Korean recipes, too. Her oi-sobagi recipe is here:

Clicking Web Elements with Selenium WebDriver

Selenium WebDriver is the most popular open source package for Web UI test automation. It allows tests to interact directly with a web page in a live browser. However, using Selenium WebDriver can be very frustrating because basic interactions often lack robustness, causing intermittent errors for tests.

The Basics

One such vulnerable interaction is clicking elements on a page. Clicking is probably the most common interaction for tests. In C#, a basic click would look like this:

webDriver.FindElement(By.Id("my-id")).Click();

This is the easy and standard way to click elements using Selenium WebDriver. However, it will work only if the targeted element exists and is visible on the page. Otherwise, the WebDriver will throw exceptions. This is when programmers pull their hair out.

Waiting for Existence

To avoid race conditions, interactions should not happen until the target element exists on the page. Even split-second loading times can break automation. The best practice is to use explicit waits before interactions with a reasonable timeout value, like this:

const int timeoutSeconds = 15;
var ts = new TimeSpan(0, 0, timeoutSeconds);
var wait = new WebDriverWait(webDriver, ts);

wait.Until((driver) => driver.FindElements(By.Id("my-id")).Count > 0);
webDriver.FindElement(By.Id("my-id")).Click();

Other Preconditions

Sometimes, Web elements won’t appear without first triggering something else. Even if the element exists on the page, the WebDriver cannot click it until it is made visible. Always look for the proper way to make that element available for clicking. Click on any parent panels or expanders first. Scroll if necessary. Make sure the state of the system should permit the element to be clickable.

If the element is scrolled out of view, move to the element before clicking it:

new Actions(webDriver)
    .MoveToElement(webDriver.FindElement(By.Id("my-id")))
    .Click()
    .Perform();

Last Ditch Efforts

Nevertheless, there are times when clickable elements just don’t cooperate. They just can’t seem to be made visible. When all else fails, drop directly into JavaScript:

((IJavaScriptExecutor)webDriver).ExecuteScript(
    "arguments[0].click();",
    webDriver.FindElement(By.Id("my-id")));

Do this only when absolutely necessary. It is a best practice to use Selenium WebDriver methods because they make automated interaction behave more like a real user than raw JavaScript calls. Make sure to give good reasons in code comments whenever doing this, too.

Final Advice

This article was written specifically for clicks, but its advice can be applied to other sorts of interactions, too. Just be smart about waits and preconditions.

Note: Code examples on this page are written in C#, but calls are similar for other languages supported by Selenium WebDriver.

A Panda’s Retrospective on The Legend of Zelda: Breath of the Wild

Warning #1: This post has nothing to do with software development or testing. It is purely a post for my own pleasure.

Warning #2: SPOILER ALERT! This post is full of ’em.

The Legend of Zelda: Breath of the Wild is one of my favorite video games. Ever. I’ve been playing it regularly on my Wii U since Christmas, and I just beat the main quest. I redeemed all four Divine Beasts, found all 120 shrines, recovered all memories, and got a majority of the Korok seeds and Hyrule Compendium. It was truly the (virtual) adventure of a lifetime! In celebration of finally beating the game, I’m going to retrospect on what I loved about it so much.

What’s It About?

If you don’t know what this game is about, watch this:

Here’s the story: You, as a young man named Link, wake up in a cave with no memory about yourself or your past. As you explore the world and recover your memories, you discover that the kingdom of Hyrule was destroyed 100 years ago by the evil Calamity Ganon. Princess Zelda has sealed Ganon inside Hyrule Castle since then, but her power is weakening. You, along with four other “champions,” fought together to defeat Ganon but failed, and you yourself were nearly killed. Your main mission is to explore the land to strengthening yourself, recover lost memories, and reawaken four divine beasts. Then, you need to storm the castle to defeat Ganon, rescue Zelda, and bring a lasting peace to Hyrule.

How I Played

When BOTW was first announced as an open-world adventure, I immediately shut out all news and reviews aside from official trailers. I adamantly did not want any spoilers whatsoever. I wanted to fully explore the game on my own. And, truth be told, it was totally worth it.

When I play video games, I tend to be a completionist. I like to leave no stone unturned, and I take time to power up as much as possible before big challenges. As a result, my progress rate in BOTW was rather slow, but I feel like I didn’t miss out on anything. Even then, there are still side quests left incomplete and Korok seeds left unfound. All told, I spent 241 hours on a single save file.

botw_playtime

My total playtime after defeating Ganon.

After completing the Great Plateau and fully restoring the Sheikah Slate, my main approach to exploration was to look for shrines in a new area using the binoculars, make a mad rush for the shrine to claim a safety post, and then thoroughly search the surrounding areas for new stuff. I literally went around blind because I didn’t know where to find the Divine Beasts. Here’s roughly the path I took:

  1. Scoured Necluda for shrines for a while.
  2. Went to Zora’s Domain and reclaimed Vah Ruta.
  3. Went to the Great Hyrule Forest but couldn’t take the Master Sword.
  4. Found the Ancient Tech Lab in Akkala.
  5. Criss-crossed Hyrule Field and got blown up by Guardians.
  6. Climbed Mount Lanayru and OH CRAP THERE’S A DRAGON!
  7. Cross-dressed my way through the Gerudo Desert and reclaimed Vah Nabooris.
  8. Froze my butt off in the Gerudo Highlands.
  9. Rode through Lake Hylia and Faron.
  10. Took wings in Tabantha and reclaimed Vah Medoh.
  11. Set my butt on fire on Death Mountain and reclaimed Vah Rudania.
  12. Decided to freeze my butt off again, but this time in Hebra.
  13. Stormed Hyrule Castle, kicking butt and taking names.

For combat, I really preferred nimble melee combat. I favored single-handed weapons and spears. I got pretty good with arrows, too, and I would often try to sneak around as an archer before wailing on enemies with a sword.

guardian_artwork

Guardians – ruining Link’s day, every day.

I did not purchase the DLC, and I probably won’t anytime soon. Why? I simply don’t have time. I didn’t feel like the game was lacking without the DLC, though it looks like it adds some cool content.

What I Loved

The open world was breathtaking. The scenery was truly beautiful, and the whole game was completely explorable. Except for the very edges of the map, there were no invisible walls: if it was there, you could climb it or jump it.

https3a2f2fblueprint-api-production-s3-amazonaws-com2fuploads2fcard2fimage2f6711412fcfc041c6-cef4-4b89-8745-b3cf1d7a8570

It’s just wild.

As the player, you chose the order of events. In other Zelda games, as in most adventure games, the main quest is completely linear. The player must complete events in a certain order to progress. However, in BOTW, after fully restoring the Sheikah Slate, you can make up your own adventure. The divine beasts are recommended but not required. You can even attempt to fight Ganon immediately after stepping off the Great Plateau (though the King’s ghost rightfully says that would be foolish). If I were to play it again, I’d probably do this order: Necluda/Faron, Great Hyrule Forest, Zora’s Domain, Death Mountain, Tabantha, and Gerudo Desert.

The non-linear unfolding of memories made the story so much more engaging. Until you beat the main quest, you really don’t know everything that happened in the past. You find out bits and pieces about Link, Zelda, and the Champions all throughout the quest. The flashbacks become wonderful rewards for progression.

The Sheikah. My favorite race in BOTW are the Sheikah. They are bona fide ninjas with Jōmon stylistics. They’re the underdog good guys who help Link. They made the greatest technology in Hyrule. And for once, the Sheikah are a living, thriving people group. I loved running around Hyrule wearing the Sheikah armor in honor of the Sheikah.

tumblr_oq01gwfwmp1qhgj2mo1_500

I want a man bun like that.

The Gorons are hilarious. If the Sheikah are my #1, then the Gorons are my #2. They’re very friendly people with priceless facial expressions.

This slideshow requires JavaScript.

The music was elegantly serene. It always perfectly complemented the location. From the piano taps while crossing the fields on horseback, to the rambunctious trombones of Goron City, to the violins of Hateno Village that could move you to tears, the music was on point.

Sheikah technology is so cool. Link is running around Hyrule with a mobile phone. The only thing it can’t do is make phone calls!

sheikah-slate-breath-of-the-wild

So, somehow, Hyrule regressed after 10,000 years?

Cooking. I love cooking. The fact that it’s in a Zelda game is incredible. ‘Nuff said.

5c06dba4cd3432d490d919c0895ff0e4

Iron Chef Link!

Favorite Moments

Meeting the Sheikah monk at the end of every shrine. I loved how each monk had a unique name and pose. I also loved how the blue light would shatter as the monk blessed you with the Spirit Orb.

026

Every time was a special encounter.

The first horse. Taming, riding, and domesticating my first horse blew my mind. (Remember, I had no spoilers, not even horses.) I still have my first horse, a brown one with a white butt who I named Buttercup.

buttercup.jpg

Look at mah horse – mah horse is amazing!

The first Lynel defeat. I felt UNSTOPPABLE.

ee9

This is not easy.

The first blood moon. I panicked because I had no idea what was happening.

Discovering the Great Skeleton in Hebra. It’s in a secret, beautiful cavern under a mountain peak.

Defeating Master Kohga. I delighted in sending him straight down to hell where he belonged. I had zero tolerance for the Yiga Clan.

Finding Gorons in Gerudo Town. They don’t even know how they got it!

gtxdtk9zj7ny

It must be the gems.

Building Tarrey Town. Tarrey town is the place where all the races come together to build a productive new settlement together in peace. Everyone gets along. You can find some of the best goods in Hyrule there. It’s also the antithesis of the Great Calamity’s destruction: rather than finding previous towns decimated into ruins, Tarrey Town is the face of the new Hyrule rising from the ashes.

Finding the Hylian Shield. From the beginning, I hoped that the Hylian shield would be available somewhere in the game. I didn’t find it until the end of the game, but when I did, the shield was righteous.

Zelda’s face at the end of the game. #worthit

a87b59737f442d9c36c5eea72d68f923984f6f79_00

How is she 117 years old?

Finding out the location of Paya’s birthmark. Paya is probably my favorite character in the game. She’s a sweet, innocent Sheikah lass with a monster crush on Link. She will tell you she got her name from a papaya-seed-shaped birthmark somewhere on her body, but she won’t say where. I spent the whole game wondering where that birthmark was. Forget defeating Ganon and saving the princess, I just wanted to know. After finally completing the heirloom shrine in Kakariko Village (which was my last shrine because I overlooked the diary), Impa finally tells you.

5ynciqb

And now we know.

Some Critique

Although BOTW is marvelous, there were a few things I felt could have been improved. These critiques do not in any way mean that the game was poor.

It rains too much in Hyrule. Wanna go climbing? Nope – it’s a slip ‘n slide. Need to cook some food on-the-go? Sorry – flame goes out. Stuck waiting for time to pass? No sitting by a fire. Trying to explode a powder keg on some Bokoblins? Not today. And let’s just hurl some lighting bolts at you while you cross an open field, too. The rain does nothing but block your progress. My biggest frustration was climbing: I couldn’t explore the areas I wanted to go, and I couldn’t pass the time with a fire. Coming back later would be a hassle because I had already climbed high to get there, so I was effectively stuck. It would have been nice to have some sort of item to climb in the rain.

The weapons break too easily. The game mechanic of weapons breaking is pretty clever because it forces the player to always try new weapons. However, it feels like they break too soon. Weapon durability is a significant problem early in the game, because the player has very few inventory slots and strong weapons are hard to find. Later in the game, this is less of an issue. I think some better balancing could have been done.

There was no fishing rod. For being such an outdoorsy game, I was truly surprised that there were no fishing rods in BOTW. Link hunts deer with bows and arrows, but he needs to use bombs to catch fish? This struck me as strange and, to a small extent, thwarted the immersive feeling of surviving in the wild. A fishing rod may have been challenging to implement, but I think it could have been possible as a key item like the glider.

I craved more story points. Much of my gameplay narrative shifted between Divine Beast sagas and open exploration. At times, I felt like there was too much to explore without enough reward. I found myself avoiding enemy hideouts late in the game because I simply didn’t need their weapons or spoils. Having more memories to discover would have been awesome, though I suppose that’s what the DLC is for.

Behind the Scenes

Nintendo released a 4-part video series entitled The Making of The Legend of Zelda: Breath of the Wild that’s really cool. Video links are below.

Final Impression

I love The Legend of Zelda: Breath of the Wild. I can’t wait to see where Nintendo takes the franchise from here!

Why Choose BDD Over Other Test Frameworks?

People are heavily opinionated about Behavior-Driven Development. I frequently hear opponents say things like this:

Why would I use a BDD test framework instead of a traditional test framework like JUnit, NUnit, or pytest? The extra layer of plain language Gherkin steps gets in the way of the automation code. I can directly write code for those steps instead. BDD frameworks require lots of extra work that just doesn’t seem to add value. My team isn’t doing behavior-driven development practices, anyway.

I can sympathize with these sentiments, especially for those who have participated in projects where BDD was done poorly. Even if a team isn’t doing full behavior-driven development practices, I still assert that BDD test automation frameworks are better than traditional test frameworks for most feature testing (above-unit, black box). Here are reasons why.

Separation of Test Cases from Test Code

Test cases and test code are separate concerns. I should be able to design, discuss, and digest a test case without ever touching code. We describe features in plain language, and so we should also describe tests in plain language. Step definitions are nothing more than the automation behind the test case steps. Traditional test frameworks simply don’t have this separation of concerns, even if test methods are loaded with comments.

Guide Rails

BDD frameworks enforce good structure and layers for automation. There are designated places for test cases, step definitions, and support classes. The framework encourages good practices. Traditional test framework, however, are much more free-form. Programmers can do scary and stupid things with test classes. Functionally, a traditional test framework can still be structured well with layers and support classes, but it’s not required. Based on my experiences seeing less experienced automationeers shoving everything into Frankenstein’ed test methods, I much prefer to have the guide rails of a BDD framework.

Inherent Reusability

Steps are the building blocks of test cases, and test cases almost always have the same steps. BDD frameworks identify the step as a unique concern. One step with its definition can be used by any scenario, and steps can be parametrized for flexibility. This creates a “snowball” effect once enough steps have been developed: new tests may not require any new automation code! Traditional test frameworks simply don’t have this mechanism. It could be implemented by calling functions and classes outside of test classes, but not all automationeers are disciplined to do so, and everyone who does it will do it differently.

Aspect-Oriented Controls

Good frameworks handle cross-cutting concerns automatically. Things like logging, dependency injection, and cleanup shouldn’t interfere with test cases. BDD frameworks provide hooks to insert extra logic for these concerns around steps, scenarios, features, and even the whole test suite. Hooks can squeeze into steps because the framework is structured around steps. For example, hooks can automatically log steps to Extent Reports, instead of forcing programmers to write explicit logging calls in each test method.

giphy

The HOOKS, me bucko!

Easier Reviews

Nothing ruins your day like an illegible code review on features you don’t know. You are responsible for providing valuable feedback, but you can’t figure out what’s going on in the short amount of time you can dedicate to the review. Good Gherkin, however, makes it easy. A reviewer can review the test case apart from any code first to make sure it is a good test case. At this level, the reviewer could even be a non-technical person like a product owner. Then, the reviewer can either send the test case back with suggestions or, if the test case passes muster, dig deeper into the automation code.

Easier Onboarding

It can be hard to onboard new team members. They have so much to learn about the product, the code base, and the team practices. If tests are written using a BDD framework, then newbies can learn the features simply by reading the behavior specs. New automationeers likewise can rely on existing steps both for reuse and for examples as they develop new tests.

Other Reasons?

I’m sure there are other benefits to BDD frameworks, but these are the big ones for me. It’s an opinionated thing. Feel free to add comments below!