fun

Tutoring: A Lifelong Impact

On Saturday, February 17, 2018, I delivered the keynote address at RIT TutorCon 2018 at the Rochester Institute of Technology in Rochester, NY. I was a student tutor at RIT from 2007-2010. The Academic Support Center asked me to speak about my experiences. Below is the transcript of my speech.

It’s good to be back in Ra-cha-cha! Happy Presidents’ Day weekend, and also Happy Chinese New Year! Let me get a good look at our tutors: If you are a tutor, please stand up.

[Wait for tutors to stand up.]

Great! It’s awesome to see so many of you here today. Is anyone in Computer Science?

Now, remain standing if you have been a tutor for at least one year.

[Wait for people to sit down.]

Not bad. What about two years?

[Wait for more people to sit down.]

Three? [Wait.] Four? [Wait.] Five? [Wait.]

What about ten years? Ten years of tutoring? [Give anyone who remains standing a round of applause, and then ask them to sit down.]

Ten years is a long time! A lot can happen; a lot can change. Here’s a question for you today, though: Will your tutoring make an impact in ten years? [Repeat the question for emphasis.]

Ten years ago, I was one of you. I was in my second year at RIT studying computer science, and I worked for the Academic Support Center and TRIO as a tutor for math, physics, and basically anything that was needed. I would have been sitting in your chair if we had these fancy tutoring conferences back them. Things were quite different a decade ago. Let me drop some knowledge bombs on you for the world in February 2008:

  • We were still on the iPhone 1. iPads did not exist yet.
  • Barack Obama was still seen as a surprise challenger to Hillary Clinton in the 2008 Democratic primaries.
  • The Great Recession was looming but had not yet hit.
  • The Summer Olympics were going to be held in Beijing, China. (Michael Phelps & Usain Bolt)
  • Lady Gaga had not yet released her debut album.

Now, let me contextualize this for RIT:

  • Bill Destler was still in his first year as university president.
  • RIT was still on the quarter system.
  • Park Point was being built.
  • The Simon Center (a.k.a. the “Toilet Bowl”) was being built.
  • The main drop-in study center was the “Math Lab” in Building 1, not Bates.

One thing that looks like it hasn’t changed, though, is Gracie’s. [Assume the audience will laugh.]

By the way, have they knocked down Riverknoll yet? I lived at 232 Kimball Drive. [Assume the audience will laugh or somehow respond.]

A lot happens in ten years. But, will your tutoring have an impact in ten years? Will the tutoring you do today benefit your students years from now? It should.

As college students, life is typically fast-paced. You have classes, you have papers, you have projects; quarters – excuse me, semesters – fly by; and it’s all over after about four years. And, for you, tutoring is just a part of that overall experience. It’s just a part-time job. As we saw earlier, most of you will spend only a few years tutoring before entering your career fields. Personally, I haven’t done any tutoring since 2010. It’s tempting to think that the time you spent tutoring doesn’t matter. So what if you help people finish their homework problems a few times a week? Students come and go anyway. It’s no big deal, right?

Well, if you’re here today at this tutoring conference, I’m pretty sure that tutoring is a big deal to you. You know it’s important. I’d be willing to bet that many of you would do tutoring even if you didn’t get paid – although, the pay is certainly deserved! I want you all to understand that what you do as a tutor will impact your students and will also impact you for the rest of your lives. Tutoring is a vector: I want you to see the line and not just the dot.

Your students come with a myriad of different circumstances. Some are just looking for a healthy environment for doing their homework. Maybe they’re stuck on a tricky physics brain-buster. Others struggle. Some really struggle – and may be one more failure away from academic suspension. But all students have one thing in common: they come to you because they want to do better. Whoever they are, they look to you as tutors to help them succeed. And every question you answer – or rather, every guiding question you turn back to them – puts them further down their paths to success. Today’s practice problems become tomorrow’s degrees. With you, they’ll learn not just the course material but, more importantly, they will learn how to learn. They will learn what questions to ask themselves. They will learn how to find answers using their resources. They will learn to teach themselves. Plutarch once said, “The mind is not a vessel to be filled but a fire to be ignited.”

With my perspective of the line, I want to give you three big ways you can make your tutoring today leave an impact for a lifetime.

First, own your role. As tutors, you have a very unique role with your students: you are peers; you are not professors. That’s a big difference! Professors are experts in their fields with years of experience and dozens of publications. You, as tutors, are students yourselves, just a few more years ahead. You can relate to your students on much more common ground. You’ve taken the same courses. You’ve taken the same tests. You’ve probably even done the very same problems. One of the tutoring tricks is to always work with a student at their level – if they sit at the table, you sit; if they stand at the board, you stand; and unless you’re making a really good example, don’t stand on the table! The equal-level principle also applies to your role as a peer tutor. There’s camaraderie. There’s energy. There’s less embarrassment to ask “stupid” questions. There’s a sense that they can do it because you can do it. So own your role as a peer tutor.

Second, focus on the student and not the problem. The problem is the dot; the student is the line. Tutors aren’t there to solve the world’s problems! Nobody comes to a tutoring center to watch a tutor show off with how much they know or how fast they can solve problems. “Look at how smart I am” – NO! Let’s be real, here: the solution to any given practice problem doesn’t really matter. What does matter is how the student learned to handle problems. Did they make an attempt? Did they look at their formulas? Did they write out their work? Did they persevere when they got stuck? Let me ask you a question: Do you think that I remember specific details to any homework assignments from ten years ago? [Wait for audience response.] Nope! But, I remember that a derivative is a rate of change. And, if I had to solve a derivative again, I’d know exactly where to look in my books to figure it out. That’s how you want your students to be in ten years. Cultivate your students to become independent.

Third, build camaraderie. Your students are already your peers – make them your friends. I don’t have any fancy statistics to share, but I know anecdotally that most students become “repeat customers.” You’ll see them again, and again, and again. Whether intended or not, you will forge relationships with your students. As your tutoring shifts become part of your everyday life, so, too, do the students who show up. Treat every single one of them the way you’d want to be treated. Work to form good relationships. Work to form trust. Be honest when you don’t know something. And furthermore, build camaraderie with your fellow tutors as well! Tutors are a team – each one brings fresh eyes and unique expertise. My specialty? Discrete math and differential equations – what a combo! We, as tutors, are trained in common techniques and share the common burdens to help our students. It’s almost like we have a special, unspoken club. I still keep up with my students and my tutors. I dined with a former student on top of the Space Needle. I partied with another on New Year’s Eve. I’m attending another student’s wedding this summer. A fellow tutor came to mine. So build camaraderie with your students and your fellow tutors.

As I close, I’d like to remind you that you are all in tutoring together. For some of you, this might just be the best job you ever have. I challenge all of you today to make your tutoring count: for now, for ten years from now, and for a lifetime. Tutors don’t make bad students good – tutors make students learn to teach themselves. That is how your tutoring will make a lifelong impact. Thank you.

The Airing of Grievances: BDD

Behavior-Driven Development – one of my favorite blog topics. When done right, it’s a wonderful way to foster better collaboration and automation. When it’s not… well, let’s just say I got a lot of problems with bad BDD practices, and now you’re gonna hear about it!

PROCESS

Treating BDD as a Tool and Not as a Process

BDD is a process – it is a set of tools and practices designed to help teams deliver better software. BDD is not just a test automation framework; the framework is just one of the tools that support BDD. Heck, the word “development” is in the name!

Complaining that Gherkin is Too Technical

Really? Really!? Gherkin is basically just plain language with some buzzwords mixed in! It is specifically designed for non-technical people to handle it! It is not a full-fledged programming language – it is essentially a simple format for behavior specification that automation frameworks can easily parse. The steps are meant to be read like plain English (or any other spoken language) so that better collaboration can happen. If Gherkin is “too technical” for you, then I hate to know what isn’t.

No Buy-In from All Roles

The three major roles on an Agile team, a.k.a the “Three Amigos,” are biz, dev, and test (regardless of fancy names or assignments). For BDD to work well, all three role types must embrace it. Otherwise, collaboration will suffer. BDD is not just a QA thing, it’s for everyone. Biz gets better features in shorter time because requirements were communicated better. Dev wastes less time figuring out what biz wants and gets tests faster. Test can start automating right away since test scenarios are defined from the start in Gherkin. Everybody wins if everybody contributes.

No Three Amigos Meetings

Three Amigos meetings are like dietary fiber supplements: they help a team stay regular with collaboration, or else development gets constipated as engineers start building crap instead of the intended behaviors. Then the crap gets blocked up as the team must rework it, meaning it could be another sprint before there’s a healthy flush of new features. Open conversations in regularly scheduled Three Amigos meetings would have avoided the whole obstruction.

Forcing QA to Write All Behavior Scenarios

BDD is not just QA thing – it is for all roles. Pigeonholing the responsibility of writing behavior scenarios onto QA is not only unfair, it is anti-collaborative. The whole reason for writing scenarios in plain language with Gherkin is to let everyone contribute to feature behavior. Scenarios are primarily about capturing behavior, not writing tests. If tests were the main focus, then engineers could just write test cases using traditional automation frameworks directly in general purpose programming languages like Java or Python. BDD offers the benefits of process efficiency and shifting left when the whole team helps to write behavior scenarios.

GHERKIN

Bad Gherkin

Only you can prevent bad Gherkin. Or I can – via rejected code reviews.

Typos, Poor Grammar, and Inconsistent Formatting

Gherkin needs to be readable. Steps with typos, poor grammar, and inconsistent formatting will still run fine for test automation, but they make it tough to understand the behaviors they describe. Sometimes, they can even make the meaning ambiguous.

No Double-Quotes Around Step Parameters

How do you know if something is a step parameter? “Double quotes” make it easy. However, Gherkin does not enforce double quotes around parameters. It is merely by programmer’s convention, but it’s a really helpful convention indeed.

No Tags

Tags make it super easy to filter scenarios at runtime. No tags? Good luck remembering long paths and names at runtime, or running related scenarios across different feature files together.

More Than 120 Characters per Line

Any longer is too much to comprehend. Either write the step more concisely, or split it apart. Plus, the line may go off the edge of the screen.

More Than 10 Steps per Scenario

Again, any longer is too much to comprehend. Scenarios should be short and sweet – they should concisely describe behavior. Too many steps means the scenario is too imperative or covers more than one behavior.

Multiple Behaviors per Scenario

Scenarios should not have multiple personality disorder: one scenario, one behavior. Don’t break the Cardinal Rule of BDD! So many people break this rule when they first start BDD because they are locked into procedure-driven thinking. Then, when tests fail, nobody knows exactly what behavior is the culprit. One scenario, one behavior.

Out-of-Order Step Types

Givens, Whens, and Thens each serve a specific, ordered purpose: Given some initial state, When actions are taken, Then verify an expected outcome. Jumbling them up ruins their meaning. Furthermore, duplicate When-Then pairs indicate multiple behaviors per scenario. And don’t just reassign step types to skirt the strict-ordering rule. Do it right – put integrity into the steps!

Gigantic Tables

Have you ever seen an Examples table with 13 columns? Or maybe 517 rows? I have. The horror, the horror! Tables that big make scenarios lose any semblance of specification-by-example. Make sure table rows and columns are actually needed. Use key-value lookups if the data is too gritty.

Being Imperative Rather Than Declarative

Given I’m logged into the app, when I click here, and I click there, and I type P, and I type L, and I type E, and I type A, and I type S, and I type E, and I type D, and I type O, and I type N, and I type T, and I type W, and I type R, and I type I, and I type T, and I type E, and I type S, and I type C, and I type E, and I type N, and I type A, and I type R, and I type I, and I type O, and I type S, and I type L, and I type I, and I type K, and I type E, and I type T, and I type H, and I type I, and I type S, then go directly to jail, and do not pass GO, and do not collect $200. Steps should focus more on what than how.

Prefixing Existing Test Procedure Steps with Gherkin Buzzwords

Let’s just take our existing test procedures from a tool like HP QualityCenter or ALM and put the words “Given,” “When,” and “Then” in front of every step. Ta-da! We’re now doing BDD! …WRONG!! I kid you not, I have see this happen. These people clearly never took BDD 101. It hurts to see.

AUTOMATION

Unorganized Step Definitions

Programmers like to throw their step definition methods anywhere. Add ’em to an unrelated existing class? Create a whole new class for only two new steps? Mix up the types? Who cares! Don’t bother to alphabetize them, either. Well, that’s how tech debt happens. That’s how duplicate steps get written, because originals can’t be found. Imagine a library without the Dewey Decimal System – that’s what an unorganized step def collection will be.

Putting Cleanup Code in Then Steps

Cleanup code belongs in After hooks, where it will be run no matter what fails during the scenario. Writing Then steps to do cleanup not only breaks step type integrity, the cleanup code will not run if a previous step aborts!

Catching and Burying All Exceptions

Here’s something I see all the time in automation code (and not just for BDD):

// FYI - This is Java, but the same thing can happen in any language
@When("^do something$")
public void doSomething() throws Throwable {
  try {
    callStuff();
  }
  catch (Exception e) {
    System.out.println(e.getMessage());
  }
}

The entirety of a step (or even a whole test) is surrounded by a try-catch that catches every exception. THIS STEP CAN NEVER REGISTER A FAILURE! Even if there was a failed assertion or, worse, an exception that ought to abort the test, it will get caught and buried with not much more than a slight whimper in the log. In this case, the test will carry forth to the next step, which will probably not work, either. I’ve seen projects with this sort of exception handling around every single step definition. In modern test frameworks, the framework will catch all exceptions at the highest level, register the test as failed, and move on safely to the next test. There is no need to catch any exception, unless the test can be recovered.

Changing Steps Without Testing Affected Scenarios

Sharing steps is a wonderful thing, but changing steps without testing all scenarios that use them is a terrible thing. I’ve seen people change step text or step def code and test only their new scenarios. Meanwhile, in the continuous integration environment, a dozen other tests using those steps started failing. (Hell, I’ve seen people push code that doesn’t even compile, but that’s another grievance.)

Multiple Names for the Same Step

Just because you can do something doesn’t mean you should. Different names for the same step may be useful for readability, but please keep the name variants limited.

No Dependency Injection

Dependency injection is the best way to share objects in an automation framework. (Singletons work well, too, but DI allows more careful control of scope.) Many frameworks like Cucumber-JVM even integrate with existing DI frameworks like PicoContainer and Spring. DON’T MAKE NON-CONSTANT VARIABLES GLOBAL! DON’T BLINDLY MAKE THINGS “STATIC” JUST TO SHARE THEM! Globals (or “statics” in Java/C# like languages) are dangerous: they can be easily misused, they are a nightmare to trace, and they can break multithreaded execution. Just use the appropriate design pattern: dependency injection.

The Airing of Grievances: Agile

Agile has essentially replaced the Waterfall model as the “right” software development methodology. It’s a really great process when it’s done right, but people ruin it when they do it wrong. And, oh, how badly it can go wrong. I got a lot of problems with bad Agile practices, and now you’re gonna hear about it!

Breaking the Rules

Agile is a lot like the board game Monopoly. The rules are long and complicated, but they are designed to make the game efficient. However, for some reason, everyone insists on making up their own rules for the game, rather than following the official instructions. For example, players won’t put a property up for auction when they land on it and refuse to buy it, or they will build houses before securing a monopoly. Then, as a result, the game goes on forever and loses its fun. In Agile, every organization seems to want to do things their own special way (as many of these grievances describe), and it almost never goes well when they do. The rules are not meant to be broken, and if they are, there will be consequences.

Going Rogue

Agile is meant to keep people focused on the most important tasks. Much time is spent planning and pivoting to stay on top of priorities. Team members should not deviate from committed work. Don’t go rogue! Don’t work on uncommitted tasks! If something is absolutely pressing, then talk with the scrum master to change the commitments.

Teams that are Too Big

How big is your Agile team? If the answer has more than one digit, then the team is too darn big. The ideal size is 5-9 people because communication becomes too hard with more. Large teams just don’t scale – it’s the law of diminishing returns.

Long Meetings

Nobody wants to be stuck in a long, boring meeting. While there are many Agile ceremonies (planning, grooming, stand-up, review, and retrospective), their meetings are meant to be efficient and productive. Stand-ups should be 15 minutes tops – nobody should ever need to give more than three sentences for their status, and nobody really wants to hear anything longer anyway! People should come prepared for planning and grooming so they don’t literally take all day. Demos should be short and sweet. Keep things moving!

Putting People on More Than One Team

Nobody should be cursed to provide deliverables for more than one Agile team. That’s not fair to the individual, who must spend double-duty in meetings, nor is it fair to the teams, who don’t have a dedicated resource for their work. It applies to every role: developer, tester, product owner, or scrum master. It also burns people out very quickly.

Too Many Top Priorities

I was once part of an Agile team where the product owner issued about a dozen “top priorities.” For. Every. Sprint. Our team had no clue what was really important.

Agonizing Over Story Points

Story points are meant to be sizing estimates for velocity. They don’t need to be perfectly accurate. They shouldn’t track hours. Don’t make big fights over it. Don’t go back and change values. Don’t twist planning poker into a political gambit. PLEASE!

Missing User Story Descriptions

The user story is the primary work artifact. It tells how a new feature should work from the perspective of the user… or, at least it should. If your user story contains just one line (like saying “Build the profile page”), then you just might be doing it wrong. Write user stories in the “As a ___, I want ___, so that ___” format, and provide extra descriptions to help the team understand what the story covers. Non-descriptive stories lead to poorly developed features.

Missing Acceptance Criteria

How do we know when a story is complete? If there’s no acceptance criteria, we don’t! Testers also won’t know what to check. Please write helpful acceptance criteria. A bullet list is fine, and Gherkin would be even better.

Not Including Testing and Automation in the Definition of Done

No. No. No. No. No. No. NO! A story is not complete if it is not tested. It must not be accepted without tests passing and automated. Otherwise, be prepared for an avalanche of technical debt as bugs pile up and the team can’t keep up. The premise of Agile is to deliver small, working features in iterations. Testing must be included! Don’t create separate stories for testing. Don’t push it off to the next sprint. If a team cannot get testing done, then perhaps it should increase story point sizings to include testing and/or commit to less work during a sprint.

Blaming QA for Incomplete Stories

I once heard a developer say bluntly to my automation team, “QA is the bottleneck.” Don’t shoot the messenger! Tests fail because the product under test has problems. Many times, testers don’t even receive builds until very late in the sprint. When stories don’t get done, don’t start a blame game – it’s the whole team’s fault. Try shifting left (perhaps by using BDD) or committing to less work per sprint.

Ignoring Technical Debt

Technical debt is the cost of consequences from poor development decisions. Examples may include: using single-threading when multi-threading is needed, avoiding design patterns, and even building up a test automation framework. Product owners don’t seem to like tech debt tasks because they don’t deliver new features. Unfortunately, tech debt will often cripple a team’s ability to deliver new features – pay now or pay later. Don’t ignore tech debt!

Confusing Agile with “Short Waterfall”

Agile is meant to be a process paradigm shift. It is not meant to be a condensed version of the Waterfall model. Sprints should be short. Responsibilities should be shared. Teams should be self-empowered. Break down silos and become truly Agile!

Using “Agile” and “Lean” Interchangeably

The Lean Startup is a methodology for starting a new business using minimal overhead and reacting quickly to lessons learned. It involves using Agile for product development, but it encompasses so much more than just Agile. Don’t use the terms interchangeably! Get on point with your buzzword bingo game.

Misusing the Term “Continuous Integration”

A nightly build is not CI. A weekly regression run is not CI. Manually-triggered tests are not CI. Manual deployments are not CI. Hand-written test reports are not CI. Don’t lie to yourself – CI is continuous integration, and everything must be automatic.

Forcing Scrum When Kanban May Be Better

Scrum is probably the most widely used Agile process, to the point where most people presume “Agile” means “Scrum.” However, Scrum is not appropriate for all teams. Kanban is a much better process when work items must be done “just in time” – like tech support tickets, build deployments, system maintenance, or emergency recoveries. Good candidates for Kanban are IT help desks and DevOps teams. I’ve used Kanban on automation tools/frameworks teams very successfully. Don’t shoehorn everyone into Scrum.

Hanging Agile Manifesto Posters on the Wall

What are you, Communist?

Complaining about Agile

Complaining doesn’t make it better! Honestly, in my experience, the worst complainers are old-school people who just don’t like change. Then, problems become a self-fulfilling prophecy. Or, they try to break rules and then gripe when things don’t work. If your complaint is about Agile in general, then go take a long, hard look in the mirror. However, if you find a problem in how your team is doing Agile, then bring it up during the retrospective – that’s Agile’s auto-correct mechanism. Complaining for complaint’s sake drags everybody down.

The Airing of Grievances: Selenium WebDriver

Selenium WebDriver is the de facto standard for Web UI automation. It’s a great tool, but like anything good, it can also be misused. And that’s where I have grievances. I got a lot of problems with Selenium WebDriver abuses, and now you’re gonna hear about it!

WebDriver “Unit Tests”

“WebDriver unit tests” are like square circles – definitionally, they are logical fallacies. In my books, a unit test must be white box, meaning it has direct access to the product code. However, Web UI tests using WebDriver are inherently black box tests because they are interacting with an actively running website. Thus, they must be above-unit tests by definition. Don’t call them unit tests!

Making Every Test a Web Test

NO! The Testing Pyramid is vital to a healthy overall testing strategy. Web tests are great because they test a website in the ways a user would interact with it, but they have a significant cost. As compared to lower-level tests, they are more fragile, they require more development resources, and they take much more time to run. Browser differences may also affect testing. Furthermore, problems in lower level components should be caught at those lower levels! Sure, HTTP 400s and 500s will appear at the web app layer, but they would be much faster to find and fix with service layer tests. Different layers of testing mitigate risk at their optimal returns-on-investment.

No WebDriver Cleanup

Every WebDriver instance spawns a new system process for “driving” web browser interactions. When the test automation process completes, the WebDriver process may not necessary terminate with it. It is imperative that test automation quits the WebDriver instance once testing is complete. Make sure cleanup happens even when abortive exceptions occur! Otherwise, zombie WebDriver processes may continue on the test machine, causing any number of problems: locked files and directories, high memory usage, wasted CPU cycles, and blocked network ports. These problems can cripple a system and even break future test runs, especially on shared testing machines (like Jenkins nodes). Please, only you can stop the zombie apocalypse – always quit WebDriver instances!

Using “Close” Instead of “Quit”

Regardless of programming language, the WebDriver class has both “close” and “quit” methods. “Close” will close the current browser tab or window, while “quit” will close all windows and terminate the WebDriver process. Make sure to quit during final cleanup. Doing only a close may result in zombie WebDriver processes. It’s a rookie mistake.

Not Optimizing Setup/Cleanup with Service Calls

Web tests are notoriously slow. Whenever you can speed them up, do it! Some tests can be optimized by preparing initial state with service calls. For example, let’s say a user visiting a car dealership website needs to have favorite cars pre-selected for a comparison page test. Rather than navigating to a bunch of car pages and clicking a “favorite” icon, make a setup routine that calls a service to select favorites. Not all tests can do this sort of optimization, but definitely do it for those that can!

Web Elements with No ID

Developers, we need to talk – give every significant element a unique ID. PLEASE! WebDriver calls are so much easier to write and so much more robust to run when locator queries can use IDs instead of CSS selectors or XPaths. Let’s pick ID names during our Three Amigos meetings so that I can program the tests while you develop the features. Determining what elements are import should be easy based on our wireframes. You will save us automators so much time and frustration, since we won’t need to dig through HTML and wonder why our XPaths don’t work.

Changing Web Elements Without Warning

Hey, another thing, developers – don’t change the web page structure without telling us! WebDriver locator queries will break if you change the web elements. Even a seemingly innocuous change could wipe out hundreds of tests. Automation effort is non-trivial. Changes must be planned and sized with automation considerations in mind.

Not Using the Page Object Model

The Page Object Model is a widely-used design pattern for modeling a web page (or components on a web page) as an object in terms of its web elements and user interactions with it. It abstracts Web UI interactions into a common layer that can be reused by many different tests. (The Screenplay pattern, also good, is an evolution of the Page Object Model; tutorial here.) Not using the Page Object Model is Selenium suicide. It will result in rampant code duplication.

Demonizing XPath

XPaths have long been criticized for being slower than CSS selectors. That claim is outdated baloney. In many cases, XPaths outperform CSS selectors – see here, here, and here. Another common complaint is that XPath syntax is more complicated than CSS selector syntax. Honestly, I think they’re about the same in terms of learning curve. XPaths are also more powerful that CSS selectors because they can uniquely pinpoint any element on the page.

Inefficient Web Element Access

Web element IDs make access extremely efficient. However, when IDs are not provided, other locator query types are needed. It is always better to use locator queries to pinpoint elements, rather than to get a list of elements (or even a parent/child chain) to traverse using programming code. For example, I often see code reviews in which an XPath returns a list of results with text labels, and then the programming code (C# or Java or whatever) has a for loop that iterates over each element in the list and exits when the element with the desired label is found. Just add “[text()=’desired text’]” or “[contains(text(), ‘desired text’)]” to the XPath! Use locator queries for all they’re worth.

Interacting with Web Elements Before the Page is Ready

Web UI test automation is inherently full of race conditions. Make sure the elements are ready before calling them, or else face a bunch of “element not found” exceptions. Use WebDriver waits for efficient waiting. Do not use hard sleeps (like Java’s Thread.sleep).

Untuned Timeouts

WebDriver calls need timeouts, or else they could hang forever if there is a problem. (Check online docs for default timeout values.) Timeout value ought to be tuned appropriately for different test environments and different websites. Timeouts that are too short will unnecessarily abort tests, while timeouts that are too long will lengthen precious test runtime.

The Airing of Grievances: Test Automation Code

More grievances! Test code ought to be developed with the same high standards as product code, but often it is neglected. That bothers me so much, and it costs teams a significant amount of time and money through its bad consequences. I got a lot of problems with bad test automation code, and now you’re gonna hear about it!

Copypasta

Code duplication is code cancer. It is particularly rampant in test automation because test steps are often repetitive. That’s no excuse, however, to allow it. Use better programming practices, or face my code review rejections!

Hard-Coded Configuration Data

Automation should be able to run on any environment without hassle. Don’t hard-code URLs, usernames, passwords, and other config-specific values! Read them in as inputs or config files. Nothing is more frustrating than switching environments but – surprise! – not running with the correct values.

Re-reading Inputs for Every Single Test

Config files and input values should not change during a test suite run. So, don’t repeatedly read them! That’s inefficient. Read them once, and hold them in memory in a centrally-accessible location.

Leaving Temporary Files or Settings

Clean up your mess! Don’t leave temporary files or settings after a test completes. Not doing proper cleanup can harm future tests. Files can also build up and eventually run the system out of storage space. It’s a Boy Scout principle: Leave No Trace!

Incorrect Test Results

Test results need integrity to be trusted. Watch out for false positives and false negatives. If there’s a problem during a test, handle it – don’t ignore it. Don’t skip assertion calls. Business decisions are made based on test results.

Uninformative Failure Messages

Here’s one: “Failed.” That tells me nothing. Why did the test fail? Was something missing from a web page? Was there an unexpected exception? Did a service call return 500? TELL ME! Otherwise, I need to waste time rerunning and even debugging the test. The failure may also be intermittent. Tell me what the problem is right when it happens.

Making Assertion Calls in Automation Support Code

Every automated test case must make assertions to verify goodness or badness of system conditions. But, as a best practice, those assertion calls should be written only in the test case code – not in support code like page objects or framework-level classes. Assertions are test-specific, while support code is generic. Support code should simply give system state, while test cases use assertions to validate system state. If support code has assertion calls, then it is far less reusable and traceable.

Treating Boolean Methods like Assertions

This is an all-too-common rookie mistake. Boolean methods simply return a true/false value. They do not (or, as least according to the previous grievance, should not) contain assertion calls. However, I’ve seen many code reviews where programmers call a Boolean method but do nothing with the result. Whoops!

Burying Exceptions

Exceptions mean problems. Test automation is meant to expose problems. Don’t blindly catch all exceptions, log a message, and continue on with a test like nothing’s wrong. Let the exception rise! Most modern test frameworks will catch all exceptions at the test case level, automatically report them as failures, and safely move on to the next test. There is no need to catch exceptions yourself if they ought to abort the test, anyway – that adds a lot of unnecessary code. Catch exceptions only if they are recoverable!

No Automatic Recovery

True automation means no manual intervention. An automation framework should have built-in ways to automatically recover from common problems. For example, if a network connection breaks, automatically attempt to reconnect. If a test fails, retry it. If too many failures happen, either end the run early or pause for some time. And make sure recovery mechanisms are built into the framework, rather than appearing as copypasta throughout the codebase. The purpose of retries (especially for tests) is not to blindly run tests until they turn from red to green, but rather to overcome unexpected interruptions and also to gather more data to better understand failure reasons. Interruptions will happen – handle them when they do. And be sure to log any failures that do happen so that the reasons may be investigated.

No Logging

Please leave a helpful trail of log messages. Tracing through automation code can be difficult after a failure, especially when you’re not the original author. Logging helps everyone quickly get to the root cause of failures. It also really helps to create reproduction scenarios. If I can copy the log from a failing tests into a bug report and be done, then AMEN for an easy triage!

Hard Waits

Ain’t nobody got time for that! Automation needs to be fast, because time is money. Forcing Thread.sleep (or other such equivalent in your language of choice) shows either laziness or desperation. It unnecessarily wastes precious runtime. It also creates race conditions: what if the wait time ends before the thing is ready? Always use “smart” waits – actively and repeatedly check the system at short intervals for the thing to be ready, and abort after a healthy timeout value.

The Airing of Grievances: Test Automation Process

Test automation is a big deal for me. It is my chosen specialty within the broad field of software. When I see things done wrong, or when people just don’t get what it’s about, it really grinds my gears. I got a lot of problems with bad test automation processes, and now you’re gonna hear about it!

Saying “They’re Just Test Scripts”

Test automation is not just a bunch of test scripts: it is a full technology stack that requires design, integration, and expertise. Test automation development is a discipline. Saying it is just a bunch of test scripts is derogatory and demeaning. It devalues the effort test automation requires, which can lead to poor work item sizings and an “us vs. them” attitude between developers and QA.

Not Applying the Same Software Development Best Practices

Test automation is software development, and all the same best practices should thus apply. Write clean, well-designed code. Use version control with code reviews. Add comments and doc. Don’t get lazy because “they’re just test scripts” – wrong attitude!

Lip Service

Don’t say automation is important but then never dedicate time or resources to work on it. Don’t leave automation as a task to complete only if there’s time after manual testing is done. Make automation a priority, or else it will never get done! I once worked on an Agile team where automation framework stories were never included into the sprint because there weren’t “enough points to go around.” So, even though this company hired me explicitly to do test automation, I always got shunted into a manual testing scramble every sprint.

Confusing Test Automation with Deployment Automation

Test automation is the automation of test scenarios (for either functional or performance tests). Deployment automation is the automation of product build distribution and installation in a software environment. They are two different concerns. Cucumber is not Ansible.

Forcing 100% Automation

Some people think that automation will totally eliminate the need for any manual testing. That’s simply not true. Automation and manual testing are complementary. Automation should handle deterministic scenarios with a worthwhile return-on-investment to automate, while manual testing should focus on exploratory testing, user experience (UX), and tests that are too complicated to automate properly. Forcing 100% automation will make teams focus on metrics instead of quality and effectiveness.

Downsizing or Eliminating QA

Test automation doesn’t reduce or eliminate the need for testers. On the contrary, test automation requires even more advanced skills than old-school manual testing. There is still a need for testing roles, whether as a dedicated position or as shared collectively by a bunch of developers. The work done by that testing role just becomes more technical.

Saying Product Code and Test Code Must Use the Same Language

For unit tests, this is true, but for above-unit tests, it is simply false. Any general purpose programming language could be used to automate black-box tests. For example, Python tests could run against an Angular web app. A team may choose to use the same language for product and test code for simplicity, but it is not mandatory.

Not Classifying Test Types

Not all tests are the same. You can play buzzword bingo with all the different test type names: unit, integration, end-to-end, functional, performance, system, contract, exploratory, stress, limits, longevity, test-to-break, etc. Different tests need different tools or frameworks. Tests should also be written at the appropriate Testing Pyramid level.

Assuming All Tests Are Equal

Again, not all tests are the same, even within the same test type. Tests vary in development time, runtime, and maintenance time. It’s not accurate to compare individuals or teams merely on test numbers.

Not Prioritizing Tests to Automate

There’s never enough time to automate everything. Pick the most important ones to automate first – typically the core, highest-priority features. Don’t dilly-dally on unimportant tests.

Not Running Tests Regularly

Automated tests need to run at least once daily, if not continuously in Continuous Integration. Otherwise, the return-on-investment is just too low to justify the automation work. I once worked on a QA team that would run an automated test only once or twice during a 2-year release! How wasteful.

HP Quality Center / ALM

This tool f*&@$#! sucks. Don’t use it for automated tests. Pocket the money and just develop a good codebase with decent doc in the code, and rely upon other dashboards (like Jenkins or Kibana) for test reporting.

The Airing of Grievances: Version Control

Let the Airing of Grievances series continue: I got a lot of problems with version control misuse, and now you’re gonna hear about it!

Not Using Version Control

You’ve got to be some special kind of stupid to not use a version control system. Software is just too dang fragile to go without protection.

Using Outdated Version Control Systems

Still using CVS like it’s 1999? How about Rational ClearCase? If so, it’s time to upgrade. Git seems to be the go-to standard these days, though Subversion still has a place for projects where centralized control is better than distributed control. My opinion? Just use Git.

Gigantic Commits

Code changes ought to be incremental and small, and they ought to be committed in frequent intervals. However, some people like to make one giant, killer commit at a time with bajillions of lines of changes. Nobody wants to review those pull requests. Break things up into smaller pieces!

No Comments with Commits

Look back in your version control history to see how many messages look like this:

  • .
  • updated
  • fixed
  • done

Really? How is this helpful? Please give a meaningful message. It doesn’t need to be long – a one-liner is fine. But describe what makes the commit significant. Otherwise, tracing through history is dang near impossible!

Never Committing Changes

For whatever reason, some people will check out code, make changes, run locally, and NEVER commit the changes back to the repository. This happened frequently at a previous job with offshore test automation contractors. They would add new tests and fix bugs but never share them back with the team. They’d even email code back and forth, rather than commit! I just can’t even.

Committing Code that Doesn’t Even Compile

jackie-chan-wtf

Never Pushing Changes

In Git, there is a difference between “commit” (which commits the change locally) and “push” (which pushes all local commits to the remote repository). Some people never push. Then, they wonder why they can’t open pull requests. Or, they lose their code after a Blue Screen wipes out their machine. Make it a habit to push at the end of the workday, whether it’s needed or not.

No Branching

Branches are like swim lanes: every contributor (or group) can develop code without interfering with others. They also make concurrent release work possible. Using only one branch doesn’t “simplify” development – it just causes an integration mess. For example, I once worked in an organization where the QA architect insisted that test automation should not use multiple branches and, instead, have if/else conditions for differences between release branches. (I fought tooth-and-nail against it and lost.) Code duplication became rampant. Always adopt a good branching strategy, even for test automation. Gitflow is a good example workflow.

Stale Branches

Stale branches mean old code and merge conflicts. Nobody wants those. Keep your local branches up-to-date.

Not Deleting Feature Branches

In Git, feature branches are meant to have a short lifespan: you create one to develop a new feature or fix a bug or whatever, and then you delete it after the pull request is completed. However, some people don’t delete those old branches. Over time on a team, those old branches really add up and pollute the repository. Delete them as a common courtesy.

Botched Merge Conflict Resolution

Merge conflicts themselves are not the grievance. Nobody likes them, but they are inevitable on any team. However, botched merge conflict resolution is a HUGE grievance. Would you be mad if you spent a lot of time fixing a problem, only to have some other schmuck accidentally undo your code change with an overwrite? Please be careful when merging. If you aren’t sure that your merge will be good, there’s no shame in asking for help.

Not Re-compiling and Re-testing after Merging

What’s worse than messing up a merge? Committing the changes without testing them first! Merges are risky, and mistakes happen – but that’s why it is imperative to make sure everything is still good after a merge. I’ve seen people blindly post pull request updates after resolving merge conflicts, only to have the build fail with compiler warnings. Don’t do that.

Granting Everyone Full Repository Permissions

While everyone on the team should contribute, not everyone should contribute in the same ways. If there are no security policies set up, then users will do dangerous things, whether accidentally or deliberately. They could circumvent the review process. They could rename things unexpectedly. One time, I saw a guy delete the remote master branch! (Thank goodness we recovered it quickly.) Put permissions into place before bad things happen.