Which programming languages are best for writing test automation? There are several choices – just look at this list on Wikipedia and this cool decision graphs for choosing languages. While this topic can quickly devolve into a spat over personal tastes, I do believe there are objective reasons for why some languages are better for automating test cases than others.
Dividing Test Layers
First of all, unit tests should always be written in the same language as the product under test. Otherwise, they would definitionally no longer be unit tests! Unit tests are white box and need direct access to the product source code. This allows them to cover functions, methods, and classes.
The question at hand pertains more to higher-layer functional tests. These tests fall into many (potentially overlapping) categories: integration, end-to-end, system, acceptance, regression, and even performance. Since they are all typically black box, higher-layer tests do not necessarily need to be written in the same language as the product under test.
On the other hand, languages like C, C++, C#, and Perl are less suitable for test automation. C and C++ are very low-level and lack robust frameworks. Although C# as a language is similar to Java, it lives in the Microsoft bubble: .NET development tools are not as friendly or as free, and command line operations are painful. Perl simply does not provide the consistency and structure for scalable and self-documenting code. Purely functional languages like LISP and Haskell are also poor choices for test automation because they do not translate well from test case procedures. They may be useful, however, for some lower-level data testing.
8 Criteria for Evaluation
There are eight major points to consider when evaluating any language for automation. These criteria specifically assess the language from a perspective of purity and usability, not necessarily from a perspective of immediate project needs.
- Usability. A good automation language is fairly high-level and should handle rote tasks like memory management. Lower learning curves are also preferable. Development speed is also important for deadlines.
- Elegance. The process of translating test case procedures into code must be easy and clear. Test code should also be concise and self-documenting for maintainability.
- Available Test Frameworks. Frameworks provide basic needs such as assertions, setup/cleanup, logging, and reporting. Examples include Cucumber and xUnit.
- Available Packages. It is better to use off-the-shelf packages for common operations, such as web drivers (Selenium), HTTP requests, and SSH.
- Powerful Command Line. A good CLI makes launching tests easy. This is critical for continuous integration, where tests cannot be launched manually.
- Easy Build Integration. Build automation should launch tests and report results. Difficult integration is a DevOps nightmare.
- IDE Support. Because Notepad and vim just don’t cut it for big projects.
- Industry Adoption. Support is good. If the language remains popular, then frameworks and packages will be maintained well.
Below, I rated each point for a few popular languages:
|Available Test Frameworks||awesome||awesome||good||okay||poor|
|Powerful Command Line||awesome||good||terrible||poor||okay|
|Easy Build Integration||good||awesome||poor||poor||poor|
I won’t shy away from my preference for Python and Java, but I recognize that they may not be the right choice for all situations. For example, we use C# at my current job because our app is written in C# and management wants developers and QA to be on the same page.
Now, a truly nifty idea would be to create a domain-specific language for test automation, but that must be a topic for another post.