Which programming languages are best for writing test automation? There are several choices – just look at this list on Wikipedia and this cool decision graphs for choosing languages. While this topic can quickly devolve into a spat over personal tastes, I do believe there are objective reasons for why some languages are better for automating test cases than others.
Dividing Test Layers
First of all, unit tests should always be written in the same language as the product under test. Otherwise, they would definitionally no longer be unit tests! Unit tests are white box and need direct access to the product source code. This allows them to cover functions, methods, and classes.
The question at hand pertains more to higher-layer functional tests. These tests fall into many (potentially overlapping) categories: integration, end-to-end, system, acceptance, regression, and even performance. Since they are all typically black box, higher-layer tests do not necessarily need to be written in the same language as the product under test.
My Opinionated Choices
Personally, I think Python is today’s best all-around language for test automation. Python is wonderful because its conciseness lets the programmer expressively capture the essence of the test case. It also has very rich test support packages. Check out this article: Why Python is Great for Test Automation. Java is a good choice as well – it has a rich platform of tools and packages, and continuous integration with Java is easy with Maven/Gradle/ANT and Jenkins. I’ve heard that Ruby is another good choice for reasons similar to Python, but I have not used it myself.
Some languages are good in specific domains. For example, JavaScript is great for pure web app testing (à la Jasmine, Karma, and Protractor) but not so good for general purposes (despite Node.js running anywhere). A good reason to use JavaScript for testing would be MEAN stack development. TypeScript would be even better because it is safer and scales better. C# is great for Microsoft shops and has great test support, but it lives in the Microsoft bubble. .NET development tools are not always free, and command line operations can be painful.
Other languages are poor choices for test automation. While they could be used for automation, they likely should not be used. C and C++ are inconvenient because they are very low-level and lack robust frameworks. Perl is dangerous because it simply does not provide the consistency and structure for scalable, self-documenting code. Functional languages like LISP and Haskell are difficult because they do not translate well from test case procedures. They may be useful, however, for some lower-level data testing.
8 Criteria for Evaluation
There are eight major points to consider when evaluating any language for automation. These criteria specifically assess the language from a perspective of purity and usability, not necessarily from a perspective of immediate project needs.
- Usability. A good automation language is fairly high-level and should handle rote tasks like memory management. Lower learning curves are preferable. Development speed is also important for deadlines.
- Elegance. The process of translating test case procedures into code must be easy and clear. Test code should also be concise and self-documenting for maintainability.
- Available Test Frameworks. Frameworks provide basic needs such as fixtures, setup/cleanup, logging, and reporting. Examples include Cucumber and xUnit.
- Available Packages. It is better to use off-the-shelf packages for common operations, such as web drivers (Selenium), HTTP requests, and SSH.
- Powerful Command Line. A good CLI makes launching tests easy. This is critical for continuous integration, where tests cannot be launched manually.
- Easy Build Integration. Build automation should launch tests and report results. Difficult integration is a DevOps nightmare.
- IDE Support. Because Notepad and vim just don’t cut it for big projects.
- Industry Adoption. Support is good. If the language remains popular, then frameworks and packages will be maintained well.
Below, I rated each point for a few popular languages:
| Python | Java | JavaScript | C# | C/C++ | Perl | |
| Usability | awesome | good | good | good | terrible | poor |
| Elegance | awesome | good | okay | good | poor | poor |
| Available Test Frameworks | awesome | awesome | awesome | good | okay | poor |
| Available Packages | awesome | awesome | okay | good | good | good |
| Powerful Command Line | awesome | good | good | okay | poor | okay |
| Easy Build Integration | good | good | good | good | poor | poor |
| IDE Support | good | awesome | good | good | okay | terrible |
| Industry Adoption | awesome | awesome | awesome | good | terrible | terrible |
Conclusion
I won’t shy away from my preference for Python, but I recognize that they may not be the right choice for all situations. For example, when I worked at LexisNexis, we used C# because management wanted developers, who wrote the app in C#, to contribute to test automation.
Now, a truly nifty idea would be to create a domain-specific language for test automation, but that must be a topic for another post.
UPDATE: I changed some recommendations on 4/18/2018.
Automation Panda 
True Python stands out absolutely for its ease of implementation, its object orientedness being its core feature.
LikeLike
Interesting post, lots of good points. I do mainly C# and lisp coding nowdays, just a little bit of Python on the side. I don’t think that lisp is particularly badly suited for test automation, especially since it’s great language for writing DSLs for a particular task at hand. True, there’s quite big focus on functional aspects on it, especially in Clojure world, but that doesn’t necessarily hinder test case writing, you just have to look things from slightly different angle.
If I may be so bold, here’s a link to a testing library (or rather glue between several libraries) written in Hy (lisp embedded inside Python) that I have been tinkering with: http://archimedes.readthedocs.io/en/latest/ It adds some basic building blocks for DSL aimed specifically for testing (test case specification, test data generation, writing object matchers). It’s aimed towards relatively low level tests (unit and integration). DSLs are nice way to raise abstraction level, so you don’t get lost with objects, strings and magic numbers.
Same probably applies to Haskell, although I haven’t written that much code with it. It however has famous QuickCheck library that makes generating test data breeze. And the fact that most parts of a Haskell program are just functions (data in, data out, no side-effects), testing them separately from rest of the code is simple.
C# isn’t that bad, when you find correct libraries (Fluent Assertions comes to mind first) and tools. I like that when using TFS and related tools, everything is so nicely integrated together.
Python on the other hand is pure joy to work with. I don’t know if you’re aware of Hypothesis (http://hypothesis.readthedocs.io/en/latest/) for test data generation. It adds yet another angle to testing by enabling property based testing. And PyHamcrest (https://pypi.python.org/pypi/PyHamcrest) adds ability to write object matchers (they probably could be useful to other things too, but I’m using them mainly for testing).
All this from point of view of developer, who doesn’t do much UI testing, but plenty of unit and integration testing.
Also, I like reading your blog. Looking forward seeing more posts in the future.
LikeLike
tuturto, thanks for following my blog and for commenting!
I wholeheartedly agree with you that DSLs are great for test automation. They can expressively capture many redundant test operations that often cause copypasta. When I worked at NetApp, I developed a testing DSL. As I alluded in the post, I hope to write more about DSLs and testing in future posts.
Personally, I love functional programming, and I understand you completely when you say that FP requires a slightly different angle of approach. However, I still maintain that FP is often not a pragmatic paradigm for higher-level test automation. Typical test cases are conceived as procedures: do this, then this, then this. It is best for maintainability that the automation code outlines the test procedure as clearly as possible. This becomes difficult in FP for longer or more intense test cases. It’s not simply a matter of the libraries a language has for testing – any general-purpose language has testing capabilities. Some tests also require side effects, which is something FP attempts to avoid. Furthermore, from my experiences, most engineers in the discipline of test automation are largely ignorant of FP. For example, I’ve heard others say that C is a functional language because it has functions! There are some domains for which FP languages are great for above-unit testing, such as data processing, but I would not recommend FP for testing in most projects.
LikeLike
Scripting engines or Scripting languages like PHP, JavaScript and the popular scripting language Python are very nice but they have limitations also. Python may work well for many testing application but it’s all based on what you testing to make the most effective choices. I’m always so curious why these scripting languages however, are so often misrepresented as programming languages. I would like to see someone Program an A15 with Python script and have it boot… lol
LikeLike
I appreciated your post, and agree with @tuturto that there are lots of good points made here! In my career I have done a fair amount of consulting for various clients, and inevitably each client will have their own “preferred” language for development purposes. I have had to learn to be versatile based on client needs, but my personal preferences are also Python and Java. Currently I am trying a bit of PHP based on, yet again, corporate need; so far it hasn’t been terrible!
My career took a turn toward Front End validation, but lately I’ve rediscovered my passion for creation through the use of Robot Framework and Python! I was wondering if you had ever given Robot Framework a try and what your thoughts were?
LikeLike
Hi @cbramsell! Thanks for reading the blog, and thanks for commenting!
I have not used Robot Framework myself, but I have heard about it. From the very little bit I have seen, it looks great for handling test data/inputs/variation. Robot Framework is also self-billed at ATDD. Personally, I prefer BDD over ATDD – though they are very similar, BDD is more business-oriented, whereas ATDD is more developer-oriented. Definitely check out my BDD posts if you are interested in it.
LikeLike
Thanks for the post.
LISP is not a purely functional language though.
LikeLike
Hey Andy, I stumbled into your blog and found myself a software testing wonderland! Really cool posts and awesome viewpoint, thanks for sharing all that and keep it up! :).
Personally I’ve been using Python for test automation development on various projects and technology stacks for 3 years. I found the same as you pointed out, perhaps one thing to add is it has the best of both worlds as in (scripting and OOP), to do test automation that is pure gold.
I’ve build a framework which incorporate BDD (behave) and data-driven (scv file as test input) in Python for a Bank’s trade reporting engine backend system (Java + JMS + Oracle) that works perfectly for progression and regression test for the team.
I’ve also done a Protractor + Python test automation solution for E2E testing a SPA project in the past, it’s just so flexible and versatile.
I’ve used C# Specflow and Groovy in SoapSUI as well as Java in the past, but if you let me choose freely, it has to be Python.
The other point is for the tester in the team who is not so strong at coding, based on my experience they tend to pick up Python fairly quickly, the learning curve is significantly short than the framework done in Java for C#.
LikeLike
Hello,
Which language is more effective for automation script and Which tool is best for automation? In that any automation script like slider, upload profile photo, upload document can be easily done.
LikeLike
Hi Pooja. For Web interactions, look into Selenium WebDriver (https://www.seleniumhq.org/), which provides packages in many big languages (Java, C#, Python, Ruby, JavaScript).
LikeLike
Ruby is missing 😉
LikeLike
I did call out Ruby in the article, but since I haven’t used it myself, I did not include it in the comparison table.
LikeLike
Nice article. the programming language is mostly used in the automation filed is python , prel , and its working windows is Linux. we also provide the selenium software testing course in trichy in all streaming students.
LikeLike