Testing

“In God, we trust. In all else, we test.” (Unknown)

Overview

Testing is a vital aspect of software development. It is also a very complex discipline within software engineering. Definitions may vary and even overlap, but there are two primary categories for testing:

  1. Functional – does the software work correctly
  2. Performance – does the software work within desired metrics

Functional tests yield a deterministic pass/fail result. They can be written at many layers:

  1. Unit – very short tests for individual “units” (functions, methods, or classes) of code
  2. Integration – tests that cover where two components come together
  3. End-to-End – lengthier tests that cover an execution path through a system

These layers form the Testing Pyramid. Unit tests should be white-box tests written in the same language as the product code and run automatically as part of product builds with near-100% code coverage. Integration and end-to-end tests should be black-box tests run against live, deployed product builds. They are more complex and require more investment.

Performance tests yield measurement under given system conditions to be evaluated based on ranges of acceptability (example: < 500ms response time). Performance tests often need special tools or instrumentation like JMeter.

There are two ways to run tests:

  1. Manual – a person performs test steps themselves
  2. Automated – special software is developed to automatically run tests

Test automation is my professional specialty. It makes it possible to reliably run tests automatically and repeatedly, which is essential for continuous integration and continuous delivery. However, not all tests can or should be automated due to varying returns-on-investment. Furthermore, manual exploratory testing is a better way to discover unexpected problems or edge cases. The two approaches complement each other: automated testing is best for defensive test procedures, and manual testing is best for offensive exploration.

Major Posts

I love to develop test automation in Python. Check out the Python page for the Python Testing 101 series.

Testing is a big part of behavior-driven development. Check out the BDD page for more information.

Automation Frameworks