Testing

“In God, we trust. In all else, we test.” (Unknown)

What is Software Testing?

Software testing is the practice of:

  • validating goodness and identifying badness
  • in software product code and features
  • for the purpose of enforcing high quality standards

There are 3 fundamental categories of software testing:

  1. Functional: Does the software work correctly?
  2. Performance: Does the software work within desired system metrics?
  3. Experimental: Does the software work with improved feature metrics after a change?

Functional Testing

Functional testing determines if software features work correctly. They yield a deterministic pass/fail result.

“Correctness” means mutually-agreed goodness. It may be determined by:

  • requirements
  • specifications
  • test cases and plans
  • domain knowledge
  • user expectations
  • common sense

Modes and Methods

There are two modes of functional testing:

  1. Scripted: Test cases are written, reviewed, and then run according to plan.
  2. Exploratory: Experts interact with product features without a script in attempts to uncover issues.

There are also two methods for running functional tests:

  1. Automated: Test software runs tests automatically without manual intervention.
    1. Best at being defensive to protect software against regressions.
    2. A good fit for scripted tests.
  2. Manual: A human tester exercises system behavior through direct interaction.
    1. Best at being offensive to find bugs in new features.
    2. A good fit for exploratory tests and for scripted tests that are difficult to automate.

The methods and modes are complementary: one does not supplant another.

Levels and Layers

There are two access levels for functional testing:

  1. White-box: Tests interact directly with product code, thus covering code.
  2. Black-box: Tests interact with a live instance of the built product, thus covering features.

There are also three layers for functional testing:

  1. Unit: Very short white-box tests for individual “units” (functions, methods, or classes) of code.
  2. Integration: Black-box tests that cover where two components come together (often service layer).
  3. End-to-End: Lengthier black-box tests that cover an execution path through a system (often Web UI layer).

These three layers form the Testing Pyramid.

Test Automation

Test automation is indispensable for software development success. Tests should be automated when they give positive returns-on-investment.

AdvantagesChallenges
  • Tests can be rerun at any time the same way.
  • Tests can run as part of CI/CD.
  • Teams can run more tests in less time.
  • Setup and cleanup can also be automated.
  • Frameworks can provide reports, logs, and screenshots.
  • Manual testers can focus on exploratory testing.
  • Test automation is software development and requires the same skills and practices.
  • Automation is inherently fragile because it depends upon ever-changing products.
  • Interruptions and intermittent failures can easily break automated tests.
  • Tests should be automated at the same time the feature is developed to give the best feedback.

Functional test frameworks and related packages are available in all major programming languages. Don’t reinvent wheels.

LanguageFrameworks
C++
C#
Java
JavaScript
Ruby
Perl
PHP
Python
Scala

Advice

  • Each test should focus on one main behavior or variation.
  • Tests should run independently of each other. (They should be runnable in any order.)
  • Tests should cover behaviors as close to the point of origin as possible.
  • Tests should be self-descriptive and intuitively understandable.
  • Count(unit) > Count(integration) > Count(end-to-end)
  • Target near-100% code coverage for unit tests.
  • Do not automate every test – use a risk-based strategy with ROI.
  • Try to run automated tests in parallel, especially in CI/CD.

Looking for free courses on test automation? Check out Test Automation University!

Performance Testing

Performance testing determines if a software product works well. As a precondition, features must work correctly (meaning, functional tests pass).

Performance testing should not use functional testing frameworks. They should use performance-specific tools like JMeter or Visual Studio load testing.

“Good” performance means minimal impact to the 4 primary software performance metrics in the system under test:

  1. Processor Usage
  2. Memory Usage
  3. Response Time
  4. Throughput

Major Posts

I love to develop test automation in Python. Check out the Python page for the Python Testing 101 series.

Testing is a big part of behavior-driven development. Check out the BDD page for more information.

Also, check out this great article by my friend Aly Sivji, Testing 101: Introduction to Testing.