“In God, we trust. In all else, we test.” (Unknown)
What is Software Testing?
Software testing is the practice of:
- validating goodness and identifying badness
- in software product code and features
- for the purpose of enforcing high quality standards
There are 3 fundamental categories of software testing:
- Functional: Does the software work correctly?
- Performance: Does the software work within desired system metrics?
- Experimental: Does the software work with improved feature metrics after a change?
Functional testing determines if software features work correctly. They yield a deterministic pass/fail result.
“Correctness” means mutually-agreed goodness. It may be determined by:
- test cases and plans
- domain knowledge
- user expectations
- common sense
Modes and Methods
There are two modes of functional testing:
- Scripted: Test cases are written, reviewed, and then run according to plan.
- Exploratory: Experts interact with product features without a script in attempts to uncover issues.
There are also two methods for running functional tests:
- Automated: Test software runs tests automatically without manual intervention.
- Best at being defensive to protect software against regressions.
- A good fit for scripted tests.
- Manual: A human tester exercises system behavior through direct interaction.
- Best at being offensive to find bugs in new features.
- A good fit for exploratory tests and for scripted tests that are difficult to automate.
The methods and modes are complementary: one does not supplant another.
Levels and Layers
There are two access levels for functional testing:
- White-box: Tests interact directly with product code, thus covering code.
- Black-box: Tests interact with a live instance of the built product, thus covering features.
There are also three layers for functional testing:
- Unit: Very short white-box tests for individual “units” (functions, methods, or classes) of code.
- Integration: Black-box tests that cover where two components come together (often service layer).
- End-to-End: Lengthier black-box tests that cover an execution path through a system (often Web UI layer).
These three layers form the Testing Pyramid.
Test automation is indispensable for software development success. Tests should be automated when they give positive returns-on-investment.
Functional test frameworks and related packages are available in all major programming languages. Don’t reinvent wheels.
- Each test should focus on one main behavior or variation.
- Tests should run independently of each other. (They should be runnable in any order.)
- Tests should cover behaviors as close to the point of origin as possible.
- Tests should be self-descriptive and intuitively understandable.
- Count(unit) > Count(integration) > Count(end-to-end)
- Target near-100% code coverage for unit tests.
- Do not automate every test – use a risk-based strategy with ROI.
- Try to run automated tests in parallel, especially in CI/CD.
Looking for free courses on test automation? Check out Test Automation University!
Performance testing determines if a software product works well. As a precondition, features must work correctly (meaning, functional tests pass).
Performance testing should not use functional testing frameworks. They should use performance-specific tools like JMeter or Visual Studio load testing.
“Good” performance means minimal impact to the 4 primary software performance metrics in the system under test:
- Processor Usage
- Memory Usage
- Response Time
- 5 Things I Love About SpecFlow
- 10 Gotchas for Automation Code Reviews
- 10 Things You Lose Without Automation
- 12 Traits of Highly Effective Tests
- Arrange-Act-Assert: A Pattern for Writing Good Tests
- Behavior-Driven Blasphemy
- Clicking Web Elements with Selenium WebDriver
- Cypress.io and the Future of Web Testing
- Democratizing the Screenplay Pattern
- EGAD! How Do We Start Writing (Better) Tests?
- Gotta Catch ’em All!
- Introducing Boa Constrictor: The .NET Screenplay Pattern
- Mentoring Software Testers
- Our Test Automation Has Problems. Should We Start Over?
- Passing test inputs into pytest
- Please Hang Up and Dial Again: Handling Test Interruptions in CI/CD
- Quality Metrics 101: The Good, The Bad, and The Ugly
- Should We Rewrite Our Test Automation in Another Language?
- Software Testing Lessons from Luigi’s Mansion
- Test Automation Myth-Busting
- The Best Programming Language for Test Automation
- The Software Engineer in Test
- The Testing Pyramid
- To Infinity and Beyond: A Guide to Parallel Testing
- Tutorial: Web Testing Made Easy with Python
- Unpredictable Test Data
- Using Multiple Test Frameworks Simultaneously
- Web Element Locators for Test Automation
- WebDriver Element Existence vs. Appearance
- Why is Automation Full of Duplicate Code?
- Why Python is Great for Test Automation
- Writing Good Bug Reports
I love to develop test automation in Python. Check out the Python page for the Python Testing 101 series.
Testing is a big part of behavior-driven development. Check out the BDD page for more information.
Also, check out this great article by my friend Aly Sivji, Testing 101: Introduction to Testing.