TestWorks Quality Index: Table I -- Software Quality Filters

TestWorks Quality Index ^TM
Table I -- Software Quality Filters

TestWorks Quality Index `(TOP)`	Assess the quality of your quality process with the TestWorks Quality Index.
Table I -- Software Quality Process Filters	Assesses relative advantages and disadvantages of various software quality methods.
Table II -- TestWorks Quality Index Definitions	Summarizes the definitions of the factors that make up the TestWorks Quality Index and shows how to compute it for YOUR process.
Table III -- Product Application Profile	Gives a recommended composition of use of TestWorks products and indicates likely CMM-like levels and relative overall process efficiencies.

Basic Explanation

This table shows a range of software quality filters (test methods) and how they are used, applied, respected, and managed. It shows where a particular filter is used in the software life cycle and indicates some of the the relative "costs" for particular management choices.

Selection factors in the table below are expressed in relative terms. The possible scores are: Very-Low, Low, Moderate, High, Very-High.

Very few organizations apply all of these quality filters, even the highest-criticality types of software development efforts. However, everyone wants the results to be as if they did. The best way to use this information is to gain a management level understanding of exactly when to use what method and why.

Definition of Quality Filters

The table below describes the effectiveness of a broad range of software quality filters. Obviously, no single filter finds all the latent defects in a product, so you have to consider the fact that you will probably need to use several of these methods.

Requirements Based Test Planning: Often a manual, but possibly automated, this process distills requirements from specifications (formal or informal) into test plans that guide functional test process steps. Use Case Editors or FSM descriptions can be used with good effect.

Code Inspection/Code Review: A manual process, often supported with detailed process documentation, forms, and management structures, that aims to find defects by carefully reading/studying code. Inspections and Reviews differ by the level of formality and the type of management used.

Static Analysis: Programs called static analyzers examine source code and suggest problems or find defects automatically.

Metric Analysis: Metric analyzers examine source code and assign various software science metrics (Halstead volume, effort, cyclomatic complexity, etc.) to each function to help managers decide where best to apply available resources.

Leak Detection: Find instances where, during actual executions, an application allocates memory but does not return it. Can only find a "leak" if a passage is actually executed.

Leak Correction: Identify instances of memory leaks when they occur and fix them on the spot. May be coupled with automatic identification of offending source passages.

Bounds Checking: Perform a range of runtime checks, including leak detection, array bounds checking, value range checking, and report all violations found.

Unit Testing (Testbed w/Stubs): Create a local test driver with stubs and use it to thoroughly test out a function or a small set of functions BEFORE putting them into a bigger application. Normally a coverage analyzer is used to assess completeness.

GUI Capture/Playback Testing: Individual tests (part of a regression suite, see below) are based on recorded and/or edited scripts that drive an application's GUI.

Non-GUI Capture/Playback Testing: Individual tests (part of a regression suite, see below) are based on recorded and/or edited scripts that drive a non-GUI testbed, e.g. and xterm or a special-purpose test driver.

Object-Mode Browser-Based Testing: Tests recorded and played from WWW browser used as testbed. Object-mode necessary for easy of use and stability of testsuites.

Regression Testing: Create a set of tests, usually using a capture/playback system (see above), that can confirm that today's performance is the same as yesterday's performance, or show where it is different.

White Box Coverage Analysis: By instrumenting the source code, report test coverage metrics for individual tests and for groups or suites of tests. Coverage measures used range from statement coverage (weak) to path coverage (very strong).

Remote Beta Testing: Use instrumentation and other methods to provide feedback from selected early-release sites (beta sites) on levels of usage, errors encountered, and coverage achieved.

Load/Stress Testing: Find out how many users an application can tolerate by simulating N users, and/or study how a system's performance degrades under gradually increasing loads.

Capacity/Tuning Analysis: Carefully analyze the internals of a application and improve overall performance with a variety of techniques, including "hotspot" identification and elimination. The goal is application wide performance increase.

Defect Tracking: Keep precise track of reported problems, carefully categorizing their severity and complexity. Use this data as a feedback loop into the internal test process.

Definition of Effectiveness Factors

The factors in the table below are intended to characterize a particular software quality filter independently of any other quality filters. That is, each description assumes that ONLY that particular filter is being applied. However, all estimates are relative to the other estimates in the table.

Life Cycle Stage: The stage in the conventional waterfall life cycle model at which one would normally apply this method. Even if you're not using a waterfall model, this should give you an idea of what role is intended for the method.
Cost to Buy: The cost to buy the products and associated software to support doing this work.
Cost to Apply: The cost an organization would incur when applying this particular quality filter. A high value here means that a lot of organizational effort would be needed to properly use this approach.
How Hard to Manage: How difficult is the technical management of this work? Will engineers like to do the work or will they resist until proper structures are in place?
Payoff: Defects Found: Obviously the method will find defects, but how important are they and how often are the really important ones found. A "high" score here means that really "good", i.e. subtle or critical, defects are discovered.
Payoff: Product Confidence: Even if you don't find significantly many new serious defects, you still gain confidence in the product because you didn't find defects: you know that you didn't break something!
How Complete: What level of confidence is there, in either hard or soft evidence from field application of the method, that it will find ALL of the available defects.
Field Utilization Achieved:: How widespread is the method in the field. Here "high" means that the method is in relatively common use.
COMMENTS: Additional comments, warnings, and caveats.

Go Back To Top Of Page

Table I -- Software Quality Filters -- Summary Table
FILTER
NAME Life
Cycle
Stage Cost
to
Buy Cost
to
Apply How
Hard
to
Manage Payoff: Defects
Found Payoff: Product
Confidence How
Complete Field
Utilization
Achieved COMMENTS

Requirements Testing Design &
Develop Moderate Low Moderate High High Moderate Beginning Use Case Editor or FSM specification plus automated test case creation.
Code
Inspection Design &
Develop Lowest Moderate Moderate High Low Low Low Human centric, seen by most as an annoyance.
Code
Review Design &
Develop Lowest Low Moderate High Low Low Low More informal, viewed as less intrusive.
Static
Analysis Develop
[Static] Moderate High Very High Moderate Mode Moderate Lowest Too many non-critical errors found.
Metric
Analysis Develop
[Static] Moderate Moderate Moderate Low Low Low Low Controversy on meaning of the metrics.
Leak
Detection Develop
[Dynamic] Moderate Low Low Low High Moderate Highest? Leaks are inoffensive; ineffective method without high coverage.
Leak
Correction Develop
[Dynamic] Moderate Low Low Low High High Low Leaks are inoffensive; ineffective without high coverage.
Bounds
Checking
Develop
[Dynamic] Low Moderate Low Moderate High High Moderate Good dynamic checks with high coverage.
Unit
Testing
(Testbed w/Stubs) Develop
[Dynamic] Moderate Moderate Low Moderate Moderate Moderate Moderate Very few tools now available.
GUI
Capture/Playback
(BlackBox) Develop
QA/Test Moderate Moderate Moderate Moderate High Moderate High Object mode may simplify testing.
Non-GUI
Capture/Playback Develop
QA/Test Moderate High High Moderate Moderate High Low Local testware can be expensive.
Object-Mode
Browser-Based
Testing Develop
QA/Test Moderate Low Moderate Moderate High Moderate Very Low Requires special technology for full utilization.
Regression
Testing
(BlackBox) Develop
QA/Test Moderate Moderate High Moderate Moderate Low Moderate Very sensitive to product changes.
Minimal White Box Coverage (C0/C1?/S0) Develop
Integrate
QA/Test Free or Low Moderate Low Moderate Low Low Moderate Better than nothing at all.
Basic
White Box
Segment
Coverage
(C1 [S1]) Develop
Integrate
QA/Test Moderate Moderate Low Moderate Moderate Moderate Lower Very good method.
Basic
White Box
CallPair
Coverage
(S1 [C1]) Integrate
QA/Test Moderate Moderate Low Moderate Moderate Moderate Lower Very good method.
Intermediate
White Box
Dual Mode
Coverage (C1/S1) Develop
QA/Test Moderate High Moderate High High High Lowest Lots of work to do well.
Advanced White Box
Path Coverage (Ct) QA/Test Moderate High Moderate High Very High Very High Very Rare Almost like proof of correctness.
Remote
Beta
Testing Field Check
(Beta Test) Moderate Moderate Low High High Moderate Low New technology but gaining acceptance.
Load
Generation
Stress
Testing QA/Test High High Moderate Moderate Moderate Moderate Low Very specific experiment goals.
Capacity/Tuning Analysis QA/Test High High Moderate Moderate Moderate High Moderate General analysis and "hotspot" reduction.
Defect
Tracking All Stages Low to Moderate Moderate Moderate High Moderate Moderate Moderate You must *DO* it!

Table I -- Software Quality Filters -- Summary Table
FILTER NAME	Life Cycle Stage	Cost to Buy	Cost to Apply	How Hard to Manage	Payoff: Defects Found	Payoff: Product Confidence	How Complete	Field Utilization Achieved	COMMENTS
Requirements Testing	Design & Develop	Moderate	Low	Moderate	High	High	Moderate	Beginning	Use Case Editor or FSM specification plus automated test case creation.
Code Inspection	Design & Develop	Lowest	Moderate	Moderate	High	Low	Low	Low	Human centric, seen by most as an annoyance.
Code Review	Design & Develop	Lowest	Low	Moderate	High	Low	Low	Low	More informal, viewed as less intrusive.
Static Analysis	Develop [Static]	Moderate	High	Very High	Moderate	Mode	Moderate	Lowest	Too many non-critical errors found.
Metric Analysis	Develop [Static]	Moderate	Moderate	Moderate	Low	Low	Low	Low	Controversy on meaning of the metrics.
Leak Detection	Develop [Dynamic]	Moderate	Low	Low	Low	High	Moderate	Highest?	Leaks are inoffensive; ineffective method without high coverage.
Leak Correction	Develop [Dynamic]	Moderate	Low	Low	Low	High	High	Low	Leaks are inoffensive; ineffective without high coverage.
Bounds Checking	Develop [Dynamic]	Low	Moderate	Low	Moderate	High	High	Moderate	Good dynamic checks with high coverage.
Unit Testing (Testbed w/Stubs)	Develop [Dynamic]	Moderate	Moderate	Low	Moderate	Moderate	Moderate	Moderate	Very few tools now available.
GUI Capture/Playback (BlackBox)	Develop QA/Test	Moderate	Moderate	Moderate	Moderate	High	Moderate	High	Object mode may simplify testing.
Non-GUI Capture/Playback	Develop QA/Test	Moderate	High	High	Moderate	Moderate	High	Low	Local testware can be expensive.
Object-Mode Browser-Based Testing	Develop QA/Test	Moderate	Low	Moderate	Moderate	High	Moderate	Very Low	Requires special technology for full utilization.
Regression Testing (BlackBox)	Develop QA/Test	Moderate	Moderate	High	Moderate	Moderate	Low	Moderate	Very sensitive to product changes.
Minimal White Box Coverage (C0/C1?/S0)	Develop Integrate QA/Test	Free or Low	Moderate	Low	Moderate	Low	Low	Moderate	Better than nothing at all.
Basic White Box Segment Coverage (C1 [S1])	Develop Integrate QA/Test	Moderate	Moderate	Low	Moderate	Moderate	Moderate	Lower	Very good method.
Basic White Box CallPair Coverage (S1 [C1])	Integrate QA/Test	Moderate	Moderate	Low	Moderate	Moderate	Moderate	Lower	Very good method.
Intermediate White Box Dual Mode Coverage (C1/S1)	Develop QA/Test	Moderate	High	Moderate	High	High	High	Lowest	Lots of work to do well.
Advanced White Box Path Coverage (Ct)	QA/Test	Moderate	High	Moderate	High	Very High	Very High	Very Rare	Almost like proof of correctness.
Remote Beta Testing	Field Check (Beta Test)	Moderate	Moderate	Low	High	High	Moderate	Low	New technology but gaining acceptance.
Load Generation Stress Testing	QA/Test	High	High	Moderate	Moderate	Moderate	Moderate	Low	Very specific experiment goals.
Capacity/Tuning Analysis	QA/Test	High	High	Moderate	Moderate	Moderate	High	Moderate	General analysis and "hotspot" reduction.
Defect Tracking	All Stages	Low to Moderate	Moderate	Moderate	High	Moderate	Moderate	Moderate	You must DO it!