Test Process Measurements and Improvement

(CTM #3) or (CSTP Elective)

This tutorial covers area 3 of the Test Management Body of Knowledge (TMBOK) required for the Certified Test Manager (CTM) certification. This tutorial also covers the Elective area of the Certified Software Test Professional requirements.

Concepts

Testing effectiveness is determined by one's testing process—the way we do things that produces our results. To improve test results, we must improve our testing process; and that requires meaningful, valid, and reliable objective measurement. Defect data are the most obvious testing artifact to measure, but they must be put in context and coupled with other key factors that are critical to enabling improvement. Not only do we tend to miss important measures, but we also often overwhelm ourselves with too many measurements. This interactive presentation describes a minimum set of metrics that testing needs to know, ways to make those measures within appropriate contexts, and methods to analyze and report to guide improvement. Techniques are suggested for overcoming resistance when getting started. Exercises enhance learning by allowing participants to practice applying practical techniques to realistic examples. This course has been designed for test, project, and system managers, as well as for testers, analysts, designers, programmers, auditors, and users who are concerned about the efficiency and effectiveness of testing within the overall development process.

Learning Objectives

• Relation of measurement to processes and distinguishing real from presumed processes.
• Key variable dimensions that a measurement set needs to address.
• Relating testing project measures to assessing, managing, and improving the testing process.
• Ways to analyze and evaluate effectiveness of testing practices.
• Presenting and reporting measurements so people pay attention.

Outline

• Measurement and Processes
  • What a process is, and is not, why it matters
  • Distinguishing “real” from “presumed”
  • Defined and documented processes
  • Relation between process and measurement
  • Silos, measuring a process to full end result
  • Measuring results vs. guiding improvement
  • Non-procedural process components
  • Key measurement principles, core measures
  • Validity, reliability, materiality, Hawthorne
  • Project and product vs. process measures
  • Measures vs. metrics, meters and predictors
  • Goal-question-metric (GQM), process goals
  • Why most measurement programs fail
  • Avoiding resistance to measurement
• Defining the Testing Process
  • Test execution time, effort, quality measures
  • Test automation, support, administration
  • Test cycles, defect fix turnaround time
  • Impact of level of quality required
• Outcomes, Analysis and Reporting
  • What is a defect; fault vs. failure, duplicates
  • Operations and performance measures
  • Reliability, durability, usability
  • Incident/defect categorization, tracking
  • Defects found vs. number projected
  • Defect age, source including testing errors
  • Charting over time, projecting delivery date
  • Defect patterns, causal analysis
  • Test sensitivity, detection rates and yields
  • Find and fix time, effort, cost by phase
  • Defect detection efficiency
  • Historical profiles, subjective view of value
  • Evaluating effectiveness of test techniques
  • Reporting, using measures to improve
  • Key testing process procedural elements
  • Proactive Testing™ Life Cycle
  • IEEE Standards for testing, documentation
  • What should be measured, how to do it
  • Testing/development good metrics practices
  • Statistical process control, capability
  • Externally-defined software process models
  • Test maturity, CMM-based vs. TPI models
  • Test effectiveness indicators
  • Benchmarks, surveys, assessments adequacy
• Testing Activity Measures
  • What is a test case, basic unit of size
  • Development sizing, derived test sizing
  • Test case coverage measures, limitations
  • Functional test coverage, traceability matrix
  • Degrees of structural, white box coverage
  • Dynamic vs. active, passive static testing

Biography

Robin Goldsmith JD is internationally recognized as an authority on business engineering and software acquisition/development quality, testing, and productivity. He is a frequent speaker at leading conferences and formerly International Vice President of the Association for Systems Management. Robin is the author of the book:" Discovering REAL Business Requirements for Software Project Success"