Tutorials: Series T
Tuesday, March 5, 2002

Two concurrent tutorials taught by nationally recognized quality experts. Each tutorial is a one full day of an in-depth instruction in a specific software quality topic.

Or PSQT/PSTT Conference Sessions

   Series M - Monday, March 4, 2002

   Series W - Wednesday, March 6, 2002

   Series H - Thursday, March 7, 2002

   Series F - Friday, March 8, 2002   

Tutorial #T1 (Tuesday 8:30 - 4:30 p.m.)

This tutorial counts as one day of training towards the Certified Software Test Professional requirements.

This tutorial outlines the Automated Test Life-cycle Methodology, a structured process for designing, developing, executing and managing testing that parallels the System Development Life-cycle. It is based on the book titled "Automated Software Testing" co-authored by the instructor and published by AWL, ISBN 0-201-43287-0.

Automated Testing Life-Cycle Methodology

How test teams introduce an automated software test tool on a new project is nearly as important as the selection of the most appropriate test tool for the project. A tool is only as good as the process being used to implement the tool.

Over the last several years test teams have largely implemented automated testing tools on projects, without having a process or strategy in place describing in detail the steps involved in using the test tool productively. This approach commonly results in the development of test scripts that are not reusable, meaning that the test script serves a single test string but cannot be applied to a subsequent release of the software application. In the case of incremental software builds and as a result of software changes, these test scripts need to be recreated repeatedly and must be adjusted multiple times to accommodate minor software changes. This approach increases the testing effort and brings subsequent schedule increases and cost overruns.

The fallout from a bad experience with a test tool on a project can have a ripple effect throughout an organization. The experience may tarnish the reputation of the test group. Confidence in the tool by product and project managers may have been shaken to the point where the test team may have difficulty obtaining approval for use of a test tool on future efforts. Likewise, when budget pressures materialize, planned expenditures for test tool licenses and related tool support may be scratched.

By developing and following a strategy for rolling out and implement an automated test tool as part of the Automated Testing Life-cycle methodology, the test team can avoid having to make major unplanned adjustments throughout the test process. The tutorial "Automated Software Testing" addresses these various issues and their solutions.

The ATLM describes how and where "Automated Software Testing" fits into the system development life-cycle.

This tutorial will cover the following topics:

1. Overcoming False Expectations for Automated Test. This part of the tutorial will address the false expectations that automated testing will result in :
• Automatic Test Plan Generation
• Test Tool Fits All
• Immediate Test Effort reduction
• Immediate Schedule Reduction
• 100% Test Coverage
2. Outline Benefits of Automated test. This part of the tutorial will address the actual benefits of automated testing:
   • Production of a reliable System
   • Improvement of the quality of effort
   • Reduction of Test Effort and Minimization of Schedule
   • Some tests cannot be executed unless they are automated
     
3. Acquiring Management Support.  This part of the tutorial will address how to best acquire management support (i.e writing a proposal, content of the proposal, how to convince management, etc.)
   
4. Acquiring Tools. This part of the tutorial will address how to acquire an automated tool.
   • Review System Engineering Environment
   • Review Tools Available on the market
   • Tool Research and Evaluation
   • Reviewing the Test Life-Cycle Tools
   • Tool Purchase
     
5. Automated Testing Introduction Process. This part of the tutorial will address how to best introduce automated testing to a new project team, including:
   • Test Process Analysis
   • Process Overview
   • Goals and Objectives of Testing
   • Test Strategies
   • Test Tool Consideration
   • Test tool compatibility check
   • Review of Training requirements
   • Test Team Recruiting and Management
     
6. Test Planning, Design and Development. This part of the tutorial will address how to best address test planning, design and development of tests in an automated testing environment, to include:
   • Test plan documentation
   • Test Program Scope
   • Test Requirement Management
   • Test Environment
   • Test requirements Analysis
   • Development-Level Test Analysis
   • System-Level Test Analysis
   • Test Program Design
     • Review Test program design modules
     • White-Box Techniques ( Development-Level Tests)
     • Black-Box Techniques ( System-Level Tests)
     • Test Design Documentation – examples of Test Design using Orthogonal Array Testing Techniques.
     • Test procedure definition
     • Automated vs. Manual Test Analysis – what to automate and what not to
     • Automated Test design standards
   • Test Development
     • Set up Test Environment
     • Automation Framework Reuse Analysis
     • Test Procedure Development/Execution Schedule
     • Calibration of the test tool
     • Compatibility and work around solutions
     • Manual Execution of Test Procedures
     • Test Procedure inspections and Peer reviews
     • Test Procedure Configuration management
     • Reusable Test procedures
       
7. Automated Test Execution and Management of it. This part of the tutorial will address how to best address automated test execution and the management of the testing execution cycle, to include:
   • Unit Test Execution and Evaluation
   • Integration Test Execution and Evaluation
   • System Test Execution and Evaluation
   • Test Analysis of Regression Testing
   • User Acceptance Test Execution and Evaluation
   • Testbed baseline
   • Creating Baseline data
   • Tools to Generate Data
     • Defect Tracking
     • Identify the priority of a defect
     • Assign a unique identifier to each defect
     • Link each defect to test procedure/build
     • Log all the appropriate dates for tracking
     • Create metrics for Defect Management
     • Test progress Tracking
     • Test Metrics
       
8. Post release. Test Process Improvement – This part of the tutorial will address the post release and evaluation period and how to improve the test process, to include:
   • Documenting Lessons learned
   • What worked and what did not?
   • How would you do things differently?
   • Reviewing standards for future projects

Various companies throughout the world have adopted the ATLM and incorporated it into their project.

Learning Objective

• Best practices for test automation
• A Case Study will be presented that covers how the ATLM was implemented on one particular project. This case study will address each phases of the Automated Testing Life-cycle. Students can bring their own project specific problems, which can be addressed during the tutorial.
• Acquiring management support
• Test tool evaluation and selection
• The automated testing introduction process
• Various tools used during the various life-cycle phases (i.e. Rational Rose during Architecture phase, ReqPro or DOORS during Requirements Management, Robot or Winrunner during Test Execution Phase, Performance Studio or E-Suite for performance testing)
• Automated and manual test planning and preparation
• Test procedure development guidelines
• Automation reuse analysis and reuse library

John Paul is the founder and President of Minjoh Technology Solutions,Inc., a company dedicated to providing quality Information Technology Solutions to government and commercial agencies. He has a BS degree in Computer Science and 12 years of professional software development experience. His experience extends through all phases of software development life cycle. He has been a speaker/presenter at various professional seminars and conferences.

Top of Page Tutorials: Series T

Tutorial #T2 (Tuesday 8:30 - 4:30 p.m.)

Ed Weller

This tutorial counts as one day of training towards the Certified Software Test Professional requirements.

Introductions and Tutorial Objectives:

·        Identify the major concerns with testing

·        Examine test metrics that help us do a better job of managing test

·        Keep a “real world” perspective - What’s useful, and more importantly, what’s useable?

Setting the Stage

·        What are your biggest problems with test?

·        How can Measurement help?

·        What should we measure?

·        Definitions

Test Metrics

·        Effort Metrics

·        Testware Metrics

·        Test Execution Metrics

·        Test Evaluation Metrics

·        Evaluation and Planning Metrics

·        Product Metrics

Effort

·        Definitions

·        Who provides it

·        How it is used

·        Test planning and tracking

Testware

·        Test development metrics

·        Test Objective Inventory

·         Combining Effort and Task Measures

Execution

·        Who provides it

·        What should be collected

·         How is it used?

• How good are the tests
• Problem Priority/Severity
• Regression Testing
• Cautions

Evaluation

·        Effectiveness

·         Efficiency

·        How are these used?

• The “defective fix” problem
• How good is the product
• Cautions

We Can’t Do Everything, So What’s First?

·        Inventory

·        Plan, Check, Do, and Act

·        Grow Incrementally

Summary and Wrap-up

·        Where are you?

·        Keep it Simple!

·         Where are you going?

·         Communicate!

Biography:

Ed Weller has over 30 years of experience in hardware, test, software, systems, and software process engineering, with software process his primary area of focus for the last 12 years. Prior to joining STT full time, he was the senior technical leader for division level software process improvement programs at Bull Information Systems and Motorola’s Satellite Communications Division. In this role he has developed and implemented long-range process improvement programs with demonstrated success. In 1992, Bull achieved SEI Level 2 and ISO 9001 certification under his leadership. He is a certified Lead Assessor in the SEI CMM-Based Appraisal for Internal Process Improvement and instructor of the “introduction to the CMM”. He has participated in numerous assessments and software supplier process evaluations based on the SEI Appraisal model. 

Ed was a co-founder and first co-chair of the Software Inspection and Review Organization, a special interest group promoting the use of inspection process. He was a member of the SEI's Software Measurements Steering Committee and the DoD Software Best Practices Initiative Steering Committee. He is a member of the Embry Riddle Aeronautical University Industry Advisory Board, and was the Program Chair of the 7^th International Conference on the Applications of Software Measurement in 1996, 1999, 2000, and 2001, and 2002. 

His September 1993 IEEE Software article was awarded best article of the year honors by the IEEE Software Editorial Board, and his presentation "Using Metrics to Manage Software Projects" was selected as the best presentation at the 1994 Applications of Software Measurement Conference. An article by the same title was published in the 1994 IEEE Computer. He recently published “Practical Applications of Statistical Process Control” in the May/June 2000 IEEE Software, and “Applying Quantitative Methods to Software Maintenance” in the December 2000 Software Quality Professional. He was the author of the Software Metrics column in the Software QA Quarterly (now Software Quality and Test Engineering) and has presented seminars at several major software companies on the inspection process. 

Ed received his BSE in Electrical Engineering from the University of Michigan, and MSEE from the Florida Institute of Technology.  Ed is a Senior Member of the IEEE.

Top of Page Tutorials: Series T