Determine the scope and requirements for quality assurance activities
The development of a Quality Assurance (QA) plan is an essential practice for Information Technology (IT) project in order to maximize the effectiveness of the quality assurance efforts. The overall project QA plan and the related detail test plans define the project test strategies and implementation. The purpose of quality assurance planning is to establish the review and testing methodologies and resource requirements for the project.
Objectives
Determine the scope and requirements for quality assurance activities
Establish a managed test environment
Specify the team and customer staff, data and other resources required for QA
Establish measurable criteria for the completion of QA activities
Benefits
To the customer:
Increases reliability of the project deliverables
Lowers overall project and ongoing support costs
Ensures that business requirements are met by the system
Improves customer confidence in the process and product
To the project manager and team:
Reduces rework
Increases ability to deliver a superior product
Makes cost and schedule more predictable
Increases team confidence and their reputation
Documentation
The overall quality assurance plan is either included as an attachment to the project plan or is referenced by the project plan as a separate deliverable. It is based on the initial high-level test strategy that is typically described in the Requirements Definition Documentation (RDD). Subsystem and module test plans specified by the QA plan are developed during detailed design and are included in the QA portion of the project notebook.
Quality Assurance Planning Process
Quality assurance is a process that is necessary during the entire life cycle of the project. There is no point along the project that a requirement to search for errors and improvements is not a value-added activity.
The nature of error detection changes as the project and the software products head to completion. In the early phases of the project, Peer reviews of project deliverables such as the RDD, project plan and Appropriation Request (AR) are conducted. Red teams are the reviews conducted by peers outside of the project team or by customers. (Refer to the Peer Review and Approval process).
Inspections are conducted on design and programming deliverables. Inspections are, by definition, peer reviews conducted by members of the project team (Refer to the Software Inspections process). Inspections are typically conducted on detailed design documents, test plans, code and system documentation.
As code is completed, classic hands-on testing will be employed in place of the peer reviews. Tests of executable code are used during the rest of the project at the unit, integration, systems, and acceptance levels of testing.
The following table shows the quality assurance technique performed during each phase of the project life cycle and relates the typical types of error detection to the phases of the project.
Quality Assurance by Project Phase
Roles matrix for QA
The following is a view of the variety of responsibilities for the project team and customer in the area of quality assurance. Ensure that these responsibilities are in the project's roles matrix:
Project Team Norms
When establishing project team norms, determine the customer and project team expectations in the area of testing. Recommendations in the area of norms as they relate to quality assurance are:
All deliverables will be reviewed and tested (ensure that you define what "all" means).
No additional work will be performed on previously tested material without a change request or problem report task associated with it.
All rework will be tested.
Reviews and tests of other people's work will be performed thoroughly and in a timely manner. The work will be treated with a high priority.
Testing will be performed by an independent person or group whenever possible.
Any work delivered in an emergency mode will still be tested after delivery.
Quality Assurance Cost Considerations
The cost and time associated with the quality assurance processes can be considerable. Past experience indicates that the range for a reasonable QA program may be from 20% to 30% of the overall project costs. However, the biggest paybacks in the entire quality assurance cycle are obtained from the early red teams and inspections of requirements definitions, preliminary designs, and detailed designs. Therefore, an emphasis should be placed on the reviews, instead of an over reliance on traditional testing.
Although a firm percentage for QA and testing activities cannot be easily provided because of the many factors that impact costs and time on a project, the following guidelines should be considered:
A consistent test methodology should be specified to enhance estimating and task tracking.
The time invested in early inspections of requirements and design are the most cost effective in detecting and removing errors.
Inspections generally require 15% of the time during the implementation phase of the project.
Test Completion Criteria
One of the difficult areas to measure is the "completion" of a test. The determination of when enough reviewing and testing has occurred can be viewed in several different manners:
In all phases and for all tests, ensure that 100% of the business requirements are satisfied. This is probably the absolute minimum level of testing that should be attempted.
Conduct red teams and inspections of documents and deliverables early in the life cycle (e.g., requirements, project plans, and designs) until there is an agreement of closure by the inspectors (typically after one or two sessions). Use the inspectors' best judgment for the closure of these reviews.
Perform unit tests until all bugs stop appearing or if the customer accepts the bugs that remain as deferrable or acceptable as is.
Perform system and integration tests (as specified in the plans) until the number of errors discovered declines to an acceptable, pre-defined level.
Test until the estimated risk is at an acceptable level.
Test until the estimated value of additional testing becomes unacceptable.
QA Planning Approach
The first specification of the test strategy for a project is in the RDD. The RDD describes the strategy to be employed throughout the project and is approved by the customer via the approval of the RDD. Based on that approved strategy, the overall Quality Assurance plan for the project is developed in conjunction with the Project Plan. The QA plan is necessary in order to understand the requirements for staff, cost, and schedule in the QA area.
The overall QA approach on typical software development projects consists of ten components:
1. Peer all project documents in the OI, Analysis and Implementation Planning phases of the project
Peers and customers should be involved in the Peer review of the key documents in the first three phases of the project. Utilize the Peer Review and Approval process in the IT Project Methodology for all of these deliverables. The early red teams are performed before the QA Plan is created. They are specified as part of the overall IT Project Methodology and specifically in the project's Analysis Work Plan.
2. Build the Quality Assurance Plan
The Quality Assurance Plan for the entire project is created during the implementation planning phase of the project. It basically determines the optimal set of test and review approaches for the project.
The plan describes the scope, approach, resources, and schedule of review and testing activities. It identifies the items to be reviewed/tested, the features to be reviewed/tested, the tasks to be performed, the personnel responsible for each task, and the risks associated with the plan.
The project manager should ensure that the project plan incorporates key features of the QA Plan in the project plan. This includes resource specification, timing, and Environmental requirements.
As part of the QA Plan, the project manager and team should ensure that a process for transitioning testing into the ongoing maintenance mode is planned. In this process, the suite of test scripts, data and are packaged in a manner that is transferable to the ongoing support team.
Refer to the Quality Assurance Planning Outlines at the end of this process for the outline of the overall QA plan.
3. Develop project test documentation
Full test documentation should be developed to manage the execution of testing. The documentation should establish:
The specification for the test
How the test will be controlled
How the test will be communicated
The overall test specification is composed of three document types:
a. The test design specification refines the test approach outlined in the overall QA Plan. It is developed in conjunction with the design specifications. The test design specification identifies features to be covered in the test, the test cases and test procedures, and the feature pass/fail criteria for each design component.
b. The test case specification documents the actual values used for the test cases, along with the expected results. It is developed as part of the detailed design.
c. The test procedure specification identifies all steps required to exercise the associated test cases. It is also developed as part of the detailed design deliverable.
Another important dimension of test documentation is the area of test control and reporting. This consists of four documents:
a. Test item transmittal reports are necessary in test environments where the test group is independent of the development group. This report identifies the test items that are ready for testing.
b. Test logs are used by the test team to record the results of each test execution. It contains the test identifier, date of the test, a description of key activities and a cross-reference to the errors identified.
c. Test incident reports (e.g., problem reports) describe the events that occurred during the test that require further investigation. These items may be confirmed as bugs or dismissed for Environmental, data or other reasons.
d. Test summary reports summarize the results of tests for the team, management, and the customer.
4. Inspect all Implementation phase documents
The most cost effective technique to be employed in the QA process is the inspection of the key development documents (e.g., design documents, code listings and test scripts). Utilize the Software Inspections process in the IT Project Methodology for these deliverables.
5. Perform unit tests
The unit test focuses on the smallest testable unit (usually at the program level). This test should be performed by (or in close coordination with) the programmer prior to the code inspection for that program. The completion of the unit test is often seen as the entry criteria for the code inspection process.
6. Perform integration tests
Integration testing is the process of testing a group of programs or subsystems. This is performed after the conclusion of the unit tests and code inspections within the target subsystem and ensures that the subsystem is not in error.
7. Perform system tests
System testing is the process of testing the whole system with respect to its employment in operation. "System" testing has, in practice, been seen as a single test. However, a true system test is actually a suite of a number of the following test types. The choice to include test types is based on risk and value:
Requirements-based System Test Types
Functional testing is a process to find discrepancies between the program and its external specification.
Performance testing attempts to demonstrate that the system cannot meet its objectives for response time and throughput.
Parallel testing tries to demonstrate that the new system cannot match a target system in certain measurable criteria.
Size-based System Test Types
Volume testing subjects the system to heavy volumes of data. The purpose of volume testing is to show that the program cannot handle the volume of data specified in the requirements.
Stress testing subjects the system to heavy loads of stresses. The difference with volume testing is that the heavy load of stress is for a short period of time. Think of stress testing as more of a speedrelated test, rather than as an overall workload test.
Storage testing attempts to demonstrate that the storage objectives of the system have not been met.
User-based System Test Types
Usability testing attempts to find the human-factor, or usability, problems. Probes may include:
a. Are the interfaces suitable?
b. Are the outputs meaningful?
c. Are error messages useful?
d. Are there an excessive number of options?
e. Are acknowledgments provided to inputs?
f. Is the system easy to use?
Security testing attempts to break the security controls of the system.
Documentation testing attempts to demonstrate that the documentation is not representative of the system.
Procedure testing is a test of the procedures for the data base administrator, system operator, or other relevant personnel.
Other System Test Types
Configuration testing attempts to demonstrate that the system cannot run on the full range of expected hardware and operating system expectations.
Conversion testing focuses on showing that compatibility or that conversion objectives cannot be achieved.
Installability testing demonstrates that the system cannot be installed using the prescribed procedures and capabilities.
Recovery testing tries to demonstrate that the system cannot be recovered from data, hardware, or programming errors.
8. Perform regression tests on all revisions to code
Utilize regression tests (e.g., the repeat execution of a previously successful test) to ensure that changes made to previously tested modules are done correctly and that other errors have not been introduced along with the expected modification.
9. Conduct a thorough acceptance test
The last major test responsibility of the project test team is the facilitation of the customer acceptance test. The acceptance test compares the program to its original requirements and the user's current needs. The key points associated with the acceptance test include:
Establishing acceptance test criteria during the requirements definition phase
Ensuring that the customer is actively involved in the conduct and evaluation of the acceptance test
Establishing a clear control plan for the acceptance test
Obtaining formal acceptance of the system based on the test
Communicating the acceptance to all involved with the project
10. Turn over test and review materials and the processes used to the ongoing support team
The last step in the QA process is to ensure that all test and review related information has been successfully transitioned to the ongoing support team for the project. Key transition information includes prior test and review results, test plans, and test data.
Quality Assurance Planning Outlines
I. QA Plan
The high-level QA plan should be produced as a deliverable in the implementation planning phase of the project. Peer the document to ensure that it meets the requirements of the project and that it is a quality document. Place the document under configuration management control as an overall requirements change. The document must cover:
Overview
Introduction
Purpose of document
Approval needed
Review and Test Approach
Test Deliverables
Peer inspection report
Test design specifications
Test case specifications
Test item transmittal reports
Test logs
Test incident reports
Test summary reports
Test input data and test output data should be identified as deliverables
Test tools (for example, module drivers and stubs) may also be included
Review and Test Tasks
Identify:
Peer and inspection requirements
The set of tasks necessary to prepare for and perform testing
All inter-task dependencies and any special skills required
Environment Needs
Specify both the necessary and desired properties of the test environment. This specification should contain the physical characteristics of the facilities, including:
Hardware
Communications
System software
Mode of usage (e.g., stand-alone)
Other software or supplies needed to support the test
Level of security which must be provided for the test facilities, system software, and proprietary components such as software, data, and hardware
Special test tools needed. Identify any other testing needs (e.g., office space)
Source for all needs that are not currently available to the test group
Responsibilities
Identify the groups responsible for managing, preparing, executing, checking, and resolving the reviews, inspections and tests.
Staffing and Training Needs
Specify review and test staffing needs by skill level. Identify training options for providing necessary skills.
Schedule
Include review and test milestones identified in the project schedule, as well as all item transmittal events.
Define any additional test milestones needed. Estimate the time required to do each testing task. For each testing resource (that is, facilities, tools, and staff), specify its periods of use.
II. Test Design Specification
The test design specification identifies refinements of the QA Plan test approach and describes the features to be tested by this design and its associated tests.
Introduction
Summarize the software items and software features to be tested. The need for each item and its history may be included. Reference the following documents:
Project Plan
Quality Assurance Plan
Configuration Management Plan
Relevant standards or policies
Approach
Describe the overall approach to testing. For each major group of features or feature combinations, specify the approach that will ensure that these groups are adequately tested. Specify the major activities, techniques, and tools that are used to test the designated groups of features by the phase of testing.
The approach should be described in sufficient detail to permit identification of the major testing tasks and estimation of the time required to do each one.
Item Pass/Fail Criteria
Specify the criteria to be used to determine whether each test item has passed or failed.
Suspension Criteria and Resumption Requirements
Specify the criteria used to suspend all or a portion of the testing activity on the test items associated with this plan. Specify the testing activities that must be repeated, when testing is resumed.
Features to be Tested
Identify the test items and describe the features and combinations of features that are the object of this design specification. Other features may be exercised, but need not be identified.
For each feature or feature combination, a reference to its associated requirements in the design description should be included.
Test Items
Identify the test items, including their version/revision level. Supply references to the following item documentation, if it exists:
Requirements specification
Design specification
User's guide
III. Test Case Specification
The test case specification defines a test case as identified by a test design specification.
Test Items
Identify and briefly describe the items and features to be exercised by this test case.
Input Specifications
Specify each input required to execute the test case. Identify all appropriate data bases, files and values passed by the operating system.
Output Specifications
Specify all of the outputs and features (for example, response time) required of the test items. Provide the exact value (with tolerances where appropriate) for each required output or feature.
Environmental Needs
Hardware: Specify the characteristics and configurations of the hardware required to execute this test case.
Software: Specify the system and application software required to execute this test case. This may include system software such as operating systems, compilers, and test tools.
Other: Specify any other requirements such as unique facility needs or specially trained personnel.
Special Procedural Requirements
Describe any special constraints on the test procedures that execute this test case. These constraints may involve special set up, operator intervention, output procedures, or special wrap up.
Intercase Dependencies
List the identifiers of test cases that must be executed prior to this test case. Summarize the nature of the dependencies.
Procedure Steps
IV. Test Item Transmittal Report
The test item transmittal report is used to identify the test items being transmitted for testing. It includes the person responsible for each item, its physical location, and its status.
Transmitted Items
Identify the test items being transmitted, including their version/revision level. Supply references to the item documentation and the test plan relating to the transmitted items.
Location
Identify the location of the transmitted items. Identify the media that contain the items being transmitted.
Status
Describe the status of the test items being transmitted. Include deviations from the item documentation, from previous transmittals of these items, and from the test plan. List the incident reports that are expected to be resolved by the transmitted items. Indicate if there are pending modifications to item documentation that may affect the items listed in this transmittal report.
V. Test Log
The test log is used to provide a chRonological record of relevant details about the execution of tests.
For each event, including the beginning and ending of activities, record the occurrence date and time along with the identity of the author. The following activity and event entries information is to be documented:
Execution Description
Record the test procedure being executed and supply a reference to its specification.
Procedure Results
For each execution, record the results (e.g., error messages generated, aborts, and requests for operator action).
Environmental Information
Record any Environmental conditions specific to this entry (for example, hardware substitutions).
Anomalous Events
Record what happened before and after an unexpected event occurred (for example, a summary display was requested and the correct screen displayed, but response seemed unusually long. A repetition produced the same prolonged response).
Record circumstances surrounding the inability to begin execution of a test procedure or failure to complete a test procedure (for example, a power failure or system software problem).
Incident Report Identifiers
Record the identifier of each test incident report, whenever one is generated.
VI. Test Incident Report
The test incident report is used to document any event that occurs during the testing process that requires investigation. It is comparable to a problem report after system installation.
Summary
Summarize the incident. Identify the test items involved indicating their version/revision level. References to the appropriate test procedure specification, test case specification, and test log should be supplied.
Incident Description
Provide a description of the incident. This description should include the following items:
Inputs
Expected results
Actual results
Anomalies
Date and time
Procedure step
Environment
Attempts to repeat
Testers
Impact
If known, indicate what impact this incident will have on test plans, test design specification, test procedure specifications, or test case specifications.
VII. Test Summary Report
The test summary report is used to summarize the results of the designated testing activities and to provide evaluations based on these results.
Summary
Summarize the evaluation of the test items. Identify the items tested, indicating their version/revision level. Indicate the environment in which the testing activities took place.
For each test item, supply references to the following documents if they exist: test design specifications, test procedure specifications, test item transmittal reports, test logs, and test incident reports.
Report any variances of the test items from their design specifications. Indicate any variances from the test designs or test procedures. Specify the reason for each variance.
Evaluation
Provide an overall evaluation of each test item including its limitations. The evaluation must be based upon the test results and the item level pass/fail criteria. An estimate of failure risk may be included.
Summary of Activities
Summarize the major testing activities and events. Summarize resource consumption data; for example, total staffing level, total machine time, and total elapsed time used for each of the major testing activities.
See also:
For maintaining a copy of project test plans and results in the project notebook
For reviewing project requirements to develop testing needed and document in the test plan
For including or referencing the project software test plan with the project plan
For establishing and documenting software inspection requirements as part of the test plan
JB Danforth Company Proprietary