Acquiring a Software System: What Your Quality Assurance Program Should Be

Many organizations are in the processes of acquiring new major software systems, either to support new functions or replace aging systems. When acquiring significant software applications, organizations need to provide their own quality assurance program, even though the vendor is likely to have its own. Why is this program important and how should it be conducted?

Importance of Quality Assurance

In Software Project Survival Guide, software expert Steve McConnell writes:

Even if your software doesn't have to be ultra reliable, keeping defects under control matters because it affects development speed, development cost, and other project characteristics. …Defects cost 50 to 200 times as much to correct if they are detected and corrected downstream instead of upstream. That should be enough reason to focus on quality, but other bottom‐line impacts exist as well.

Poor quality is a major cause of project failure. An inability to detect and correct defects early results in a hidden backlog of rework that can prove a schedule buster. Defects in production software cost the economy billions of dollars in down‐time and remediation.

Even though the vendor conducts quality assurance, the acquirer needs its own QA program for several reasons. First, QA is the eyes and ears on the project for the project manager, IT management, and senior management. When conducted properly, QA identifies “hot spots,” deliverables or processes where the level of defects is unusually high and need management attention and remediation. Second, most software acquisition projects have aspects that are the responsibility of the acquirer. Examples are writing of requirements, building interfaces between the new system and existing systems, and user acceptance testing. The acquirer needs to take responsibility for the quality of these aspects. Third, a good quality assurance program has a measure of independence from the developers. This independence allows the QA team to assess deliverables and report results to project stakeholders without pressures to avoid “bad news.”

The acquirer’s QA program need not, and should not, be adversarial with the vendor. A knowledgeable vendor will welcome such a program. Vendors have enormous incentives to have successful implementations. They will welcome efforts that improve the chance of success. The vendor’s QA department should work cooperatively with the client’s QA program, sharing test cases, techniques, and tools. Vendors also recognize that removing defects from requirements and other up‐front deliverables saves substantial money and effort later in the project.

An Approach to Quality Assurance

Most application acquirers conduct user acceptance testing. However, the QA approach needs to be more comprehensive. It needs to include planning, test case analysis, inspections, manual and automated testing, and QA metrics.

QA Planning

Quality assurance planning needs to start when project planning starts. QA activities should represent approximately 25% to 50% of total project effort. Therefore, planning the resources for QA is a significant portion of the overall resource planning for the project. A mistake some organizations make is to fail to budget for quality assurance activities. This mistake makes it difficult to adequately staff the QA activities once the need is recognized.

Another mistake is to start planning QA late in the project. Some organizations feel that since testing starts after the system is coded, planning is only needed just before testing is scheduled. But, QA is more than just testing. Activities like inspections and test case writing need to start from the beginning of the project. Also, testing should occur throughout the project, not just at the end. This way, defects and “hot spots” are identified early in the project, when corrective action is most effective. Late planning allows insufficient time to write test cases, implement automated testing, and other important QA tasks.

Inspections

Inspections of deliverables have proven to be highly effective in reducing defects and should be part of any QA program. Requirements, specifications, and test cases are all good candidates for inspections. One reason that inspections are effective is because they detect defects early in the project when the cost of remediation is lower than late in the project.

Test Case Writing

Effective testing requires written test cases to help insure that testing covers requirements. The QA team needs a methodology for writing test cases. A good methodology is Framework for Integrated Testing (FIT), developed by Ward Cunningham. There are books and articles that teach the FIT methodology. There are open source software tools that support FIT. FIT test cases are expressed as a set of tables involving inputs and outputs. FIT test cases are written by business analysts or other business experts based on business requirements. Developers write code called fixtures to manipulate the system being tested. A FIT library is a code framework that controls the execution of the tests and provides reporting. FIT provides an easily learned methodology for preparing test cases that reduces the amount of effort required for this activity. The QA team can prepare test cases and commence testing faster with less effort than with traditional test case development.

Test Execution and Automated Testing

The testing approach should emphasize automated testing. With iterative development practices, like Extreme Programming, becoming more common, automated testing becomes increasingly important. Manual testing through the user interface becomes more expensive and less effective. Since code changes are more frequent than previously, regression testing has to be done more often. With manual testing, either regression testing is undercut or regression testing costs increase. Automated testing converts perishable manual effort into a reusable asset. Several open‐source tools are available for automated testing, including testing through the user interface, testing through programming interfaces, and load testing. Use of open‐source tools reduces costs and eases introduction of these tools into the client organization.

Metrics

Providing the project team and senior management with quality metrics is an important contributor to project success. Timely metrics allow tracking progress in quality assurance activities and early identification of “hot” spots where quality is deficient and team corrective action is required. Test manager and defect tracking software help to provide quality information and metrics to all project stakeholders while reducing the effort in assembling metrics.

User Acceptance Testing

User acceptance testing is performed as a final confirmation that the system is acceptable. User acceptance testing should usually be performed by actual users of the system. This type of testing should focus on end‐to‐end functionality, ensuring that the system performs each business process in a way that is acceptable to the end users. User acceptance testing requires written test cases just as other testing does. In a well‐run QA program, there should be no surprises in user acceptance testing.

However, exposing the user to the system should occur as early in the project as possible, not just at the end of the project. Some QA programs use conference room pilots to conduct testing with end users early in the project. In a conference room pilot, end users execute test cases with early versions of the system. The purpose is not to provide extensive test coverage, but rather to expose users to the system early in the project and to obtain their feedback.

Summary

Acquisition of major systems requires an effective quality assurance program. An effective program involves a number of activities, not just testing. Inspections and early testing have great leverage, because they detect problems early in the project when defects are the cheapest to fix. Any organization contemplating the acquisition of a major system needs to plan for the quality assurance program.

Dr. William Shaffer
President, Waysys LLC
wshaffer@waysysweb.com

« Back to Front Page