A key point in any development process, therefore, is the point at which testing ends and the organization moves ahead with deployment. This is the time in the life of every development project when the program manager has to ask the very practical questions:

• "Is it wise to continue spending money on quality assurance?"
  

• "Will further testing cost more than it is worth?"
  

• "Is the software ready for release?"
  

• "How do we know when we’re done testing?"

This white paper provides an innovative framework for answering those questions — a new barometer for measuring the business risk of release versus the cost of continued testing.

Upon initial examination, the question of when to stop testing appears to be solely technical. Organizations commonly apply exit criteria for the testing process based on factors such as the percentage of successful tests for completeness of functionality and the number of defects remaining at various severity levels. Some quality assurance teams have also measured quality using metrics in areas such as:

• Defect density and glide path.
  

• Requirements volatility.
  

• Code churn.
  

• Mean time to failure (MTTF) and repair in regression and load/stress testing.
  

• Test coverage.
  

• Usability, reliability, performance/scalability and supportability.

These technical testing measures are critical, but they are often blindly applied. Applied without a larger context, they can miss key economic implications inherent in the decision to continue testing or to release the software.

Instead of applying only technical criteria, the decision to release must also consider the timing of the window of opportunity that surrounds the software’s release date. Late release can mean lost revenue or lost efficiencies. However, early release can mean risk to the business in damaged reputation, organizational disruption and high service costs. To find this balance, companies should consider a number of factors, including:

• The amount of improvement that can be expected with more investment in testing.
  

• Economic risk reduction versus the lost economic benefit of release over time.

Early in the development cycle, the ECA presumably is much higher than the EVL.  The product does not do much, but it would generate lots of defects to fix.  As a result, the expected business risk (EBR), calculated as EBR = ECA - EVL, is a large positive number. Eventually, with good execution, the EBR should go to zero or even turn negative.

When considering the economic risks and value of deployment, a useful starting point is the common set of quality concerns known as FURPS — features, usability, reliability, performance and supportability:

• Features — Did we provide the most value? Most testing organizations do a fairly decent job of functional and regression testing.
  

• Usability — Can users accomplish the defined work efficiently? Normally, this area is examined through end-user acceptance testing.
  

• Reliability — Are the frequency of and recovery from software crashes in test or production adding unreasonable operational or liability costs?
  

• Performance — Are the software response times, throughput or both sufficient to meet business productivity needs?
  

• Supportability — Can the defects be isolated and removed economically?

The equation for reliability testing has two aspects — determining of ECA and determining EVL. In the sections that follow, we evaluate both issues.

where P[0,t] is the likelihood of failure before time t.

 Testing and repair should decrease the λ — in effect flattening and pushing to the right the failure density function distribution. Decreasing λ then means the failures are more likely to arise further in the future, making the program more reliable.

To get an approximate λ for a given build:

Once λ is known, it is possible to estimate the likelihood of failure between time 0 and any time t by computing the area under the curve from 0 to t.  Note this is true even if the time is far in the future.

To measure the EVL, the organization must have some idea of the value of the program at release. This value generally is time dependent — that is, the later the release, the less the program is worth. Some examples include:

• Contracted delivery with penalty clauses, awards fees based on acceptance or both.
  

• Delivery of a system to the marketplace when the profit depends on the timing of introduction.

The monetized business losses that result from reliability failures are directly related to the sort of software or system under development and the industry for which it is being developed.  The starting point, therefore, is to ask the question, "What does reliability look like to the business?"

A simple example is one that many individuals and businesses encounter daily: a laptop operating system (OS) is often restarted after several days of using many software programs simultaneously to avoid abnormal behaviors such as slow performance, screen errors or even locked screens that may occur later as a result of continuous use.  The developer of the OS knows that the cost of failure is low enough and that the OS meets standards of acceptable reliability if the MTTF is greater than the acceptable time between restarts.

Using the failure density function analysis described earlier, we now examine an example from the IBM Research Lab to substantiate the premise.

For example, a given build may use automated testing to run 10,000 reliability tests. Each test case launches the build into a parallel process. The number of processes that crashed during a day’s run are counted and recorded. The whole experiment is observed for a period of four days as illustrated in this table:

For build 1, λ = 2.63 × 10-3, and the chance of system failure for a continuous period of up to one month is determined by computing the area under the extrapolated curve according to the formula F(t) = 1 - e-λt = 99.96%. Note that it is possible to estimate this chance of failure even though the test may not last for the full one-month period.

Note also that additional measures of the reliability improvements from testing and repair activity can be measured by various statistical measures, e.g., the mean of the build fdf distributions. For example, the MTTF is the mean of the fdf distribution for a given build. A computation of MTTF for builds 1-5 is shown in table 2.

Measuring business risk

Now let’s consider the economic risks of testing. Once we know the likelihood of failure in any time period, we can also estimate the costs of addressing those failures. These costs depend on a variety of business variables such as the cost of labor and the purpose of the system. For each build, we need to estimate the following:

• Expected maintenance expenses if shipped now (m) — calculated from cost of repair multiplied by frequency
  
  

• Expected business loss due to system failure if shipped now (bl)

The business losses bl0, bl1 and maintenance expenses m0, m1 at times t0 and t1 are random variables, because they can only be estimated. With an assumed business loss basis of US$20 million, multiplying US$20 million by the probability of failure at the interval of time provides the bl variable. The maintenance stream is also provided as an estimated assumed set of variables.

ECA = mean(bl1 + m1 ) - mean (bl0 + m0)

Because the system or software may deliver value over some future interval of time, the net present value (NPV)—which we will refer to as net present revenue (NPR) of the system or software at delivery — must be computed. A useful form of the equation for this purpose is:

Where Ri is the expected stream of revenues or benefits (say, quarter by quarter) and rR is the interest rate associated with the time value of money (say, the treasury bill rate). Generally for development efforts, each of the Ri is a random variable, like the fdf.

Returning to our example, if we are told by the marketing department that the Ri for quarterly estimates is US$5 million, and rR = 6%, NPR for the system at different delivery dates is shown in table 4.

Similarly, the EVL due to missed opportunities during the testing period is determined by:

EVL= mean(NPR0 ) - mean(NPR1 )

The overall EBR of increased testing is:

EBR = ECA - EVL

For builds 2-5 of the example system, the ECA, EVL and EBR computation, therefore, is as follows:

Criteria for stopping tests — business risk versus cost of testing

With the EVT calculation, we have completed the economic analysis. We now find that in this example we are losing more money in the cost of testing and missed potential revenue than we are potentially saving by avoiding the risk of failure.  So now we need to ask ourselves, "Are there other reasons for creating another build?"  This is not a mechanistic decision, but the data does help guide our thinking. Other factors such as the corporate culture determine the next step.  Decision makers may want to elevate their confidence levels with one more build.

Quality governance: organizational and decision concerns

Conclusion

Deciding when testing is complete is both an economic and a technical issue.  Common FURPS-based testing provides significant insight into the needs, the process and the status of technical results.  But for a true understanding of new software’s readiness, the business impact of shipping now versus further testing and repair must be considered.

We have shown that the techniques that provide the required insights are available.  Not surprisingly, they do require some analytical capability and a new consideration in the governance of the release process.  However, by adopting these methods and release governance, the company’s management team can achieve a firm, economic basis for answering an important and very fundamental development question, -- "When am I done testing?"

Search the ENTIRE Business Forum site. Search includes the Business
Forum Library, The Business Forum Journal and the Calendar Pages.

Home    Calendar    The Business Forum Journal     Features    Concept    History
Library     Formats    Guest Testimonials    Client Testimonials    Experts    Search
News Wire     Join    Why Sponsor     Tell-A-Friend     Contact The Business Forum