The key measures of a test include coverage and quality.
Test coverage is the measurement of testing completeness, and it's based on the coverage of testing
expressed by the coverage of test requirements and test cases or by the coverage of executed code.
Quality is a measure of the reliability, stability, and performance of the target-of-test (system or
application-under-test). Quality is based on evaluating test results and analyzing change requests
(defects) identified during testing.
Coverage metrics provide answers to the question: "How complete is the testing?" The most commonly-used
measures of coverage are based on the coverage of software requirements and source code. Basically, test
coverage is any measure of completeness with respect to either a requirement (requirement-based), or the
code's design and implementation criteria (code-based), such as verifying use cases (requirement-based) or
executing all lines of code (code-based).
Any systematic testing task is based on at least one test coverage strategy. The coverage strategy guides
the design of test cases by stating the general purpose of the testing. The statement of coverage strategy
can be as simple as verifying all performance.
A requirements-based coverage strategy might be sufficient for yielding a quantifiable measure of testing
completeness if the requirements are completely cataloged. For example, if all performance test
requirements have been identified, then the test results can be referenced to get measures; for example,
75% of the performance test requirements have been verified.
If code-based coverage is applied, test strategies are formulated in terms of how much of the source code
has been executed by tests. This type of test coverage strategy is very important for safety-critical
systems.
Both measures can be derived manually (using the equations given in the next two headings) or may be
calculated using test automation tools.
Requirements-based test coverage, measured several times during the test lifecycle, identifies the test
coverage at a milestone in the testing lifecycle, such as the planned, implemented, executed, and
successful test coverage.
-
Test coverage is calculated using the following equation:
Test Coverage = T(p,i,x,s) / RfT
Where:
T is the number of Tests (planned, implemented, executed, or successful), expressed as test procedures
or test cases.
RfT is the total number of Requirements for Test.
-
In the Plan Test task, the test coverage is calculated to determine the planned test coverage in the
following manner:
Test Coverage (planned) = Tp / RfT
Where:
Tp is the number of planned Tests, expressed as test procedures or test cases.
RfT is the total number of Requirements for Test.
-
In the Implement Test task, as test procedures are being implemented (as test scripts) test coverage is
calculated using the following equation:
Test Coverage (implemented) = Ti / RfT
here:
Ti is the number of Tests implemented, expressed by the number of test procedures
or test cases for which there are corresponding test scripts.
RfT is the total number of Requirements for Test.
Successful Test Coverage (executed) = Ts / RfT
Where:
Ts is the number of Tests executed, expressed as test procedures or test
cases that completed successfully, without defects.
RfT is the total number of Requirements for Test.
Turning the above ratios into percentages allows for the following statement of requirements-based test
coverage:
x% of test cases (T(p,i,x,s) in the above equations) have been covered with a
success rate of y%
This meaningful statement of test coverage can be matched against a defined success criteria. If the
criteria have not been met, then the statement provides a basis for predicting how much testing effort
remains.
Code-based test coverage measures how much code has been executed during the test, compared to how much
code is left to execute. Code coverage can be based on control flows (statement, branch, or paths) or data
flows.
-
In control-flow coverage, the aim is to test lines of code, branch conditions, paths through the code,
or other elements of the software's flow of control.
-
In data-flow coverage, the aim is to test that data states remain valid through the operation of the
software; for example, that a data element is defined before it's used.
Code-based test coverage is calculated by the following equation:
Test Coverage = Ie / TIic
Where:
Ie is the number of items executed, expressed as code statements, code branches,
code paths, data state decision points, or data element names.
TIic is the total number of items in the code.
Turning this ratio into a percentage allows the following statement of code-based test coverage:
x% of test cases (I in the above equation) have been covered with a success rate of y%
This meaningful statement of test coverage can be matched against a defined success criteria. If the
criteria have not been met, then the statement provides a basis for predicting how much testing effort
remains.
Although evaluating test coverage provides a measure of the extent of completeness of the testing effort,
evaluating defects discovered during testing provides the best indication of the software quality as it has
been experienced. This perception of quality can be used to reason about the general quality of the
software system as a whole. Perceived Software Quality is a measure of how well the software meets the
requirements levied on it, therefore, in this context, defects are considered as a type of change request
in which the target-of-test failed to meet the software requirements.
Defect evaluation could be based on methods that range from simple defect counts to rigorous statistical
modeling.
Rigorous evaluation uses assumptions about the arrival or discovery rates of defects during the testing
process. A common model assumes that the rate follows a Poisson distribution. The actual data about defect
rates are then fit to the model. The resulting evaluation estimates the current software reliability and
predicts how the reliability will grow if testing and defect removal continue. This evaluation is described
as software-reliability growth modeling and it's an area of active study. Due to the lack of tool support
for this type of evaluation, you want to carefully balance the cost of using this approach with the
benefits gained.
Defects analysis involves analyzing the distribution of defects over the values of one or more of
the attributes associated with a defect. Defect analysis provides an indication of the reliability of the
software.
In defect analysis, four main defect attributes are commonly analyzed:
-
Status - the current state of the defect (open, being fixed, closed, and so forth).
-
Priority - the relative importance of this defect being addressed and resolved.
-
Severity - the relative impact of this defect to the user, an organization, third parties,
and so on.
-
Source - where and what is the originating fault that results in this defect or what component
will be fixed to eliminate this defect.
Defect counts can be reported as a function of time, creating a Defect Trend diagram or report. They can
also be reported in a Defect Density Report as a function of one or more defect attributes, like severity
or status. These types of analysis provide a perspective on the trends or on the distribution of defects
that reveal the software's reliability.
For example, it's expected that defect discovery rates will eventually diminish as the testing and fixing
progresses. A defect or poor quality threshold can be established at which point the software quality will
be unacceptable. Defect counts can also be reported based on the origin in the Implementation model,
allowing for detection of "weak modules", "hot spots", and parts of the software that keep being fixed
again and again, which indicates more fundamental design flaws.
Only confirmed defects are included in an analysis of this kind. Not all reported defects denote an actual
flaw; some might be enhancement requests outside of the project's scope, or may describe a defect that's
already been reported. However, it's valuable to look at and analyze why many defects, which are either
duplicates or not confirmed defects, are being reported.
The Rational Unified Process recommends defect evaluation based on multiple reporting categories, as
follows:
-
Defect Distribution (Density) Reports allow defect counts to be shown as a function of one or two
defect attributes.
-
Defect Age Reports are a special type of defect distribution report. Defect age reports show how long a
defect has been in a particular state, such as Open. In any age category, defects can also be sorted by
another attribute, such as Owner.
-
Defect Trend Reports show defect counts, by status (new, open, or closed), as a function of time. The
trend reports can be cumulative or non-cumulative.
Many of these reports are valuable in assessing software quality. They are most useful when analyzed in
conjunction with Test results and progress reports that show the results of the tests conducted over a
number of iterations and test cycles for the application-under-test. The usual test criteria include a
statement about the tolerable numbers of open defects in particular categories, such as severity class,
which is easily checked with an evaluation of defect distribution. By sorting or grouping this distribution
by test motivators, the evaluation can be focused on important areas of concern.
Normally tool support is required to effectively produce reports of this kind.
Defect status versus priority
Give each defect a priority. It's usually practical and sufficient to have four levels of priority, such
as:
-
Urgent priority (resolve immediately)
-
High priority
-
Normal priority
-
Low priority
Note: Criteria for a successful test could be expressed in terms of how the distribution of defects
over these priority levels should look. For example, successful test criteria might be "no Priority 1
defects and fewer than five Priority 2 defects are open". A defect distribution diagram, such as the
following, should be generated.
It's clear that the criteria has not been met. This diagram needs to include a filter to show only open
defects, as required by the test criteria.
Defect status versus severity
Defect Severity Reports show how many defects there are for each severity class; for example, fatal error,
major function not performed, minor annoyance.
Defect status versus location in the Implementation model
Defect Source Reports show distribution of defects on elements in the Implementation model.
Defect Age Analysis provides good feedback on the effectiveness of the testing and the defect removal
tasks. For example, if the majority of older, unresolved defects are in a pending-validation state, it
probably means that not enough resources are applied to the retesting effort.
Defect Trend Reports identify defect rates and provide a particularly good view of the state of the
testing. Defect trends follow a fairly predictable pattern in a testing cycle. Early in the cycle, the
defect rates rise quickly, then they reach a peak, and decrease at a slower rate over time.
To find problems, the project schedule can be reviewed in light of this trend. For example, if the defect
rates are still rising in the third week of a four-week test cycle, the project is clearly not on schedule.
This simple trend analysis assumes that defects are being fixed promptly and that the fixes are being
tested in subsequent builds, so that the rate of closing defects should follow the same profile as the rate
of finding defects. When this does not happen, it indicates a problem with the defect-resolution process;
the defect fixing resources or the resources to retest and validate fixes could be inadequate.
The trend reflected in this report shows that new defects are discovered and opened quickly at the
beginning of the project, and that they decrease over time. The trend for open defects is similar to that
for new defects, but lags slightly behind. The trend for closing defects increases over time as open
defects are fixed and verified. These trends depict a successful effort.
If your trends deviate dramatically from these, they may indicate a problem and identify when additional
resources need to be applied to specific areas of development or testing.
When combined with the measures of test coverage, the defect analysis provides a very good assessment on
which to base the test completion criteria.
Several measures are used for assessing the performance behaviors of the target-of-test and for focusing on
capturing data related to behaviors such as response time, timing profiles, execution flow, operational
reliability, and limits. Primarily, these measures are assessed in the Evaluate Test task, however, there
are performance measures that are used during the Execute Test task to evaluate test progress and status.
The primary performance measures include:
-
Dynamic Monitoring - real-time capture and display of the status and state of each test script
being executed during the test execution.
-
Response Time and Throughput Reports - measurement of the response times and throughput of the
target-of-test for specified actors and use cases.
-
Percentile Reports - percentile measurement and calculation of the data collected values.
-
Comparison Reports - differences or trends between two (or more) sets of data representing
different test executions.
-
Trace Reports - details of the messages and conversations between the actor (test script) and
the target-of-test.
Dynamic monitoring provides real-time display and reporting during test execution, typically in the form of
a histogram or a graph. The report monitors or assesses performance test execution by displaying the
current state, status, and progress of the test scripts.
For example, in the preceding histogram, there are 80 test scripts executing the same use case. In this
graph, 14 test scripts are in the Idle state, 12 in the Query, 34 in SQL Execution, 4 in SQL Connect, and
16 in the Other state. As the test progresses, you would expect to see the number of scripts in each state
change. The displayed output would be typical of a test execution that is executing normally and is in the
middle of its execution. However, if test scripts remain in one state or do not show changes during test
execution, this could indicate a problem with the test execution, or the need to implement or evaluate
other performance measures.
Response Time and Throughput Reports, as their name implies, measure and calculate the performance
behaviors related to time and throughput (number of transactions processed). Typically, these reports are
displayed as a graph with response time (or number of transactions) on the "y" axis and events on the "x"
axis.
It's often valuable to calculate and display statistical information, such as the mean and standard
deviation of the data values in addition to showing the actual performance behaviors.
Percentile Reports provide another statistical calculation of performance by displaying population
percentile values for data types collected.
It's important to compare the results of one performance test execution with that of another, so you can
evaluate the impact of changes made between test executions on the performance behaviors. Use Comparison
Reports to display the difference between two sets of data (each representing different test executions) or
trends between many executions of test.
When performance behaviors are unacceptable or when performance monitoring indicates possible bottlenecks
(such as when test scripts remain in a given state for exceedingly long periods), trace reporting could be
the most valuable report. Trace and Profile Reports display lower-level information. This information
includes the messages between the actor and the target-of-test, execution flow, data access, and the
function and system calls.
|