This guideline shows how to identify test ideas from statecharts and other design structures that consist
mainly of nodes connected by arcs and that show something of the possible control flows of a program. The
main goal of this testing is to traverse every arc in some test. If you've never exercised an arc, why do
you think it will work when a customer does?
Consider this statechart:
Fig1: HVAC Statechart
Here's a first list of test ideas:
-
Idle state receives Too Hot event
-
Idle state receives Too Cool event
-
Cooling/Startup state receives Compressor Running event
-
Cooling/Ready state receives Fan Running event
-
Cooling/Running state receives OK event
-
Cooling/Running state receives Failure event
-
Failure state receives Failure Cleared event
-
Heating state receives OK event
-
Heating state receives Failure event
These test ideas could all be exercised in a single test, or you could create several tests that each
exercise a few. As with all test design, strive for a balance between the ease of implementation of many
simple tests and the additional defect-finding power of complex tests. (See "test design using the list" in the Concept: Test Ideas List page.) If you have use case scenarios that
describe certain paths through the statechart, you should favor tests that take those paths.
In any case, the tests should check that all actions required by the statechart actually take place. For
example, is the alarm started on entry to the Failure state, then stopped upon exit?
The test should also check that the transition leads to the correct next state. That can be a difficult
problem when the states are invisible from the outside. The only way to detect an incorrect state is to
inject some sequence of events that leads to incorrect output. More precisely, you would need to construct
a follow-on sequence of events whose externally-visible results for the correct state differ from
those that the same sequence would provoke from each possible incorrect state.
In the example above, how would you know that the Failure Cleared event in the Failure state correctly led
to the Idle state, instead of staying in the Failure state? You might trust that the stopping of the Alarm
meant that transition had been made, but it might be better to check by lowering the temperature enough to
make the heater start or raising it enough to turn on cooling. If something happens, you're more confident
that the transition was correct. If nothing happens, it's likely the device stayed in the Failure state.
At the very least, determining whether the resulting state is correct complicates test design. It is often
better to make the state machine explicit and make its states visible to the tests.
Other statechart constructs
Statecharts consist of more than arcs and arrows. Here is a list of statechart constructs and the effect
they have on the test idea list.
Event actions, entry actions, and exit actions
These do not generate test ideas per se. Rather, the tests should check that the actions behave as
specified. If the actions represent substantial programs, those programs must be tested. The test ideas for
the programs might be combined with test ideas from the statechart, but it's probably more manageable to
separate them. Make the decision based on the effort involved and on your suspicion that there might be
interactions between events. That is, if a particular action on one arc cannot possibly share data with an
action on another arc, there is no reason to exercise the two actions in the same test (as you would if
they were part of the same path through a statechart test).
Guard conditions
Guard conditions are boolean expressions. The test ideas for guard conditions are derived as described in
Guideline: Test Ideas for Booleans and Boundaries.
In the example above, the Too Cool transition from the Idle state is guarded with [restart time >= 5
mins]. That leads to two separate test ideas:
-
Idle state receives Too Cool event when restart time is five minutes (transition taken)
-
Idle state receives Too Cool event when restart time is just less than five minutes (transition
blocked)
In both cases, any test that uses the test idea should check that the correct state is reached.
Internal transitions
An internal transition adds the same sort of ideas to a test idea list as an external transition does. It's
merely that the next state is the same as the original state. It would be prudent to set up the test such
that the state's entry and exit actions would cause an observable effect if they were incorrectly
triggered.
Nested states
When constructing tests, set them up such that entry and exit events of the composite state have observable
effects. You want to notice if they're skipped.
Concurrent substates
Testing of concurrency falls outside of the scope of developer testing.
Deferred events
If you suspect an event might be handled differently depending on whether it was deferred and queued rather
than generated while the program was actually in the receiving state, you might test those two cases.
If the event in the receiving state has a guard condition, consider the ramifications of changes to the
condition's variables between the time the event is generated and the time it is received.
If more than one state can handle a deferred event, consider testing deferral to each of the possible
receiving states. Perhaps the implementation assumes that the "obvious" state will handle the event.
History states
Here is an example of a history state:
Fig2: History State Example
The transition into the history state represents three real transitions, and thus three test ideas:
-
BackupUp event in Command state leads to Collecting state
-
BackupUp event in Command state leads to Copying state
-
BackupUp event in Command state leads to CleaningUp state
Chain states
Chain states do not seem to have any implications for test design, except that they introduce more actions
that need to be checked.
The preceding discussion focuses on checking whether the implementation matches the design. But the design
might also be wrong. While examining the design to find test ideas, also check for two types of problems:
Missing events. The statechart shows a state's response to events that the designer anticipated
could arrive in that state. It's not unknown for designers to overlook events. For example, in this
statechart (repeated from the top of the page), perhaps the designer forgot that a failure can occur in the
Ready substate of Cooling, not just when the fan is Running.
Fig3: HVAC Statechart
For this reason, it's wise to ask, for each state, whether any of the events that apply to other states
might apply to this one. If you discover that one does, correct your design.
Incomplete or missing guard conditions. Similarly, perhaps guard conditions on one transition will
suggest guard conditions on others. For example, the above statechart takes care not to restart the heater
too often, but there is no such restriction on the cooling system. Should there be?
It is also possible that variables used on one guard condition will suggest that other guard conditions are
too simple.
Testing each arc in a graph is by no means complete testing. For example, suppose the start state
initializes a variable to 0, state Setter sets it to 5, and state Divider divides it into 100
(100/variable). If there's a path from the start state to Divider that does not pass through Setter, you
have a divide-by-zero exception. If the statechart has many states, simply exercising each arc might miss
that path.
Except for very simple statecharts, testing every path is infeasible. In practice, tests that are complex
and correspond to use case scenarios are often sufficient. If you desire stronger tests, consider requiring
a path from each state where a datum is given a value to each state that uses it.
|