Like physical objects, tests can break. It's not that they wear down, it's that something's changed in
their environment. Perhaps they've been ported to a new operating system. Or-more likely-the code they
exercise has changed in a way that correctly causes the test to fail. Suppose you're working on
version 2.0 of an e-banking application. In version 1.0, this method was used to log in:
public boolean login (String username);
In version 2.0, the marketing department has realized that password protection might be a good idea. So the
method is changed to this:
public boolean login (String username, String password);
Any test that uses login will fail
It won't even compile.
Since not much useful work is possible at this point, not many useful tests can be written without login.
You might be faced with hundreds or thousands of failing tests.
These tests can be fixed by using a global search-and-replace tool that finds every instance of
login(something) and replaces it with login(something, "dummy
password"). Then arrange for all the testing accounts to use that password, and you're on your way.
Then, when marketing decides that passwords should not be allowed to contain spaces, you get to do it all
over again.
This kind of thing is a wasteful burden, especially when-as is often the case-the test changes aren't so
easily made. There is a better way.
Suppose that the tests originally did not call the product's login method. Rather,
they called a library method that does whatever it takes to get the test logged in and ready to proceed.
Initially, that method might look like this:
public boolean testLogin (String username) {
return product.login(username);
}
When the version 2.0 change happens, the utility library is changed to match:
public Boolean testLogin (String username) {
return product.login(username, "dummy password");
}
Instead of a changing a thousand tests, you change one method.
Ideally , all the needed library methods would be available at the beginning of the testing effort. In
practice, they can't all be anticipated-you might not realize you need a testLogin
utility method until the first time the product login changes. So test utility
methods are often "factored out" of existing tests as needed. It is very important that you perform
this ongoing test repair, even under schedule pressure. If you do not, you will waste much time dealing
with an ugly and un-maintainable test suite. You might well find yourself throwing it away, or being unable
to write the needed numbers of new tests because all your available testing time is spent maintaining old
ones.
Note: the tests of the product's login method will still call it directly. If
its behavior changes, some or all of those tests will need to be updated. (If none of the login tests fail when its behavior changes, they're probably not very good at detecting
defects.)
The previous example showed how tests can abstract away from the concrete application. Most likely you can
do considerably more abstraction. You might find that a number of tests begin with a common sequence of
method calls: they log in, set up some state, and navigate to the part of the application you're testing.
Only then does each test do something different. All this setup could-and should-be contained in a
single method with an evocative name such as readyAccountForWireTransfer. By doing
that, you're saving considerable time when new tests of a particular type are written, and you're also
making the intent of each test much more understandable.
Understandable tests are important. A common problem with old test suites is that no one knows what the
tests are doing or why. When they break, the tendency is to fix them in the simplest possible way. That
often results in tests that are weaker at finding defects. They no longer test what they were originally
intended to test.
Suppose you're testing a compiler. Some of the first classes written define the compiler's internal parse
tree and the transformations made upon it. You have a number of tests that construct parse trees and test
the transformations. One such test might look like this:
/*
* Given
* while (i<0) { f(a+i); i++;}
* "a+i" cannot be hoisted from the loop because
* it contains a variable changed in the loop.
*/
loopTest = new LessOp(new Token("i"), new Token("0"));
aPlusI = new PlusOp(new Token("a"), new Token("i"));
statement1 = new Statement(new Funcall(new Token("f"), aPlusI));
statement2 = new Statement(new PostIncr(new Token("i"));
loop = new While(loopTest, new Block(statement1, statement2));
expect(false, loop.canHoist(aPlusI))
This is a difficult test to read. Suppose that time passes. Something changes that requires you to update
the tests. At this point, you have more product infrastructure to draw upon. In particular, you might have
a parsing routine that turns strings into parse trees. It would be better at this point to completely
rewrite the tests to use it:
loop=Parser.parse("while (i<0) { f(a+i); i++; }");
// Get a pointer to the "a+i" part of the loop.
aPlusI = loop.body.statements[0].args[0];
expect(false, loop.canHoist(aPlusI));
Such tests will be much easier to understand, which will save time immediately and in the future. In fact,
their maintenance costs are so much lower that it might make sense to defer most of them until the parser
is available.
There's a slight downside to this approach: such tests might discover a defect in either the transformation
code (as intended) or in the parser (by accident). So problem isolation and debugging may be somewhat more
difficult. On the other hand, finding a problem that the parser tests miss isn't such a bad thing.
There is also a chance that a defect in the parser might mask a defect in the transformation code. The
chance of this is rather small, and the cost from it is almost certainly less than the cost of maintaining
the more complicated tests.
A large test suite will contain some blocks of tests that don't change. They correspond to stable areas in
the application. Other blocks of tests will change often. They correspond to areas in the application where
behavior is changing often. These latter blocks of test will tend to make heavier use of utility libraries.
Each test will test specific behaviors in the changeable area. The utility libraries are designed to allow
such a test to check its targeted behaviors while remaining relatively immune to changes in untested
behaviors.
For example, the "loop hoisting" test shown above is now immune to the details of how parse trees are
built. It is still sensitive to the structure of a while loop's parse tree
(because of the sequences of accesses required to fetch the sub-tree for a+i). If that
structure proves changeable, the test can be made more abstract by creating a fetchSubtree utility method:
loop=Parser.parse("while (i<0) { f(a+i); i++; }");
aPlusI = fetchSubtree(loop, "a+i");
expect(false, loop.canHoist(aPlusI));
The test is now sensitive only to two things: the definition of the language (for example, that integers
can be incremented with ++), and the rules governing loop hoisting (the behavior
whose correctness it's checking).
Even with utility libraries, a test might periodically be broken by behavior changes that have nothing to
do with what it checks. Fixing the test doesn't stand much of a chance of finding a defect due to the
change; it's something you do to preserve the test's chance of finding some other defect someday. But the
cost of such a series of fixes might exceed the value of the test's hypothetically finding a defect. It
might be better to simply throw the test away and devote the effort to creating new tests with greater
value.
Most people resist the notion of throwing away a test-at least until they're so overwhelmed by the
maintenance burden that they throw all the tests away. It is better to make the decision carefully
and continuously, test by test, asking:
-
How much work will it be to fix this test well, perhaps adding to the utility library?
-
How else might the time be used?
-
How likely is it that the test will find serious defects in the future? What's been the track record of
it and related tests?
-
How long will it be before the test breaks again?
The answers to these questions will be rough estimates or even guesses. But asking them will yield better
results than simply having a policy of fixing all tests.
Another reason to throw away tests is that they are now redundant. For example, early in development, there
might be a multitude of simple tests of basic parse-tree construction methods (the LessOp constructor and the like). Later, during the writing of the parser, there will be a
number of parser tests. Since the parser uses the construction methods, the parser tests will also
indirectly test them. As code changes break the construction tests, it's reasonable to discard some of them
as being redundant. Of course, any new or changed construction behavior will need new tests. They might be
implemented directly (if they're hard to test thoroughly through the parser) or indirectly (if tests
through the parser are adequate and more maintainable).
|