Quality By Design - Part 3

Abstract

"I'm going to cure the world of software bugs and make everybody's lives much easier." [Chaplin 2001]

Bugs in software are bad. They cost more and more to fix as they are discovered later in the project life-cycle. Often bugs that were fixed previously arise again after changes to the software. Testing is often cut short due to pressing deadlines and software is released with bugs still present. Developers often do not exhaustively test their own code each time they make changes.

This article, in three parts, explains how some simple design and development techniques can be used to eliminate all those bug concerns and massively accelerate your productivity. The three techniques described are Automated Unit Testing, used heavily in Extreme Programming(XP); Design By Contract, and Response Matrices.

By using these techniques you can quite quickly double your productivity, and in some cases increase productivity four or even five fold.

Introduction

In Part 1 I explained how to build Automated Unit Test Harnesses using simple assertions together with the benefits of implementing Automated Unit Test Harnesses.

In Part 2 , I explained how you can use Design By Contract techniques to make your software more robust and easier to integrate.

Finally, in this party, Part 3 , I’ll explain how you can use Response Matrices with Design By Contract to drive out exhaustive sets of Unit Tests for your classes.

Response Matrices

Response Matrices are the final step to take once you have mastered Design By Contract and Automated Testing. Using Response Matrices helps you drive out an exhaustive set of tests for a method to ensure it is 100% correct (adheres to specification) and that it is 100% robust (reacts gracefully to exceptions – i.e. doesn’t crash).

You use the preconditions and postconditions defined to help work out what tests you should write.

In general, a programmer would test there code (without preconditions and postconditions) by writing a form with an input box for the amount and then key in a few values and report the balance to the screen, and visually check that things were working okay. This is time consuming (see Automated Unit Testing in Part 1) and also not always exhaustive – how would you pass that ‘unique’ test harness onto someone else? How would they know what to test?

Ideally, you want to build an exhaustive set of tests that run very quickly and exhaustively. Then you can pass them onto someone else and of course re-use them yourself. We have solved the quickly part using Automated Unit Testing. It just remains to work out what the tests are for the exhaustive part. Here goes…..

To build a Response Matrix we build a truth table using the preconditions and the postconditions. We then define the tests we need to run to cover all possibilities. E.g.

Account::Withdraw(idblAmount)
------------------------------------

Pre1: idblAmount > 0
Pre2: idblAmount <= Balance+Overdraft
Post: Balance = Balance@pre - idblAmount

Pre1	Pre 2	Expected Outcome	Test No
TRUE	TRUE	Balance = Balance@pre - idblAmount	1
TRUE	FALSE	Error: Amount must be greater than zero	2
FALSE	TRUE	Error: Amount must be less than balance + overdraft	3
FALSE	FALSE	N/A [Can't have a -ve amount and fail Pre2]	4

Thus, we could use the following tests:

What are the Automated Unit Tests?

They will look like this:

Dim objCAccount As CAccount

Set objCAccount = New CAccount
[Assume at this stage that the overdraft is set to 1000, and the balance is 300]

'/* Test 1. */'
On Error Resume Next
[Assume you can call something here to set the account state, balance = 20, od = 0]
dblBalanceBefore = objCAccount.Balance
ObjCAccount.WithDraw(10)
Debug.Assert Err.Number = (objCAccount.Balance = dblBalanceBefore - 10)

'/* Test 3. */'
On Error Resume Next
[Assume you can call something here to set the account state, balance =2, od = 5]
ObjCAccount.WithDraw(10)
Debug.Assert Err.Number = ERR_OD_VIOLATED

'/* Test 4. */'
On Error Resume Next
[Assume you can call something here to set the account state, balance =2, od = 5]
ObjCAccount.WithDraw(-10)
Debug.Assert Err.Number = ERR_AMT_INVALID

Summary

This is a very simplistic example. When you have an object with complex behaviour with many states, and methods with many input parameters this matrix can become quite large. If the number of tests for an interface becomes large (say more than 20) you might want to re think the design. Keeping things simple is the key to bug free software.

The maximum number of tests for an interface is two to the power of the number of preconditions.
Putting Parts 1, 2 and 3 toget

For beginners to these techniques I would suggest the following track:

1. Start by writing simple automated test harnesses using assertions. These harnesses should have no screens, but simply run on start up.
2. Migrate your tests to use a front end unit test tool – xUnit is the standard. 
3. Once you have got yourself up to speed with automated tests, try using Design By Contract when writing your next set of classes. You’ll find them much easier to debug, and also you’ll find you do not spend as much time trying to help others use your components.
4. Once you have mastered the basic of DBC for single state objects, try building a response matrix for the next class you write. You will discover tests that you had never though of before.
   

Getting Very Advanced

Once you get very advanced at this, you will be using a standard automated unit test tool for everything written on your project. You will be able to load in sets of tests and run them remotely on any machine. If they take a while to run you will be running them overnight and checking the results in the morning. You will have used Design By Contract throughout the code to make sure all contracts are adhered to. Your critical components will have been released completely bug free, since you used formal methods (Response Matrices) to drive out the set of tests required.

At this point you will spend most of your time creating new products, developing code, and of course writing automated unit tests for them.

Perhaps when you get to this stage you can buy me a pint or two !

Good luck.

Dave Chaplin