Moving to XP

[Thanks to Ron Jefferies (see www.XProgramming.com) for great feedback on the original article. Ron is a co author of Extreme Programming Installed]

Abstract

Agile Development and XP

The Agile Manifesto For Software Development was born. It has 4 core values:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

While there is value in the items on the right, the items on the left are valued more.

On the whole programmers tend to love XP. This is because they spend more time on the things they enjoy, like designing and coding, than the activities they find boring, like requirements documentation and test plans.

There are contentious debates in the market between the heavy methodology folk and the Agile folk. XP is often accused of being nothing but glorified hacking. I strongly believe that it isn’t. However, if you don’t practise all the Core XPvalues then it can easily become that and you can get into quite a mess, something I have experienced first hand. There are situations when XP is a good choice and situations when it is wholly inappropriate.

When Is XP Appropriate?

One could argue that if you don’t know the requirements then you are asking the users the wrong questions and failing to diagnose their problem. [This often happens in IT projects where the developers (and even the business analysts) ask the users what they want rather than trying to diagnose the business problem. This is a bit like doctor asking you what medication you would like for your illness, rather than diagnosing the problem and suggesting appropriate medication.] Anyway, the brutal truth of matter is that in many projects you don’t know if you are heading in the right direction until you get feedback from the first and subsequent releases. This is why iterative development is so widespread. In XP, the iterations can be very small (2 weeks) and there is always a working system that can be released at any point, as a result of Continuous Integration.

There is also a very compelling argument to use XP when the requirements are also known up front. Since XP provides irreplaceable feedback on what is happening in the project, through delivering tested running software, even if there is no requirements risk.

It would not be appropriate to use XP in critical applications where a waterfall development and rigorous formal testing methods would be required to ensure the safety of the users. Another example where XP isn’t required is the situation where the requirements are known up front, and hence no requirements risk. An example might be porting a windows application to a web application. Granted, you would need some further interaction design doing, but the functionality set would remain the same.

In summary, XP is well suited to projects where the requirements are unknown, vague, and are bound to change. The excellent book Questioning Extreme Programming addresses many of these issues in much more detail.

I discuss how to make XP a success in another article.

Values and Techniques Of XP

XP is based around 4 key values:

Simplicity: Build the simplest thing that will work. No future proofing code.
Communication: High quality and quantity increases understanding.
Feedback: As much as often as possible keeps the direction on course.
Courage: Confidence and courage to embrace those changes when they arrive.

Rather than designing and building a whole bunch of code to meet future requirements, we build the simplest thing that will work for the immediate requirements. The code must be clean, and well designed, but there must be no future proofing. We acknowledge from our experience that on many occasions all that future proofing code often never gets used as the project and requirements change. On projects whose direction changes the cost of designing, building, and then not using future proof code is far higher than just building what you need when you need it. Keeping it simple lowers the complexity which increases the chance of higher quality and productivity. In XP this is called YAGNI (You Aren’t Gonna Need It).

When changes come along that require an enhanced design we use Refactoring to make the changes, ensuring that everything we has previously written still works. Refactoring is the process of improving the design of existing code without changing the functionality. The thought of ripping up existing working code can give project mangers a concern (if it ain’t broke don’t fix it etc….). But, because we have an entire suite of automated tests that run very quickly (<5 minutes), we can refactor away and evolve the design ensuring that we do not break existing functionality. To do this effectively you need good design: good abstractions and good decoupling. Suffice to say, to be an effective XP developer you need to be good at layered design, good at OOP, and know your patterns. If you develop using a 2-tier procedural type style then XP really is not for you.

The tests I mentioned above are unit tests that are written before the code using Test Driven Development (TDD) – one of the integral parts of XP. The Test Driven side of things ensures that what you can write is automatically testable and meets the requirements. The outcome is that you end up with a suite of unit tests you can use as a regression test harness. Expect to have as many lines of test code as production code. This is by no means a bad thing. By writing tests first you often find your initial design needs updating, before you build it, rather than after a first cut.

When we have a whole suite of tests we then use Continuous Integration to ensure that the build is never broken and we can release a working version of the system. This means we have a process running that continuously gets the latest version of the source code, builds it, and runs all the tests. For a full description of this see Martin Fowlers excellent article ‘Continuous Integration’.

So, we write unit tests before we write code, then use those when we refactor to evolve the design to meet the new requirements. What about communication?

Rather than write lengthy requirements documents the documentation is kept light. XP puts more focus on working code that meets requirements than documentation. We know things are going to change, and we know the best way of proving if something works is to actually build it. XP also acknowledges that the best form of communication is face to face. If the development team needs to clarify the requirements they talk directly to the customer. XP uses the concept of On-Site Customer. This means that a customer sits with the development team at all times. This maximises the quality and quantity of communication to help ensure the right thing is built.

Pair programming is also an integral part of XP. This is a contentious part of XP. This is where each line of code is written by two programmers sitting at a single computer. One is the ‘driver’ (who types at the keyboard) and the other is the ‘navigator’ (who watches the driver and thinks about the bigger picture). The roles can be interchanged during a session. Pair Programming removes the need for code reviews at a later date, since code reviews are done as the code is being written. This removes bugs and poor design earlier in the cycle. The technique results in higher quality code being produced quicker. The reason Pair Programming is so effective is due to the driver and navigator roles each developer adopts during the session. This is explained in more detail in the book ‘Pair Programming Illuminated’. Pair programming is not for everyone and you have to be careful about who is paired together and why they are paired together. Personally, if I’m with the right partner then I think Pair Programming is superb.

In terms of design XP uses a Metaphor for each project. This helps keep everyone on track with regard to design. It’s a way of thinking about the high level design.

Requirements headlines are decided during the Planning Game. This is where the requirements are written on index cards, with enough detail for the customers to understand and for developers to put timescale estimates to. The customer then gets to decide what will go into the next iteration.

To summarise, here is the main list of techniques for XP:

• Pair Programming
• Refactoring
• Test-Driven Development
• On-Site Customer
• Continuous Integration
• Metaphor
• Planning Game

Implementing XP

My father taught me from a very young age that the best way to eat an elephant is in small manageable pieces, and implementing XP is the same. If you tried to do the whole thing in one go from just reading the theory then it is very likely that you would fail. If you can employ an XP Champion who has been there and done it, rather than just read the theory then you will get going faster. But, it isn’t mandatory. It will take much longer without one though, like anything you try to attempt without a guiding hand. I’ll talk about timescales later on.

There are some dependencies in the techniques.

• The ability to release a working system at any point is enhanced with continuous integration.
• Continuous integration relies on automated tests being in place. Refactoring a system effectively and efficiently requires solid automated testing.
• Getting a good suite of tests is enhanced by using test driven development.
• To embrace the changes quickly, the ability to refactor effectively is paramount.
• Evolving the design using refactoring is aided by the use of a metaphor.
• The chances of higher quality code built first time are increased when using pair programming.
• Lightening the requirements gathering needs good commincation. i.e. an On-Site Customer.

1. Automated Testing / Test Driven Development
2. Refactoring / Metaphor
3. Pair Programming
4. Continuous Integration
5. Planning Game / On-Site Customer

Doing Partial XP

Automated Testing / Test Driven Development (TDD)
If you are starting fresh on a project then you can practise TFP from the start. You will need to learn the techniques of writing tests which is pretty quick to do. I’d suggest using a version of xUnit which has been ported to most languages. Kent Beck’s book ‘Test Driven Development’ is a very good primer for the technique. My article also discusses the practicalities of implementing TDD.  

The real skill in TDD is knowing what to test. By this I mean the scope and strength (or weakness) of the tests. In a layered system you only want to write a test that tests the scope of the component under test, and not allow the assertions to flow further down the architecture. If you are not careful you can end up with duplication in the test code which will make the tests tough to maintain, and also longer to run. This skill is something that only comes with experience. In terms of the test time you need to get the total test time for all tests very quick. I’d suggest an ideal of less than 2 minutes, a middle ground of 5, and certainly no more than 10.

There are more projects in progress than there are projects about to start, so the question remains as to how to introduce TDD onto an existing project.

The first step is learning how to write automated tests. My article ‘Writing Automated Tests Using NUnit and VB.NET’ will show you how to do this. Next you need to decide what to test. Start by writing tests for any code that is new, being enhanced, or being fixed. Over time you will then end up with tests for the whole code base. To be able to write tests you need to ensure that your system has sufficient levels of abstraction. Using Model View Controller is a good place to start. Your functions need to be small with a single responsibility. If you are suffering from large classes with large methods (see Blob Antipattern) then you’ll need to split things out (see Extract Method in Refactoring as a starter).

It is possible to write automated tests for the UI of your application which I’ve explained in ‘Automating Browser based tests with IE and NUnit’. You can also do the same for Windows based apps. However, they tend to take a while to run (>20 minutes), and are pretty useless as a tool for regression testing during development where typically you want to run the full suite over 10-20 times a day. They are great however for automating acceptance tests, and particularly good for testing previously raised bugs. The great thing about writing tests for previously raised bugs is that you can guarantee that once a bug is fixed once it will never raise it’s ugly head again. As an example, on my current project we have just under 400 unit tests that run in 85 seconds, and 116 front end IE tests that run in 25 minutes. We have only really just started. On my previous project I had around 2500 unit tests.

The process of bolting down your existing application with automated tests will help you learn more about the art of what and when to test. With that experience and know-how you can then begin to write tests before you write code. And this is when you make the serious leap into the high quality zone. To judge when to make that leap watch out for your own personal ‘Moment of Test First Revelation’. During a design session with other developers you will be discussing how to implement a certain piece of functionality and someone will suggest a solution. When you pitch in and say ‘Yes, but how will we test it…’ that is your Moment of Test Driven Revelation. You’ve made the leap and are ready to start full TFP.

The productivity gains you get from TFP are huge on their own which you can still reap regardless of whether you are doing full XP or not. And trust me, once you have made that leap, you will never want to develop any other way again. I discuss other ways to maximise productivity in a seperate article.

Refactoring / Metaphor
So, you now have the application bolted down with tests and are starting to write some tests before you write code. You now have some new requirements to implement which require some refactoring of existing code.

If you are working alone then you can probably do without the Metaphor concept, since you will have a good one in your head anyway. The Metaphor concept is for teams of developers as a useful aid to communication. You need to make sure that everyone knows what the high level design of the system is. Before you start refactoring in teams you need to choose a Metaphor. Personally, I’ve not got stuck into the whole Metaphor thing. I use them all the time when describing architecture but tend to choose different ones for different purposes. Instead I tend to use a small example application for an architectural spike that the developers can refer to, in conjunction with a good high level class diagram and supporting sequence diagram for the front to back architecture. This gets supplemented with a document – albeit a very light one.

There are also different sizes of Refactoring that will need to be done. The small ones are things like Extract Method, to make the code more manageable or remove duplication. A larger refactoring might involve putting in a security pattern for access control. For the larger refactorings I strongly recommend doing some whiteboard design work before you start. I like to call this Small Model Up Front. Also, before implementing any new functionality I strongly advise doing some Small Model Up Front. One of the reasons XP is labelled as hacky is due to the fact that people think it does not involve design.

[Warning: If you don’t practise refactoring you are hacking. The code base will rot and turn into a time bomb that is waiting to go off. I know this from personal experience where a bad decision was made to put off a 2 week refactoring job to hit the deadline. Further functionality was built on top of rotting code which resulted in 8 weeks worth of extra effort down the line. Never put off refactorings. Do them as you need them otherwise they build up and blow up in your face. It is something that must be done continuously rather than as separate task on the plan.]

So, to summarise, when refactoring, ensure the team knows the architectural pattern, and for the larger refactorings or new functionality practise some Small Model Up Front.

Pair Programming
At this point you aren’t doing Pair Programming yet. So, you should still be doing code reviews instead. So, when should you start doing Pair Programming? You can actually do this whenever you want, even as the first thing if you wish. It is a great tool for teaching other developers. But, as I stated at the start, the best way to eat an elephant is in small manageable chunks. The most effective mouthful in my opinion to start on is Automated Testing. Pair Programming is lower on the menu.

Before deciding to do Pair Programming I strongly recommend reading ‘Pair Programming Illuminated’ so you understand the techniques. Deciding who to pair together depends on what you are trying to achieve. Also, each developer in a pair needs to understand the roles and responsibilities of driver and navigator to make it work most effectively.

So, when should you start Pair Programming? If you have an XP champion on board, then introduce this right at the start as a means to teaching. When you have the team up and running with Test Driven Development, it is then a good time to start pair programming to increase the code quality and turf out those code reviews. Code reviews are traditionally much too late in the cycle and are often left out.

Continuous Integration
When you have all the application bolted down with tests you are doing what I would call ‘discontinuous Integration’. This is when the developer runs the tests before and after the coding to ensure they do not break the build before checking their work back in. Bear in mind that developers are not perfect and accidents to happen where the build gets broken. This situation can slow down the development pace whilst all developers down tools to get it working again. To know immediately when the build gets broken helps the developers identify where in the code the break was introduced.

Earlier on I talked about automating the system tests and the amount of time they took to run. Continuous integration resolves this issue, since a developers time is not taken away whilst their machine is tied up running tests that take a long time.

There are tools on the market to help with continuous integration. Some are Draco.Net, Ant, Nant, and Cruise Control by ThoughtWorks. On my current project we wrote something ourselves to do this, which took about a week. This was before Cruise Control.Net came along which would have done the same thing for us. It’s not that tough to write your own. The advantage of writing it yourself is that you can add lots of project specific detail to it. For example: updating version numbers in your bug tracking system, emailing people, or moving code to drop zones. Saying that, I understand Cruise Control.Net provides a framework for plugging your own pieces in. I’d recommend talking a look at it. The ThoughtWorks folks know a lot about Agile Development and from a first glance their tool looks very well thought out.

When should you start attacking Continuous Integration? My advice is to wait until you’ve got a good automated test driven approach running with a full suite of tests, both unit test and system tests. Before trying to automate it, start manually doing a build at the end of each day and running those tests. Doing a daily build and running the tests is also a good way of getting a crude measure of your project velocity. i.e how many new tests are added to the system. Remember that at this point you aren’t writing any code until you’ve written the tests. When you find you get a couple of weeks worth of green lights at the end of every night you’ll know your okay, and it is worth automating the integration. This saves the team time at the end of each day doing a build, and also means you can build and test many times a day.

Planning Game / On-Site Customer
Once you have a good development process up and running you’ll be in a situation where you can react quickly, and effectively to requirements changes without allowing the code base to rot. It’s at that point you can go for lighter requirements documents and an On-Site Customer. In some projects you will find it difficult to get an On-Site Customer. If you can’t it is still very important to get as much access to the primary users as possible. The more face to face conversational access to users you can get the more understanding you will get of their business problems. The more understanding you get, the better you will be at being able to suggest good uses of technology to solve their problems, and the more change you will get at getting the requirements right and building the right thing. If you cannot get as much access as you want then you’ll need to go heavier on the requirements beforehand, perhaps using tried and tested techniques like prototyping, and big heavy requirements documents (yuk!)

So, use the planning game when your house is in order and you’re well placed to react well to changes in requirements. Try to get as close as possible to an On-Site Customer in order to lighten the amount of up front requirements gathering.

How Long Will It Take
You can start tomorrow and start getting immediate results. It will just get better and better as time goes on. The first step is getting Automated Testing and Test Driven Development in. Although you can show someone in a week how to do this using a tool like xUnit, it takes quite while before you get to the point where you get very good at it. The first time I implemented this on a team I was learning the technique myself from scratch with no mentor. It took about 3 weeks to start getting some real value, and about 6 months to get the whole team seriously well oiled in just Automated Testing. It then took about 1 day to get started on Test Driven Development, and a further 3-4 months to get really good at it. The benefits of implementing this technique alone were huge. If you do not have an XP Champion to guide you then expect to take double the amount of time than if you had one.

You can practise refactoring from the start without having automated tests. It is just far riskier and takes much longer, due to the length of time it takes to manually regression test. I’d advise you bolt the system down with automated tests before you attack any large refactorings.

Once you’ve got Test Driven Development in you’ve tackled the biggest part. After that I’d suggest 6 weeks to get Continuous Integration bedded in. You can do Continuous Integration from the start if you wish, but this will rely on a good development process in place so that the build is not broken that often. Otherwise, you can spend ages sorting that out, when the real issue is actually a process one.

In terms of iteration planning. If you can do this from the start then great. This is again a process issue. If it takes 1 week to integrate and release, then there is not much point having a 2 week iteration. Once you've got this licked though, you can iterations of a few hours if you really want. I find one month is pretty good, with 2 days at the end for tidy up and preparation for deployment. Deployment then takes about 30 minutes.

Use these estimates as a very rough guide. The time it will take depends on the size of the project, and whether the design lends itself to easy testability.

Closing Comments
If you are thinking about embracing XP on your project then I hope you found this article useful. XP is both an extremely productive way of working and also a very enjoyable one. If you wish to email me any comments or questions based on this please do. I’d also like hear how you get on.

Good luck. Happy developing.