Tuesday, June 28, 2005

Iterative Software Development 

There's an interesting discussion over on Tom Kyte's blog about the software development process. It drew the following comment which I thought worth looking at a little more.

RAD and iterative development methods have pretty much killed design skills as far as I can see. Not that there is anything wrong with these as concepts. As always the problem is their implementation. A large number of development projects take the view that iterative development means you don't have to think at all, just start. I hear quotes like, "We'll refactor that later!", all the time. The problem is that later never comes.

I repeatedly think back to my early days in IT working on real-time control systems with Oracle as a back end. We created requirement specs, functional specs and module specs, all of which were signed off by the customer. We then designed our DB to support the requirements. We wrote pseudocode, had it reviewed, then turned it into real code (PL/SQL and Pro*C). More often than not it worked first time due to the details and care taken along the way. when it was over the customers liked what they were given because they were involved in the whole process.

Fast forward a few years and what do we have, a spec in a single email that the customer hasn't approved, but, "It's OK! We can change it later. It's iterative development!"

Now that's a cariacature, but I'm rather afraid not actually that far from the truth.

RAD and iterative development methods have pretty much killed design skills as far as I can see. Not that there is anything wrong with these as concepts. As always the problem is their implementation. A large number of development projects take the view that iterative development means you don't have to think at all, just start. I hear quotes like, "We'll refactor that later!", all the time. The problem is that later never comes.

Consider also the Agile Manifesto
Manifesto for Agile Software Development

We are uncovering better ways of developing
software by doing it and helping others do it.
Through this work we have come to value:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

This is another formulation of the same idea, that good software is developed not by careful design, appropriate modelling of the business problem you are trying to solve or the task you are trying to automate, but by putting it together as you go. If it doesn't work then undo it and try again, or more probably throw in a case specific fudge. There is something wrong with this as a concept, as well as practical implications, but it is the idea that RAD to some extent, but iterative development in particular is OK as a concept that I wish to remark upon.

The reason I object so heavily to this is partly the last observation from the original comment, later never comes, but mostly because I think this betrays that we as a profession have lost sight of the disciplines of engineering. Consider almost any other engineering discipline that you care to imagine, ship building, tool manufacture, car design, house or office building, computer hardware design even. Can they be characterised by the sort of approaches outlined above, iterative rethinks of core parts of the product (I know lets make the captain's quarters out of marble, marble is very fashionable these days)?

It is unfashionable amongst the /. crowd for example to point out as Fabian Pascal does so effectively that the RDBMS is a (flawed implementation of) a mathematically sound model and that if you wish to effectively use the relational approach you have to understand the model and do the ground work of application of relational design to your specific situation. It seems that it is also becoming fashionable to try and avoid any modelling of what you are trying to achieve at all. Hidden in both the comment and the practical application of the agile manifesto is the fact that the software developer, and probably the client as well, doesn't actually know or understand the problem they are trying to solve or the business process they are trying to automate. Now repeatedly trying something until you find something that works is certainly one approach - it reminds me of my toddler trying all the holes in his posting toy until he finds the hole that the shape goes through - but its surely not as good as knowing what you are trying to build before you start out.


I think I'm being a bit dim here. I don't really know what are saying with this post. At first I thought you were disagreeing with my perspective, that iterative development can be a nightmare, but by the end you sound like you're agreeing with me.

Just to clarify my stance, I like to design before I proceed. I find iterative development often results in a big mess, not because it is a flawed concept, but because lazy people use it as an excuse not to do the job properly.

Is that what you thought I meant? Or have I misunderstood completely ;)


i remember doing things like ER diagrams, and even using CASE tools to lay out the database tables and relationships as part of the design process. and trying to think out all that BEFORE putting the database together.

now it's more like "here's the application, make it work with the database."
yes Tim, I am agreeing with you (well apart from the "not that theres anything wrong with these approaches in principle" comment. I think there is).

you just articulated well the actual state of software development as I see it today. I just go further in that i think that not only can iterative development be a nightmare, when it isn't its by accident rather than design. it enshrines "we don't know what we are doing" as a recommended approach to an engineering task.
I've also removed the "I wish to take issue with... " sentence. it was just badly phrased and misleading as to what i have problems with. In particular it did suggest that I had a problem with what Tim wrote, I don't. Its the attitude he characterises that I have issues with.
I guess if you are going to try the iterative route, the main thing you have to consider is the scope of your decision.

As Jeff points out in his post Process or No Process?, some projects have a small scope and 80% is good enough. Here iterative development is fine.

Others have a very big scope, like multi-phase projects. Making a mistake in the first phase can trash the whole project, like building a house on shakey foundations.

If someone convinced me they had taken the time to understand the issues and were happy to deal with the redesign and refactoring as they went along I wouldn't be so bothered about this approach. The trouble is people all say the right thing until they start adding up the costs of redesign and refactoring then it goes out of the window.

I'm not saying I like doing it, I'm just saying if you do it properly iterative development can work. The trouble is everyone I come across seems to do it wrong because they underestimate the shear volume of rework.

I guess I should just shut up and take the money ;)


PS. Thanks for the alteration to the blog. I don't sound quite so stupid now :)
The advent of the PC allowed directors , managers and users to free themselves from the I.T. department - albeit with hidden costs which were conveniently swept under the carpet .

Now RAD and Iterative design promise to free the masses from the minority who are capable of rigorous design .

Most business requirements have been computerised many times before and we are now on the fourth or fifth generation of "designs" so what possible need could there be for iterative development or even prototyping ?

No doubt the proponents of RAD would cite upfront design as "premature optimisation" .

It saddens me but I suspect that if you want to design something really worthwhile you have to set a company up yourself and take the immense gamble that the majority who forced you to go to these lengths will buy your product .
This post has been removed by a blog administrator.

Here is what Ambler understood from my critique:

Pascal criticism, if you're actually to read it, is inane. He
complains about disagreeing with the requirements, interesting but it
has nothing to do with the actual technique, and complains about how
difficult it is to collect people's emails after the fact if you
don't get the email column in the database for your first
release. The problem domain was a karate school, and I can safely
tell you that it's possible to teach people karate without knowing
their email address. Furthermore, considering how volatile email
addresses are, and considering that many people don't have them, the
quality of the data in that column will always be questionable at best.

Post a Comment