Monday, September 19, 2005

Swiming Up the Waterfall

Sometimes, you write something that you have to keep a lid on for a little while, because you don't want to offend someone who might be reading. This is one of those cases.

A fellow co-worker and I found ourselves in a client site who was a big believer in my most favorite of dysfunctional of methodologies, Waterfall. We were hired to help them design an advanced new system (that, of course, interacted with their still very active Mainframe development) with innovative technology. My coworker and I showed up with great enthusiasm (because we didn't know what was going on yet).

Here is what we were tasked to create: First, we need a Functional Design document (a technical-ish set of requirements) to pass off to the business unit to get approval for the project (which was already approved -- that's why we were there). It took man-weeks of effort to produce this utterly useless document, which was created by emailing Word documents around, because they don't put design documents in version control (What? Did I hear that right? Why not? "We just don't" “No, why?” “If it’s in version control, others can see it”). OK.

Next comes the Technical Design Document, another multi-man-week project, where we take the Functional Design and produce a Technical Design, meaning that we finally get to reveal all the technical decisions we made while writing the Functional Design but couldn't put in that document. And we can't produce "prototypes", because that's a bad word -- business people want to see prototypes run. "Can we call them spikes? Sure, no one knows that that means". A few times, I thought my coworker's head was actually going to explode. We set up our own secret version control system and starting writing our design documents (oh, sorry, Design Documents) using XML so that we could diff them, and converted them to Word documents when the big flurry of Consolidating the Documents happened.

Then we made a decision. Given the amount of time that we had to create the Technical Design Document, we could just write the code, using XP techniques (test-first coding, iterative design, etc.) and produce the design document based on the working, tested code. And that's what we did with part of the system. We kept this under wraps (after all, we're supposed to be designing, not coding) and made good progress. To show the quality of the code, I implemented code coverage using Cobertura so that we could demonstrate that the unit tests were actually exercising the code. With just a couple of days left in the Technical Design phase, we were asked to provide estimtes as to how long it would take to implement our design. That's when we let the cat out of the bag: "Do you want us to tell you how long it would take, or how long we have left? We can give you really good estimates because it's already done". We showed them the implementation and the unit tests, along with code coverage (96% code coverage, 100% branch coverage) . The Technical Design document that was already based on working code.

They weren't upset, but weren't particularly happy either -- just kind of stunned. This way of writing software is so foreign that they have no perspective on it. They've never seen code with unit tests, or code coverage, nor any code that was produced in an iterative way. You are supposed to spend months on design documents, then write the code, right? How is it that you can not do all that? Instead of the elation that we expected, we just got a weak "Good job, guys", and they took their working code and incorporated it where it was supposed to go. We were finished with our tasks, so we left.

But not without planting some seeds. You see, we involved some of their other programmers in what we were doing. We didn't code with them, but explained how we were going about it. And we showed them unit tests (which they had heard of) and code coverage (which was new). The other developers clearly saw the benefit of what we had done. Sometimes, you can't dig a deep hole all at once because you have to move too much dirt. But a river can dig a deep hole a little at a time, eroding away grime and stone bit by bit. Even if we didn't dig a deep hole, maybe we created a rivulet.

6 comments:

John said...

Like many other "Methodologies", Waterfall is really a reporting technique, not a tool that one can follow to do development. The higher up the management ladder you go, the more Waterfall fits the way people think.

I know of no software developement project that has actually followed Waterfall (but many, many that used it as the reporting tequnique). [OK, some of the really big Military software catastrophes may have actuualy come close to following Waterfall].

The art is to follow the way you know works, while you document progress according the the proscribed methodology. Or maybe I'm just getting cynical?

Anonymous said...

How are you generating the .doc formatted documents using XML? Apache FOP?

Anonymous said...

Did you consider not immediately showing the code, and offering instead to do a very rapid code development period, using the time instead to 'test' the code with the local programmers? ie: use the time as a mentoring program for their people? It's possible doing this would have made them more involved in the process and allowed them to really take pleasure in the amazingly fast development time - and possibly onsell future business for you with the firm?

Neal Ford said...

I'm using DocBook => WordML, which Word likes

We discussed several different options on when to best "spring" our subversive agility on them, and decided to do it when we did. I've left out lots of political stuff that surrounded this to protect the...innocent.

Anonymous said...

Neal,

I'm always a little wary about introducing (process) change by stealth as you have described. Whilst it can be extremely successful in the short term, it can be viewed as abuse of trust by the customer, something which can have far longer reaching political ramifications.

Have you seen any other fallout (beyond the muted thanks at the end of your tenure) on that front?

Neal Ford said...

No, actually, our immediate supervisors eventually came around and understood the implications of what we had done. But, introducing proceess change above their head was even more daunting (we were working with the Java group, and they are peers with the Mainframe group).

It was a risk, no doubt, but it was really the only way we could provide reasonable value. The analogy I used: they had hired cario-vascular surgeons to hack off limbs. At the end of the day, we delicately made out point. In fact, other development groups are working on engaging us to apply Agility more directly, if they can get upper management buy-in.