The Nature of Software Developers

I once read a book, which shall remain nameless, that seemed to have a quota of illustrations per page: an average of one-third of each page covered by an illustration.  It was a horrible read, made all the worse when I realized that the figures were often repeated to illustrate different concepts with only the captions changed.

Happily, although Ron Jeffries’ own illustrations manage to cover perhaps an average of a fifth of each page in his new book, The Nature of Software Development, they actually add value rather than just taking up space.  Take the illustrations that Ron uses to introduce the idea of incremental software delivery.  It’s very easy to see how people envision their final product in terms of the magic they believe it will be.

The very next illustration, though, helps us recast our thinking in terms of incremental delivery and how it helps us start deriving value from the product we’re building and then another illustration shows how we can use the feedback we get from the earlier releases to produce something better than we’d originally imagined.  Most of the time I honestly ignore illustrations and diagrams as useless, but Ron’s illustrations are generally thought provoking and I always found myself stopping to think about how they illuminated the text around them.

The first part of Nature covers “The Circle of Value,” understanding what value is, why we should try to deliver value incrementally and how we can build our product incrementally.  The second part of Nature is entitled simply “Notes and Essays” and provides more detailed thoughts on some of the subjects touched on in the first part.  One of my favorites was “Creating Teams That Thrive” where Ron reminds us that when the Product Champion, the term Ron is now favoring over Product Owner, brings defined solutions to the team they are less likely to feel a sense of responsibility and pride in the result.

Another nice essay was “Whip the Ponies Harder,” where Ron reminds us that trying to pressure a team into delivering faster can have deleterious effects.  But this brings up a point I wanted to make about the book itself rather than what it says: If you’ve followed along with what Ron has been thinking over the years, in the various discussion groups in which he participates, on xprogramming.com/ronjeffries.com or at the various presentations and classes he’s done, there may not be anything new for you in Nature.  Even if you’re in that situation, it’s probably worth getting the book anyway.  There’s always the possibility that there will be new thoughts for you there, and, even if the ideas themselves aren’t new to you, their presentation in Nature can spur (sorry, the horse/unicorn drawings may be doing something to my language) you to think about them more deeply and also give you a new way to explain those ideas to other people.

A final word of warning: Early on Ron says “[Y]our job is to think a lot, while I write very little.”  This reminded of the time when someone told me they’d read XP Explained (the first edition) in a weekend and understood it all.  After fifteen years, I’m still deepening my understanding of XP, mostly through trying to introduce its values to development groups that don’t necessarily understand what, if any, values they hold.  Even though it’s a short book, give yourself time to really think about what is said in Nature, even after you’ve finished reading it.  That’s when the most reward will come.

If you’ve been following this series of blog posts about why so many Agile projects seem to deteriorate into waterfall, you know that I believe failure to completely close iterations (or sprints) is a major reason why. If tasks continually spill over from one iteration into the next, the system is never stable enough to demo, and without demos, the feedback loop between the developer team and the Product Owner is broken. Without a rapid feedback loop, Agile doesn’t exist. The project is just a waterfall project with weekly or biweekly status meetings.

What can developer teams do to ensure that iterations close with functioning software that allows the feedback loop to run smoothly?
The most important thing you as a developer can do is to embrace change, as the motto of the Extreme Programming (XP) movement says. Don’t accept change. Don’t tolerate change. Embrace it. Understand that change is the very basis of the evolutionary and incremental approach to software development that is at the heart of Agile methodologies.

What does this mean?
Well, consider that most useful business functionality in an application takes a long time to develop. A single complete use case may take a month or more. And as you consider all the use cases in the system, you begin to see patterns emerging. Perhaps use case 1 requires a way to validate and persist complex incoming data to a database. Use case 20 requires a way to report invalid data to a compliance authority. Use case 35 requires you to deal with the response from the compliance authority. So you begin to think about how you can avoid having to rework what you do for use case 1 to incorporate the later use cases.

This is the road to perdition.

If you want all these use cases to “come together” all at once into a grand spectacle of software delivery, you’ll find it increasingly difficult to show progress along the way. There are too many interlocking pieces that all need to work in order for the whole thing to work. This is how work spills over from one iteration to the next – because the chunks of work you’re taking on are too big.

Instead, embrace change.

Know that some of what you do in use case 1 will likely be reworked for use case 20. This is a good thing, not a bad thing – because it allows you to chunk up your work into smaller pieces that are functional but with limited feature sets.

You may ask: but doesn’t the cost of the rework add up and inflate the project cost?
Yes, a little, but if you diligently use Agile engineering techniques like Test Driven Development (TDD), Continuous Integration (CI), and so on, the cost of change will be greatly reduced, and ultimately the cost will be less than the cost of eliminating the feedback loop. Think of it this way: if you incrementally develop and constantly demo use case 1, the Product Owner may discover that use case 20 needs modification, and that use case 35 goes away. Now you’ve actually reduced cost by using an evolutionary approach. Eliminating the short feedback loop is one of the most expensive things you can do on a software delivery project.

As a member of a developer team, take the iteration planning meetings (IPMs) very seriously. Your focus should be on working with the Product Owner to break down use cases into user stories and tasks that you and your team can complete in an iteration. Keep repeating to yourself: fully functional but not necessarily fully featured. Accept that the feature set is going to grow and evolve, but always keep the system functional so that whatever limited feature set is implemented can be demoed and incorporated into the feedback loop.

For example…
So, let’s take the example of use case 1 above: Validate and persist complex incoming data into our database. Will this fit into a 2-week iteration? Most probably not. So start small. Let’s get some valid data into the database. This needs a couple of tables and some data persistence objects. Will that fit into 2 weeks? Yes, probably, with some time left over. OK, so let’s use the time left over to do 5 of the 120 total validations we will have to do. Decouple the validation from the persistence, because that makes it easier to do each piece – but it’s better software design anyway. How do we report the validation errors with no UI yet? How about writing them to a file? Sure, you’ll end up throwing away the file-writing code, but once again, you’re practicing better software design. You’ve separated performing the validations from reporting the validations.  And you can immediately start demoing your validations to the Product Owner.

(Hint: don’t simply write directly from the validators to a file. Design a validation reporting interface. Implement the interface first as a file, and later as whatever it actually needs to be. Use mock testing frameworks to test-drive the design. This decoupling and abstraction is good design whether you’re using Agile or not.)

Later, you will hook up the validations to the persistence. Then you will deal with how to handle invalid data. Then you will deal with showing validation errors via a UI. All the while, you have full functionality but not necessarily full feature sets. Each piece you develop is small, well-tested, and isolated. You then begin to interconnect them (don’t forget integration testing!) to deliver full feature sets.

[My colleague Robert Pantall wrote a blog post on how he used this technique of accepting controlled rework and incremental discovery to rewire his living room. It’s a fun read.]

Fully Functional, Not Necessarily Fully Featured
Keep in mind that you don’t get to unilaterally decide how the use cases get broken down into small chunks of functionality. You work with the Product Owner to do this, so that each small piece serves some business purpose. It’s a back and forth negotiation between you and the Product Owner.

Everything you do in an IPM should support this goal of evolutionary development that fits into the iteration. Be prepared to estimate quickly so you know whether work will fit. Don’t be afraid to say that work won’t fit and needs to be even more finely chunked down. Speak up. Have a dialog. Work with the product team. Keep the feedback loop running continuously.

After the IPM, when you’re working on the stories or tasks, you may run into unexpected trouble that lengthens the story completion time. Surface this immediately to the product team. You may need to break work down even more. That’s fine. Keep reminding yourself: fully functional, not necessarily fully featured.

I do want to emphasize something at this point. An evolutionary and incremental approach to software development is not a license to hack and slash. Saying you’re using incremental development does not give you an excuse to abandon architectural vision, best practices for design, continuous code improvement by refactoring, or coding standards. You must still design a system for its intended lifetime, following enterprise architecture best practices. Adopting incremental development only lets you make some short-term compromises for the benefit of keeping the software continuously functional. As the software evolves, it must evolve towards your architectural vision and continue to maintain its design and code integrity. As I showed above in the example, if you do this correctly, you’ll mostly be writing throw-away implementations to well-designed, long-term interfaces, and later building permanent production-ready implementations of the same interfaces.

I hope I’ve shown you that change is your friend if you truly adopt incremental delivery. It isn’t something to be feared or managed or avoided. Instead, it’s what makes it possible to develop complex business functionality while at the same time allowing the Product Owner to touch and use the software and to offer continuous feedback. Properly managed through correct Agile engineering techniques, the cost of the required rework fades into insignificance compared to the cost savings of the continuous feedback loop.

Don’t fear change. Embrace it.

Next episode, I’ll focus on what the Scrum Master or Project Manager can do to keep Agile projects from descending into waterfall.

As discussed in the first post in this series Agile in the Federal Government – Going, Going, Gone Beyond Scrum, the history of Agile adoption in industry took the path of usage of Extreme Programming (XP) followed by the combination of Scrum and XP. In the government, the adoption of Scrum alone has become the de facto method of Agile adoption with the use of the term Agile and Scrum even becoming synonymous in some agencies. In our latest white paper we discuss why we believe the use of Agile Engineering Practices such as XP is crucial to the success of Scrum projects for the Federal government and that the government can again benefit by combining Scrum with XP as is already in use by industry. We believe this is because Agile Engineering Practices enhance the empirical process control (Inspection, Adaptation, and Transparency) of Scrum.

According to the latest State of Agile Survey by VersionOne, the top three XP techniques in practice are Continuous Integration, Test-First Programming, and Shared Code. What XP practices are you using at your agency? Is the use of them improving quality?