Agile: Where is the Line in the Sand?

The more I read on agile practices the more I hear some fairly edgy ideas about requirements gathering, design, and architecture in the agile world. While I understand the concept of embracing change, there has to be a line in the sand somewhere of how your company performs its business and how you build software to enable that business. As I work with clients adopting Scrum I find different levels of the YAGNI (Your Ain't Gonna Need It) principle and the idea of building extensible software that gracefully accepts change. My struggle is that I need to find a happy medium with the majority of my clients. Martin Fowler's article "Is Design Dead?" makes an argument for something in between also, but he focuses more with contrasting against the more edgy concepts in XP. Unfortunately a large number of agile pundits have taken up the idea that design up front is a wasteful practice as is gathering any type of detailed requirements. These are not just the inexperienced newbies blogging from their soap boxes, but well respected and experienced thought leaders in our industry.

So is up front requirements gathering and design really dead?

I have never been a agile purist and I think that for my clients some hybrid is more prudent. In the past I have been on Scrum teams that designed on the fly and kept the YAGNI mantra. Sometimes this resulted in some patchwork architectures and some disjointed business themes in the application. While I do think it is a great idea to embrace change, a basic vision for both your business case and architecture/design of any software solution are needed to keep the smaller iterations on at least some common path. This vision should be able to change as well, but there are some fundamental things that most probably will ring true for the solution. Determining that line is a black art. Too little and you risk being all over the place in your deliverables. Too much and you start becoming too rigid and have lost the benefits of an agile approach.

So how do you know how much up front work to do?

If I had a textbook answer I would give it to you. And it would most likely be wrong. Every case is different. Each client's situation and environment is different. As a company's business evolves the line might have to move also. But I do think that forming a basic business case as the initial vision and some high level architecture and design is almost always warranted. Even Ken Schwaber, one of the originators of Scrum, advises to start as soon as you have enough backlog items to start a sprint. You do not need grand functional specifications and loads of UML diagrams. As is often quoted "The plan is useless, but the planning is essential." When determining if you need to detail a specific business case ask yourself how essential is it to our core business? Higher value requirements always get the most attention. As far as architecture/design, defining the basic components and their interactions at a high level should be enough to ensure your deliverables follow a cohesive path. When determining how extensible a given component should be, concentrate of the most probable changes rather than inventing all sorts of possible future scenarios.

So how can you gather requirements and perform up front design while still being agile?

If you are going to start a sprint with just enough product backlogs items then some of those product backlog items should be forming high level business cases and prototypes so that subsequent sprints will have the benefit of an uniform vision on both fronts. I also believe that you do need some level of documentation for these outside of just the backlogs. A wiki is a great place for them since by its very nature embraces change (and the good ones can provide history for those people who just cannot get over the idea of this type of change). To take that idea to the next step, Team Foundation server allows you to add links to web pages on backlog items so that you can tie the documentation to the user functionality and development tasks. If you want to go low-tech use flip charts and save all the paper. Post it on the walls of the room where the team is literally surrounded by the business and architectural vision.

Even if you are well into an agile project, you can always have the stakeholders redefine the vision if you feel you are straying too far because of only thinking about the next sprint. Change is inevitable, but your company's core business fundamentally changing in not very probable. And good design should elegantly accept change also. I do not think up front gathering of requirements or design is dead. I just think that in agile environments it has evolved and shed its more useless features.

Comfortably Scrum: Is your company ready to be agile?

As I have posted in previous blog posts, adoption of Scrum (or any agile process) cannot be a development only effort. If done right, Scrum will have an effect on almost every department in your company that is involved in developing software. And if your company (or even just key parts of it) do not get on board, it can make for a painful process. So if you are thinking about possibly adopting Scrum, ask yourself the following questions to see how open you company might be:

1. Do key members of management buy-in to the agile concept? They do not need to understand it in its entirety, but they need to be open to the new (and sometimes radical) ideas it brings.
2. Do the other departments show interest in the agile concept? I have seen the most push back from QA and BA departments when adopting Scrum. Some of them see it as the developers wanting to "take over" some of their territory. It is sometimes a hard task to convince them this is is not taking over anything but sharing the effort across all departments as part of a cross functional team.
3. Does the nature of your business mandate processes that are counter to Scrum's? Many agile purists claim there is not such thing, but there are simply some businesses that would have a difficult time adopting Scrum because their way of doing business demands an approach in direct conflict with some of its ideals. Strict deadlines that cannot move, regulations that require comprehensive documentation, or a level of precision that requires a large amount of up front requirements gathering and system design are not impossible in Scrum, but might cause you to adapt it to the point of loosing the true benefits of the process.
4. Do you have a team of disciplined developers? This is something I talked about in another blog post. Scrum (and agile in general) is misconstrued as allowing developers to do whatever they feel like. In fact it is the exact opposite. Many developers have struggled coming from very structured shops that gave them a process to hide in and basically told them what to do. Scrum requires a developer to be cross functional and the processes themselves are meant to expose inefficiencies.
5. Does your infrastructure allow for cross functional teams? The best way to work in a Scrum team is to have them physically located together. I have worked in both situations where everyone was in the same room or a short distance away to having half the team in two other states. If you cannot be together physically, you will need some mechanisms to bridge the gap and allow easy collaboration between team members. Everyone knows I am a big TFS fan and I personally would not manage a diversely located team without it. Readily available conference calls, Live Meetings (WebEx or whatever), and instant messaging are a must for every member of the team. This does not just mean developers. BAs, QA resources, IT staff, and the users/stakeholders should all be as easy to collaborate with as the developer sitting next to me.

These are by far not all the questions to ask yourself, but they are some big ones that get the conversation started. I have heard agile experts even advise adopting agile practices without management knowing and slowly work up to involving them. In my experience that is a huge mistake. Moving to Scrum or any other agile methodology should be something the entire company works together on to make a clear and unanimous choice. There are definitely companies out there that are not yeat ready to adopt a process like Scrum and may never be able to. Scrum like any other software development process is not a silver bullet and is not meant for everyone.

Coming up next for the Comfortably Scrum series: Product Backlog Items.

Good book on Agile development

This book is a few years old, but I just got around to reading it. Practices of an Agile Developer by Andy Hunt and Venkat Subramaniam is a good, practical introduction to the daily activities and approaches software developer's can adopt to become more "agile". A good bit of the items mentioned are not even necessarily only applicable to agile development. Quite a few of the agile books I have read stick to a good deal of rhetoric and little in practical implementation. This book is part of the "Pragmatic Programmers" series and does a good job of cutting thru the theoretical hype and giving some solid advice. It was a very easy read because of the way the book is laid out. Each major section has small sub sections that tackle one specific idea in about 3 to 5 pages. It was easy to sit down for five minutes and digest one facet from the book. The book does have an angel and devil motif that is a bit cheezy, but even this provided some practical context for each section. The Nashville Agile User Group has this book as a staple in their meeting prizes (that is where I got mine). I give it a thumbs up and suggest it as an easy read with some practical advice on how to become a more agile developer.