Iterations, Batches and Flow

I've recently had a lot of conversations with people who are thinking of iterations as batches. By definition that's what they are. You sign up for a "batch" of stories, at the beginning of the iteration, and you deliver them by the end of that iteration. However, it is useful to think of requirements (stories) as units of work that flow through the value stream. Here is why I think so.

What is a batch?

"A batch is a group of items that will be processed at once as a single unit"

Items are usually batched to achieve "economies of scale". This usually means that there are some overhead activities associated with the items in question and it is better to distribute the cost of such activities over a batch of items than to perform such activities for each individual item. A good example is transportation. It is always more economical to transport a whole container full of items than to transport individual items.


You can batch items up at various levels. Coming back to the software world, you could batch requirements (stories) at an iteration level, at a release level or even at a project level if you must. That's what classic waterfall does, it batches requirements at the project level. But the point is, you need to work on the whole batch at once at each step of the process. Which means you'd prioritize the whole batch, analyse the whole batch, develop the whole batch, test the whole batch and then deploy the whole batch.

So what's the big problem?

Well, by batching items, you are limiting what you can do with each of them individually because as we discussed you have to process a whole batch at once. So where you could have prioritized, analysed, developed, tested and deployed individual requirements (stories), you can now prioritize, analyze, develop, test and deploy only iterations, releases or entire projects.

Flow

Instead, think about requirements (stories) continuously flowing through the value stream. This way you can assess individual requirements and make sure that only the most valuable make it into the value stream. Then you can analyze, develop, test and deploy these faster because they are small. Now all you need to do is take stock of how your team is doing at regular intervals. This is the time when you would showcase to your client stakeholders, check your throughput for the given time (velocity OR number of stories), have a retrospective, brainstorm technical approach, assess technical debt, etc.

And what about those "economies of scale"?

You do lose out on them to some extent. But here are two points to make you feel better because economies of scale are overrated anyways (at least in context of software development).


Firstly, remember that the more you do of something, the better you get at it. So by doing small parts of the overhead activities, you get better and faster at those activities and they don't remain such a big cost anymore. For example, take deployment as an overhead activity. The whole process of deploying to production is usually so tedious and time-consuming that you prefer to do it only when you have a sizable "batch" of features to be deployed. But if you start deploying each story to production, you will have to better that tedious process. You will have to automate most of the deployment. You will have to automate most of the testing. After 10-15 stories are deployed this way, the process will not remain tedious anymore.


Secondly, stop thinking of it as extra cost and start thinking about is as the price the business pays for being able to delay decisions and for getting the opportunity to course-correct.

Example:

Let's say you were developing a feature which was split in 10 stories. After deploying the first 3, which formed the minimum marketable feature (MMF), you saw that your users are not using the feature as expected. You either need to change the approach for the remaining 7 stories OR you need to stop working on that feature altogether because, given the users' response, other features have become more valuable.

As business, you would have saved some time & money if you had waited for all the 10 stories to be finished and then deployed. But those savings are insignificant compared to first hand user feedback about the feature and the chance to act based on that information.

As the development team, don't feel bad because your efforts on the first 3 stories were wasted. They weren't. You invested that time and effort to find out whether that feature works for your business. You already got more than what you bargained for.

So what about iterations?

Have them by all means. But don't worry about hangover*. Don't try to sign up and finish a particular set of stories in the same iteration. If you can well and good. But what you are really aiming for is to perform consistent amount of work every iteration.

*So as long as your hangover remains constant you are good. If its increasing, you need to change your approach. For example by signing up for lesser stories.

Where do RFPs come from?

Once upon a time in a big corporation...

Mr. X (you know? the one who does all the work) : Aaaaaargh! I am fed up of watching this screen 24x7 just to let Mr. Y (the boss) know if I ever see a red dot in the top right corner 

Mr. Z (co-worker) : Hey we can probably get some software to do that. You should talk to Mr. Y (the boss) about it. 

Mr. X (to himself) : Sure! that'll get me fired.

So Mr. X doesn't talk to the boss(Mr. Y). But Mr. Z plots an evil plot to get Mr. X out of the way. He tells Mr. Y that they should get software to replace Mr. X.

The Mr. Y loves the idea.
He makes a presentation to his boss...
who presents it to his boss (as if it was his own idea)...
who arranges for a meeting with Mr. P (a guy he knows in the IT department)...
who rants for the full duration of the meeting about how he doesn't have any time / resources to build new software.

Mr. P then goes and whines about it to his boss (Mr. Q)
who says he has some money left from this years budget that needs to be spent. "Let's get the software built by a vendor" he says.

Mr. Q talks to his boss
who tells him he should talk to Purchase department
who tells him he should talk to Legal first to write a request for proposal
who tell them he will need to hire 2 college grads and a really sadistic lawyer to create an incomprehensible document about what needs to be done. He spends all his budget on the college grads and sadistic lawyer and gets a document that totally impresses the Purchase department who awards him with the "King of Kickass RFPs" award.

Purchase then takes the RFP document and sits/sleeps/(i-dont-want-to-know)s on it for the rest of the year because there is no budget anyway. Then new money arrives in the next year and Purchase floats the RFP.

It goes to vendors who start acting like they actually understand the whole business and asking difficult questions. Purchase gets scared and calls Mr. Q. However, Mr. Q has left the organization to go and join another one as "Chief RFPist" after his award winning RFP last year. Mr. R has taken his job. He doesn't know head or tail of the RFP in question.

He talks to his boss who says "Oh! Yeah I remember! So and so had talked to Mr. P last year about replacing his subordinate's subordinate's subordinate's subordinate with a software." Have him talk to the vendors. But Mr. X has already left the agonizing job, Mr. Y is on vacation and Mr. Z is too busy staring at the screen looking for a red dot in the top right corner to be able to get on a call with the vendors.

So Mr. P talks to the vendors and gives the "requirements". The vendors write more bullshit in their "Response to the Request for Proposal". They too have a couple of grads and a smart ass legal guy. Mr. P then takes all the responses to the purchase and legal department for their recommendation for the cheapest quote with the maximum loopholes and awards the project to the most "suitable" vendor.

And then they say 70% of IT projects fail.

When in a inception...

Everyone is an analyst
Don't get bound by role boundaries. Everyone needs to understand the system. Talk to the clients, ask questions, draw diagrams, make suggestions, understand problems and solve them.

Make sure everyone facilitates sessions at least once
Especially the BAs since they have to interact the most with the clients. Clients need to feel confident about the BAs on the team. Other members should also be actively involved. Don't let a single person be the scribe all the time. The client might ask "why am I paying for this guy" at some point.

Have an intent for each session
and strive to achieve it. Have back up plans ready in case you find that your planned tools / techniques are not working for the client. Fail fast, change your approach, take a break and regroup, but make sure you get the right information out of a session.

Educate the client about the way you work
They should know what it would be like to work with you day-to-day. Clarify time commitments and kind of inputs required from your clients (suggestions during analysis of stories, showcase feedback, testing). Have special sessions that cover methodology. Offline the subject if it comes up elsewhere. It can derail your session plans easily.

Be honest and upfront
About any shortcuts you have taken, any information that you don't have, any help that you need.

Understand each stakeholder
You should know the pressures that they are dealing with, individually. Know how you can help them.

Create. Shared. Vision.

Livescribe Pulse Smartpen

Ooooh! Ooooooh! I love it!!

This is the most awesomest thing I have bought this year.

There are enough reviews online for me not to write one of my own. But I'll mention the most useful features here:

• Captures everything I write / draw. This is the basic promise. A very well kept one.


• Captures good audio. It's not crystal clear or anything, but is it works well in a meeting setting. I recorded three interviews today, alongwith notes. After I get the headset that someone took by mistake, I'll experiment with the more clear 3D recording capabilities.


• It links the audio and notes together so you can replay what you were saying when you wrote a particular word.


• It searches. Even cursive handwriting.


• And then there's other fun stuff like drawing a paper piano and playing it. Or using a calculator by tapping the paper calculator on the inside cover of the starter notebook. Or drawing a cross for navigating the pen's main menus. Or bookmarking audio at specific points.


• It uses dot paper for all the magic. But you are not restricted to the notebooks supplied by the company. You can print your own dot paper for free using any half decent printer!
  

All in all this is going to be very useful. The 2GB storage is more than enough. My 1:02:14 audio session used 12MB. So if you download the data even once a week, you'll be alright.

I'm loving it!

Commitments

I facilitated immersion for the March 2010 batch alongwith Arun. On the first day we went through the ThoughtWorks values, culture and history. Then had a session on feedback. We value feedback a lot in ThoughtWorks because the way we deliver is based heavily on Constant Feedback and Continuous Improvement (thanks Sarah). One the second day we went through a brief description of the ThoughtWorks way of running projects (based on agile / lean principles). On the third day we were talking about iterations and about the problems with velocity becoming a target rather than remaining as a basis for planning. I was saying that when we sign up for stories, we don't commit to the points signed up for. It is just a guess based on some information (yesterday's weather) and some gut feel. At this point someone asked the question.

When do we commit? What do you commit to? The client can't just trust you... there needs to be some concrete commitment.

I didn't realize how important this question was when it was asked. But thinking about it I see that it is the basis of everything we* do isn't it? The basic agile principle of

Customer Collaboration over Contract Negotiation

The success of a software project does not depend on its completion on time or within budget. It depends on what benefit you get out of it.

Number Lust

I am an amateur photographer while I am not not hacking project teams and building custom software at ThoughtWorks. Photographers are known to have bouts of lens lust* time and again, especially at the beginning. I realize that some managers seem to have similar urges when it comes to numbers, metrics. They suffer from acute number lust.

In ThoughtWorks, we believe in and encourage self organizing teams. BAs gather requirements, developers write code, QAs test and automate and the customer signs stories off in a flow. The Project Manager role is therefore reduced to making sure that nothing obstructs this flow.

The way this would be done is to gather and analyze the right data and take actions based on this analysis. Examples being:

• Finding bottlenecks from the wall or a CFD and taking actions to fix the problems.
• Looking critically at retrospective action items and seeing that all those issues are fixed (especially if they are issues outside the teams control, like infrastructure).
• Making sure that the right capabilities exist in the team at the right time. If not, get people from outside, arrange training, etc.

Some PMs are content in gathering just enough data to spot problems. But then there are some who can't get enough data ever.

They want to organise the whole world into neat boxes and label them and track anything and everything possible. The walls soon fill up with useless charts with graphs and numbers that the team can't use in any way. The worst part is, there is no analysis done around the data even by the people who gather it. Having people in a 15 member team draw emotional seismographs every iteration and sticking them on the wall is of no use if you are not going to make any decisions based on them.

Now I agree that gathering all possible information can enlighten us to some extent. But its a matter of marginal utility. If I have to do 5 extra things to do my job, just because we want to gather data, I am not doing it.

So guys, here's a humble request. Figure out with your client what kind of data he would like to see. Figure out as a team what you would like to improve and what data needs to be gathered to help you get there. Don't burden your team with things that obstruct OR slow down the flow. Keep it simple and keep it lean.

Our job is to provide quality software not elaborate reports.

* lens lust - the dangerous urge to keep buying new, expensive lenses not realizing that its practice that will improve your photography, not lenses!!

Onsite Business Analyst

This has come up in various discussions recently and I want to put down my thoughts about the role and responsibilities.

For a ThoughtWorks team in India (or China) most of the work is offshore agile development. Clients are usually in the UK or the USA. The team is structured as follows:

1. Offshore PM
2. Offshore Devs
3. Offshore QA(s)
4. Offshore BA(s)
5. Onsite BA(s)

This is how the communication works.

Of course there are other exchanges that take place but the Business Q & A and the Technical Q & A are the most important pieces of concrete information exchanged.

Other companies, even traditional development outfits, have a role called the Onsite co-ordinator. I believe this role facilitates similar discussions (although I hear that they are more technical than business related)

The onsite business analyst's main role is to facilitate business and technical discussions between the team sitting offshore and the client. She is not restricted to be just an onsite co-ordinator. She can take up a new stream to analyze by herself. Even feed it to an onsite development team, if there is one. But all this keeping in mind that the main purpose is bridging the communication gap between the client and the development team.

At the same time the onsite business analyst shouldn't go overboard with this and become a single point of failure. The direct communication lines between the offshore team and client should always be open.