Metrics

I have recently come out of a consulting assignment which has given me loads of time to read and think about processes, improvements and effectiveness. (People who follow me on google reader must have noticed). It also had me thinking about introduction of agile into a traditional IT outfit and what would make it more effective.

Top -> Down

The Top -> Down approach is where someone in the top management realizes (or is convinced) that agile is the solution to all their problems and goes on to "mandate" agile. This is not necessarily bad. If the organization hires the right consultants to bring agility into the project teams at the ground level.

Bottom -> Up

The Bottom -> Up is not really an approach. Its just a team realizing that they can do their stuff better and try to bring in improvements in their team without affecting the organization. This is later, agility may spread into other teams virally and this might trigger an organizational transformation with eventual buy-in from the management.

Here's what I think works best. Like any win-win solution, its a mix of the two extreme approaches described above.

Top -> Down buy-in, Bottom-Up implementation

The management decides, that they require to be agile to meet the market conditions but instead of pushing "agile" down the throats of people below, they let the people on the ground level realize what agile means and how it changes the way the team works. Agile Coaches are useful in this transition.

Sounds simple enough? Well here's the catch. How does the management know that the new process is effective?

The sad part is in a traditional organization's hierarchy, the only interface between the management and the teams is "Reports". Reports on various parameters (metrics) that the management believe should be monitored to see how stuff is going on. Whether the teams embrace agile themselves OR whether management asks teams to be agile, the teams will need to report the right metrics to the management to gain their confidence. If the teams don't do this, the management might start tracking the wrong metrics and hence encourage wrong practices. What you measure is what you get.

I opened this topic to a bunch of people from the client organization and we came up with a bunch of metrics based on the principle that we should report consistency rather than hard numbers. Consistency gives a measure of effectiveness and also encourages the right activities.

Here's the list that we came up with (There can be many more. The basic principle is to use proportions / percentages instead of hard numbers):

1. Velocity Barometer without numbers (we mark completed stories in red on a barometer, but we take out the numbers. Consistent color means good, ups and downs means trouble)
2. Cumulative Flow Diagram (consistent thickness of bands v/s actual numbers)
3. Ratio of open V/s closed bugs (Percentage. Instead of bugs found and bugs fixed)
4. Test Coverage (Percentage)
5. Automated V/s Manual tests (Percentage)

Really speaking it all boils down to using just Control Charts for project reporting and management. What do you think?

Cumulative Flow Diagrams

Once you have your wall in place, it’s time to start monitoring your progress through the project. Best done on a daily basis and best done with a “Finger Chart”. Here’s an example.

A finger chart is basically your wall turned sideways so that your swim lanes are horizontal rather than vertical. Now what you track is the number of stories in each state (finger).

This chart may not make sense to you immediately but there’s a lot of useful data hidden in here. Here are the different inferences you can make by looking at a finger chart.



1) Bottlenecks
The first thing that you monitor on a finger chart is bottlenecks. This is easy. Just check for a band thickening consistently. A band thickening means that there is either a capacity problem or a capability problem in the next band. For example, the BA/QA band consistently thickens a suddenly rises at specific intervals. This is because the Client is signing off the stories only at the end of each week. This may be due to various reasons which need to be investigated to see if something is wrong OR can be made better. The ultimate goal is to achieve a smooth graph.



2) Work in progress
At any given point if you measure the total across all bands vertically, it will give you the total work in progress. This is a total of everything that needs attention from someone or the other on the team. Always try to keep this as low as possible. You should ignore the Celebration State for this calculation.

3) Average Turnaround Time
If you measure the time across all bands horizontally, you get the turn around time for a story. This is a good measure of your entire process. How lean your processes are, how good the team is, how involved and keen the client is, etc. This is what tells you how much time your team needs on an average to turn an idea into working software.

Remember that the finger chart is a cumulative of all stories. Therefore the top line of the graph should never dip down unless you have really reduced scope.

Now whether to track Number of Stories OR Story Points on a finger chart is an interesting debate. I will try to present my ideas on that in another post.

UPDATE : As David pointed out these are called Cumulative Flow Diagrams in the Lean world. Finger Charts is/was more of a ThoughtWorks term.