Topic: What Are Agile Metrics?
Date: 27 June 2003
Convener: John Major, Cimarron Software
Email address: jmajor(at)cimsoft(dot)com

Attendees:
Libby Wright Texas Instruments
Robert Volker Captiva Software
Denise Phillips IMS
Ignacio Naval United Nations
Kashif Qayyum Golden State U.
CY Chen Medtronic
Rashmi Jain Stevens Institute of Technology
(and later joined by Zhon Johansen of Symantec, and a number of others - please enter your info!

---

Discussion Summary

We had a great time going over our wide experiences with metrics and issues of agility, but did not come to earth-shaking conclusions about just what constitutes an agile metric. We came away with some good lessons, and a number of practical, "ready to try on Monday" ideas. Some key concepts that came out include:

• "Follow trends, not numbers"
• "Get metrics tools into developers' hands"
• "Agility Measure - how are you coping with change?"
• "Measuring your technical debt"

---

"Some Metrics" Brainstorm

• Bug Count
• Estimates (time/function)
• Velocity
• KLOC
• Test KLOC/Total KLOC
• Bugs/KLOC
• Complexity
• Satisfaction
• Bugs/Testing Hour (indicates stabilization)
• Burn-down charts (Progress? This is used within an iteration)
• Effort pts done/pts in the plan
• Effort left over time
• Length of "test cycle" - time between last feature and release to customer (see Kent Beck's paper "Software in Progress" - URL????)
• Build Time (and Test Run Time - needs to be fast!)

Characteristics of an Agile Metric

• Easy to collect - see Tools! "One button"
• Somebody wants to use it! Probe - stop producing it, and see who complains... Story about 500 metrics reports, the system broke, and when they looked, they noticed that 200 were never looked at!
• Minimal - "maximizing the amount of work not done"
• Good for conversation:
• People talk about what they've learned.
• Looking beyond a single project.
• Starts to help with benchmarking - across projects, companies.
• "Must *feel* agile"?!?
• Feedback is rapid.

Some Candidates for Agile Metrics

"Lines of Code"

• This is simple and quick to find. But only look at them to see them trending, along with lines of test code. They should track each other, and flatten out - the flattening out means that we are paying back "technical debt", removing complexity.
• Bugs cluster in "error-prone" modules - so use a metric to look for these.

"Agility Measure"

• Related to "requirements churn", new stories.
• But agility is supposed to take changes in stride - bring a story in, leave a story out - there's no "scope creep" - so how do we measure this?

"Complexity"

• Take a look at the tool Sourcemonitor, for watching code quality.
• This is measuring your "technical debt" - what is your "rate of refactoring"?
• Static analyzers should help! Symantec writes code to do this...
• This metric affects your "Agility Measure" - how well are you coping with change?

"Pulling Risk Forward"

• How well are you addressing risks early?
• Build a risk number, then plot the risk # over time - what is the change rate? Where is it heading?

"Burn-down Risk Chart"

• "Burn-down" is from SCRUM - this might show your trends well.

"Rate of Innovation"

• This is related somehow to the "Agility Measure" (story about a developer coming up with a completely new kind of UI in the middle of the project, which delighted the customer).
• This is a measure of new business value entering the project. How about "Number of patents"? "I've done patents, and the process doesn't 'feel agile' ;-)"
• "Rate of innovation" is a hot topic for general business management, and drives the discussion there.

Observations

• Metrics as conversation pieces.
• What helps the team?
• What does management want?
• "Where are we today"?
• "Where are the loose ends"?
• What kinds of metrics would prove to you that we are succeeding?
• Note the difference between an "Agile Metric" (one that is useful in Agile Development) and an "Agility Metric" - something that measures how agile you are.
• You get behavior when you measure things - metrics have to balance each other (Dilbert: "Pay programmers for each bug they fix" => DOH!)
• Metrics need "selling/explaining" - train folks to read/understand them - what is the story they tell?
• Comparing? Doesn't work well to compare absolute numbers, but comparing *shapes* (i.e. trends) works. That's what happens in the financial community, which has a mature sense of typical industry shapes.
• People may/may not publish their metrics (competitive advantage). But consortiums might be able to collect this stuff anonymously (see the financial business, again).
• Are metrics the only thing that matters to management? Good ones are interested in how well the teams are doing...
• You meed metrics to back up a bid. These would help with forging an Agile Contract, so that it is not perceived as "Time and Materials".
• You can "guarantee satisfaction" (see Mary Poppendieck on this).
• "I collect metrics on our code because I'm interested in *what works*."
• "Almost always, the bugs are in the code that doesn't have unit test code."
• "Between unit and acceptance tests, we have about 70% coverage."
• Driving the process? We posted who did the most Pairing - the team didn't like that.
• KLOC of tests/KLOC - track the trend - should flatten out because we are removing code. This metric is Agile - continuous feedback, the two numbers track each other continuously.
• "If I make a change, I want only 1 test to break - in the beginning, it would be 30 - yuck!"
• "I refactor out test code as well - sometimes it's redundant."
• "Bounced Bugs"? (Returned from SQA) For XP at Symantec, went from ~50% to 10%.
• Measuring the ability to Deal with change = greater number of enhancements during the project. The difference from the original plans.
• Measuring the Code:
• Looking for places to refactor.
• Track KLOC removed.
• Code refactoring - it can be a sawtooth wave - the complexity builds up, and then you have to respond. Decrease the distance between points for more agility.
• "Are we refactoring enough?" If you put it off too long, the design becomes too complex, and you can't enhance/maintain it.
• Encurring too much "technical debt".
• Wordperfect version 5 to 6 - they had to do a big rewrite, lost customers when they did it. Killed the company!
• Static analysis - function size? There is an optimal size - not too big, not too small.
• These analysis tools are really useful when you think the code is in good shape. Problems are not obvious! Focusing on prevention of problems.

Tools

• Sourcemonitor
• CCCC - an open-source tool on Sourceforge (C++)
• MKS, Discovery - $$ but worth it (used at CMM Level 5 company - MKS has support issues!)
• Coverage Tools - Java: Optimizit, Clover ("tried several, this one worked well and was < $$") C++: Cantata
• PSM - classes of metrics - helps w/ choosing targets
• Web page of metrics posted every day ("we did this, but no-one seemed to care")

Going around the room

• "More energized to go home and use measurement - thank you for the notion of following trends - sometimes get to caught in the numbers."
• "Love what we've done - interested in tools that go into developers' hands."
• "Want to look into coverage tools. Want to look at the PMS for ideas for metrics."
• "Find something that we can discuss with others."
• "Glad I'm not the only one struggling with this!"
• "I liked the different perspectives and histories that we heard about."
• "Interested in the coverage number - 70% - but we're not agile."
• "Looking for something to take home for Monday. Test lines per lines of code."
• "Ratios of lines of code is interesting - what are the trends?"
• "The golden gate bridge effect is interesting (big, periodic refactoring efforts). Nice to hear about that phenomenon."

---

[.FrontPage] [.RecentChanges]