Show Me the Value!
Even the most skeptical of our tour visitors sense both our passion and our enthusiasm for what happens in the Menlo Software Factory™. However, a common question they ask is, “Do you have any data to show me that this actually works better than traditional approaches?”
I must admit my first reaction to this question is usually an unspoken emotional response along the lines of “You’ve got to be kidding. Can’t you just see the difference?” Sometimes it moves from emotional to reactionary as I want to ask them, “If I collect the data and show it to you, what will you compare it to in your world?”
I never actually verbalized either of these emotional responses and it wasn’t until last week when I received another version of this question via email, did I actually go and get some data and report it back. I thought you might enjoy peeking in on the interaction:
Hello Richard.
… I would be interested in any metrics you may have collected on your teams performance. Such as defect rate (defects per KLOC [defects per thousand lines of code]) and output performance (i.e. lines of code per day).
To which I responded:
Interesting question about lines of code per day. As part of our engineering practice, we actually value fewer lines of code rather than more. By analogy, it is like the old quote “I was going to write you a short letter, but I didn’t have time, so I wrote you a long one.”
We often are able to add features by reducing the number of lines of code. Counter intuitive, but powerful in both elegance, simplicity and quality. Fewer lines of code to maintain, fewer “moving parts”, etc. The other interesting statistic is that about 50%-60% of the “lines of code” we write are automated unit tests which are written BEFORE we write the production code they are going to test. In addition to this being a phenomenal automated test approach, this is also a powerful act of very intentional design (i.e. you can’t devise a test strategy without conceiving the design of the code and that design most be modular so that it CAN be tested). These lines of code are not shipped with the production product.
Regarding defect rate, I asked one of our project managers to take the total hours of one of our larger projects and separate out the number of hours spent working on “bug” cards.
Here’s the calculation: 669.25 hours working on bugs divided by 17,592.50 hours working on all (features and fixes) development. Total percentage of time working on bugs is 3.8%. Thanks for asking the question, as we’ve never had one of our clients ask for this. Even I was impressed with the result! I am going to ask this question of our other projects. Happy to share those answers with you if you like.
Hope this helps. Best wishes to you as well!
I’d be interested in hearing your thoughts on these performance metrics. What measurements have you done to demonstrate the effectiveness of your team? What measurements would you be interested in hearing more about?
Management, Menlo Software Factory, Quality Assurance, Software Development
| 
defects, executives, KLOC, QA, quality, Quality Assurance, testing, unit testing
I can easily best your bug-time-percentage metric: I won’t spend any time fixing any of my bugs!
While it fails to account for bugs that clients are not interested in fixing, this could also be viewed as a way of incorporating the importance of bugs. But it is not a straightforward metric when you conflate the two. (Bugs/kloc certainly has problems too, as you point out: kloc != good and not all bugs are equal.)
[...] the previous post we began to explore the question of “value” as it relates to our practices. We left off [...]
I would also be interested in what is sometimes called Rolled Throughput Yield. What the 3.8% represents is the combination of both final yield – bug cards generated, say during the last iteration, and the other bug cards generated throughout the other, earlier iterations. I would guess that the bug cards generated during the Menlo application development cycle are distributed inversely to those bugs that are caught in a traditional “waterfall” application development cycle. So you probably have more bug cards earlier and have fewer near the end of development. I say that because at Menlo the mantra is “make mistakes early” whereas traditionally most of the testing is left to the end. In manufacturing we use a metric call First Time Right – how many of the features are coded without error, first pass. It might be interesting to consider how to implement such a measure and what the implications might be – you don’t want to promote the wrong behavior but it might provide some creative tension to introduce more dialogue and clarity between the HTAs/ designers and the programmers.