This phrase is a familiar, and everyone involved in firmware or software knows the familiar completion. That’s not what I’m writing about.
I have been seeing lot’s of defects in both software and firmware lately, and I use lately very loosely. This isn’t a local phenomenon or a new one, but one that seems to be trending, fast, accelerating even, but not in the right direction. So if we don’t want a lot of unintended “features” in our software and firmware, what do we do?
I want to step back to the title, and consider different endings, that can hopefully begin the mindset change that can eventually reverse the trend.
It’s not a defect, it’s … a CHALLENGE
Every defect that ships, even when fully characterized, presents a challenge to Sales and Marketing with the customer. While honestly and fully describing problems and workarounds helps a customer who has decided on your product, it presents a problem in “hooking” the customer, more struggles as they are “reeled-in”. Every item needs to be explained and rationalized by the salesperson as to why it will not negatively affect them or their customers. And a tremendous amount of trust is required in this process.
So even as the design team looks at little defects as relatively easy to avoid or workaround, every defect poses an obstacle to customer adoption. Consider the recent problems with Toyota and their accelerators (actually there were two main problems, mats that got trapped with accelerators and then accelerators that surged or got stuck). Every customer who has heard of this will have questions for the salesperson, but many more won’t even give the salesperson a chance to answer. So the challenge isn’t whether there is an answer to why a defect isn’t going to affect the customer, but that many customers won’t even give you a chance, based upon the list or number of known defects. And any defects found by the customer (whether documented in advance or not) will appear even more critical than those that are explained, that the customer has been cautioned about and directed how to avoid.
The central bankers (like Alan Greenspan) have referred to a similar economic condition – they call it headwind: a force you have to continually work against and reduces your efficiency. Defects are a headwind to business.
It’s not a defect, it’s … an OPPORTUNITY
In a manufacturing environment, defects are typically test escapes. When there is a test escape, after the problem has been properly contained and the production line continues, the next step is to find the root cause of the test escape: Was there a difference in the spec’d test environment versus the deployed test environment? Or did a test get skipped? Or was it a failure to understand the need for a test? As you can see, the answer to this first question raises many more questions (all beginning with “WHY”). Traditional root cause analysis ensues and based upon the learning, apply the lessons.
Now contrast this with a software environment. What is the first thing that happens in a defect review? In my experience, one or more engineers begin brainstorming about how to correct the defect and when they can roll in the fix, OR (depending upon the phase of the project and moon) a discussion ensues on how long the delivery will be delayed by even considering fixing the defect. Both of these responses are WRONG. The number one question isn’t IF or WHEN but WHY, and the opportunity presented by every defect is rooting out the causes of defects (there ARE common causes, but you won’t see them if you don’t look).
Deadlines are deadlines, but even as hard as I push to get software I am waiting for released as soon as possible, the presence of one defect is like the presence of one cockroach. The only viable “fix” to emerging defects is to take time to get to the root of the defect. And I also think this is a major learning opportunity for the person in whose code the defect was found. If a function I wrote has a defect (of course being in marketing this is only theoretical) it should then be me who determines the root cause of this defect (even before identifying the fix). The rest of the team helps to review and vet the root cause analysis, not to tell the “defector” why they think he/she had the defect but to help the “defector” come to this reason themselves. Do not just stomp out the one cockroach seen but identify where it came from so you can look for the nest, find and destroy the eggs. If you know why you got a defect, you will know how to avoid getting one next time.
It really is a feature, but in a bad way
The tongue-in-cheek response to defects as being features really IS true. If you find defects and only ask yourself is it a “feature” to explain in release notes or a “defect” to be fixed ASAP, you are defining your product by its defects. If instead you take advantage of the opportunity presented by finding a defect to root-cause it and therefore attack the root source of future defects, you are defining your product by its quality.
And it’s measurable. If you know the root cause you can monitor and measure/count the future appearance of the same root-cause-defects.
Methods and Madness
May 24, 2011