Statistical Significance of Observable Data
- April 27, 2009
- 3 Comments
All too often I see conclusions based on observable data, where the conclusion does not necessarily follow the data presented. This doesn’t mean that the conclusion is wrong on the face of it, but that it can’t be made based on the facts presented thus far. Sometimes the conclusion is presented as causal when it is only correlated, sometimes it is extrapolating from a really small sample to describe the whole population (over generalizing). I recently read Theo Priestley’s post on why Six Sigma doesn’t work in the real world. In it, he relates a LinkedIn posting from someone attempting to do six sigma analysis on a call center process. From reading the LinkedIn posting, it is clear that the person posting is not experienced in Six Sigma or other related process improvement technologies that would be helpful to the cause. This person is trying to figure out how to get a bell curve from a formula related to rate of defects in the call center process, where a defect is defined as a call response being over 60 seconds. Theo extrapolates from this that it is an example of why Six Sigma doesn’t work in the real world, and why black belts are not needed (all that is really needed, he says, is a pragmatic approach). And he quotes a response he advocates:
“…at the end of the day if you produce a bell curve telling me the USL and LSL for my call centre, along with the number of defects per million and a sigma value of 1.727, is this really a useful measure? More to the point, what can I – as a business person – do with it?”
Well, look. The response (quoted) asks the right question – is this a really useful measure, and what can I, as a business person, do with it? In other words, are you wasting our time trying to figure this out in the first place? But here’s the rub. A good Six Sigma practitioner would not need to post to linkedIn (hardly the font of six sigma knowledge in the first place) to ask how to plot a bell curve to show USL and LSL. So the poster hardly represents the “failure of Six Sigma in the real world”. And a good six sigma practitioner could tell the respondent quite nicely:
If we can plot the response times of your call center, we can understand whether the process is “in control” or not – by which we mean, is it predictable and does it vary in a “normal” way from the median behavior. If it *is* in control, then we can endeavor to improve the process by decreasing overall response time if too great a percentage of responses are exceeding your 60s window. However, if your process is *not* under control, then this means that there are one or more special cause conditions causing the process to be more volatile. Therefore your first effort has to be on stabilization before you effectively focus on ratcheting down the long-term variation (i.e. drop the average of the process down as a whole).
Not that Six Sigma is the only tool available. Lean has several tools that are well suited for call-center style efficiencies, and so does 5s (the Japanese concepts of organizing the workplace to keep things consistent and orderly in order to keep the process consistent as well) and keep in mind just like BPM, Six Sigma is continuously evolving with new tools and techniques and while the statistics is certainly an important part it’s not the alpha-omega that many who haven’t learned Six Sigma believe it to be; certainly not the Six Sigma of the 1980’s. My problem with the post: using one example of someone struggling with the concept of Six Sigma, to challenge the whole notion of using Six Sigma – a bit like challenging the concept of a 100-meter dash after watching me run it, instead of Carl Lewis – at least look at how experts apply the techniques! Six Sigma is not a religion (or at least it doesn’t have to be!)- it is simply a set of tools that can be used to pragmatically improve your process by focusing on unemotional data rather than exposition based on anecdotes. Not all the tools are even statistical, which was a surprise to me when I first studied material at the green belt level -but even those tools are very practical process improvement tools. The biggest problem with Six Sigma in the past has been the C in the DMAIC moniker (Define, Measure, Analyze, Improve, Control). BPM really helps address C by putting the controls in software (call center software can help with this as well), and it helps with the M by helping measure the process even when no Six Sigma blackbelts are paying attention. This makes it a lot easier for them to drop in, interpret the data, and devise new improvements based on the data, rather than spending so much time collecting hand measurements. Here’s a great post by Jason Cohen (of Smart Bear software) that illustrates that humans are terrible at making gut decisions that can be disproved with statistics, and how to help yourself avoid sample size errors at least when dealing with A/B tests (as in, which is better, A or B?). Better yet, it has a cute video of Hammy the Hamster to assist. It cuts to the point about how much data you need to get a reasonable sample size to draw conclusions, and I think it makes clear why data sets of less than 10 are just generally problematic. Um, also, there are formulas involved, so if you don’t put much faith in mathematics, Mr. Cohen’s article may not be much comfort to you, but I promise the math required to evaluate your sample size will be easy to do! In my view, the real problem with applying Six Sigma is usually people. Turns out it takes skill and judgment to apply Six Sigma tools in a way that helps the business answer questions the business cares about. This isn’t that much different than the kind of challenge one confronts getting a business to adopt BPM, or a software package. Let’s not condemn a whole professional practice with proven successes just because we have one example of someone who doesn’t understand how to apply it (or even 10 examples of such people).