I know Kevin’s article was targeted at web analytics, rather than, more generally, business analytics – but the points he makes are entirely valid for business analytics – and business process analytics:

We analyzed each promotion code, using “A/B” test panels. Customers were randomly selected from the population, and then assigned to one of two test panels. The first test panel received the promotion, the second test panel did not receive the promotion. [...]

In almost all cases, the segment receiving the promotion generated more profit than the control segment. [...]

Being a huge fan of “A/B” testing, I decided to try something different. I asked my circulation team to choose two customer groups at random from our housefile. One group would receive promotions for the next six months, if the customer was eligible to receive the promotion. The other group would not receive a single promotion for the next six months. At the end of the six month test period, we would determine which strategy yielded the most profit.

At the end of six months, we observed a surprising outcome. The test group that received no promotions spent the exact same amount of money that the group receiving all promotions spent. After calculating the profitability of each test group, it was obvious that Eddie Bauer was making a significant mistake. [...]

In 1999, we backed off of almost all of our housefile promotions. At the end of 1999, the website/catalog division enjoyed the most profitable year in the history of the business.

This is a classic cautionary tale for anyone measuring business processes and looking for improvements.  In process improvement, often an efficiency gain comes at the cost of customer satisfaction – which is why many businesses now manage to a “double bottom line” or use extremely customer-focused culture to counter-act that tendency.  In the case above, they’re looking at how incentives shape behavior. The thought process is equally valid for customers as it is for internal staff incentives. Do the incentives actually drive better behavior or just more short-term-optimized behavior?

Kevin goes on to recommend several remedies, most of which revolve around having a longer term view of things.  Well. Imagine that.  Good read.

Interesting article in the MIT Sloan Management Review, “The New, Faster Face of Innovation.”  In this article the author writes about the innovation currently going on in the act of innovating itself.  It seems a bit circular in logic, but true, that companies are figuring out how to innovate more frequently, and how to apply a rigor to that innovation that brings with it lower cost of failure and greater likelihood of being able to capitalize on success.

Because of the lower costs and risks, more innovation investments are being made… which in turn drives more innovation overall.  Because more innovation efforts are being undertaken, the effort to manage that innovation is also underway, and improving in its own techniques.  In addition to lower cost, of course, the economic environment is putting a premium on innovative ways to cut costs, provide better services, or find more revenue.

One of the key tenets of this new wave of innovation is the ability to try tests on subsets of your market- a change on a website submitted to 10% of your visitors, for example – to test the efficacy of a change before making that change available to the bulk of your customers or market. This is often referred to as A/B testing and we’ve previously discussed it here (toward the bottom of the post).  Humans are notoriously bad at making gut decisions based on anecdotal data, and testing lets us actually make GOOD decisions based on real data.

I think BPM can make your business “A/B-test-capable”.  Meaning, once you implement a BPMS solution around your business process, you can actually conduct A/B testing and determine whether process changes are effective before you make either a full switch or make the change semi-permanent.  Without a BPMS and process implementation in place, A/B testing tends to decline to levels that can’t achieve statistical significance:  you ask one team to do their work differently and ask them how it felt – instead of routing work differently for a percentage of process instances and measuring the efficiencies or errors that result.  This type of innovative test-and-test-and-fix approach lowers the costs of the tests, and increases the odds of finding real improvements that can be implemented.

In a recent project, we debated how to do task assignment – do we let users pull tasks as and when they are ready (and in a way that prevents cherry picking), or do we push tasks to users when their queue is nearly empty, and expect them to pick up the work that is assigned to them.  Instead of arguing anecdotally about which approach will work better, we could simply have deployed both task assignment solutions to different user populations and see which population is more efficient.

Oh, and did I mention that these tests can often be deployed in days or weeks, instead of months or years?

Read the rest of the Review article for a fairly website-centric point of view.  But then take a step back and think about the BPM-centric point of view- this is a powerful new rationale for the value of BPM Solutions (or at the least, a powerful way to communicate an existing value driver).