Kevin Hillstrom on (Web) Analytics

Scott Francis
Next Post
Previous Post
I know Kevin’s article was targeted at web analytics, rather than, more generally, business analytics – but the points he makes are entirely valid for business analytics – and business process analytics:
We analyzed each promotion code, using “A/B” test panels. Customers were randomly selected from the population, and then assigned to one of two test panels. The first test panel received the promotion, the second test panel did not receive the promotion. […] In almost all cases, the segment receiving the promotion generated more profit than the control segment. […] Being a huge fan of “A/B” testing, I decided to try something different. I asked my circulation team to choose two customer groups at random from our housefile. One group would receive promotions for the next six months, if the customer was eligible to receive the promotion. The other group would not receive a single promotion for the next six months. At the end of the six month test period, we would determine which strategy yielded the most profit. At the end of six months, we observed a surprising outcome. The test group that received no promotions spent the exact same amount of money that the group receiving all promotions spent. After calculating the profitability of each test group, it was obvious that Eddie Bauer was making a significant mistake. […] In 1999, we backed off of almost all of our housefile promotions. At the end of 1999, the website/catalog division enjoyed the most profitable year in the history of the business.
This is a classic cautionary tale for anyone measuring business processes and looking for improvements.  In process improvement, often an efficiency gain comes at the cost of customer satisfaction – which is why many businesses now manage to a “double bottom line” or use extremely customer-focused culture to counter-act that tendency.  In the case above, they’re looking at how incentives shape behavior. The thought process is equally valid for customers as it is for internal staff incentives. Do the incentives actually drive better behavior or just more short-term-optimized behavior? Kevin goes on to recommend several remedies, most of which revolve around having a longer term view of things.  Well. Imagine that.  Good read.

Related Posts