Scale is another word for Opportunity

  • March 26, 2014
  • Scott

speedometerIn the process world, large scale is just an opportunity for process improvement.

At small scale, many improvements won’t earn back their investment in a reasonable time frame.  But at large scale, or increasing scale, more and more improvements will earn back their investment in a reasonable time frame.  Of course, the flip side of this is that, when small, exceptional occurrences are, well, exceptional.  Whereas when at large scale, exceptional occurrences become common ones – and thus a good opportunity for process improvement.

Great examples from Jason Cohen of WP Engine:

Suppose I told you that on average our servers experience one fatal failure every three years. The kernel panics (the Linux equivalent of the Blue Screen of Death), or both the main and redundant power supply fails, or some other rare event that causes outage. Does that sound like a bad batting average? […]

So, even if our servers are more hardy than your MacBook Pro, they’re taking 100x the beating, so one failure every three years seems pretty reasonable.

But remember, we have 1,000 servers. Three years is about 1,000 days. So that means, on average, every single day we have a fatal server error.

In other words, the “exception” is now a daily occurrence.  The kind of thing you might invest in improving over time.  Jason takes it further:

The insight is that that scale causes rare events to become regular. Things happen with 1000 servers that you literally never once saw with 50 servers, and things which used to happen once in a blue moon, where a shrug and a manual reboot every six months was in fact an appropriate “process,” now happen every week, or even every day.

This is a really key insight for startups, growing companies, and operations that are running at scale. And a great insight for improving process.

And many people would assume improving process means automating.  Or as Jason puts it “Automate Everything” as the knee-jerk response:

Sure, without automated monitoring we’d be blind, and without automated problem-solving we’d be overwhelmed. So yes, “automate everything.”

But some things you can’t automate. You can’t “automate” a knowledgable, friendly customer support team. You can’t “automate” responding to a complaint on social media, which as our Twitter meister Austin Gunter says is usually a customer’s last resort and thus should always be treated as the very legitimate issue that it is. You can’t “automate” the recruiting, training, rapport, culture, and downright caring of teams of human beings who are awake 24/7/365, with skills ranging from multi-tasking on support chat to communicating clearly and professionally over the phone to logging into servers and identifying and fixing issues as fast as (humanly?) possible.

He’s right.  You can’t automate away the personal touch. The knowledgeable help.  You also can’t automate the truly rare technical problems (some of which maybe haven’t been seen before).

On the other hand, the business that is scaling, as WP Engine is, can afford to invest in the improvements that take care of customers and even improve service despite the challenges of scale.  As a WP Engine customer, I can attest to the fact that, if anything, customer support has improved since we first became a customer.  At BP3, our BP Labs group is investing in improvements to handle the increased scale of our investments – and when we roll each investment out publicly, the adherence to the theme will be more obvious.  It is nice to have some good examples like WP Engine to be familiar with when we look at our own investments.




Related Posts
  • March 14, 2018
  • Scott

We're heading to IBM's Think! conference next week, and we want to share some of the sessions that jump out to...

  • March 14, 2018
  • Ariana

The Importance of DevOps from BP3 on Vimeo. What is DevOps, what does it mean to customers and how are pe...

  • March 11, 2018
  • Scott

One of the things our engineering team does really well is keep improving the rough edges of any product they ...

  • PeterJ42

    Process Improvement has become a “big business” thing because of the incredibly wasteful way we go about it.

    We let a company operate sub-optimally for months or even years while we build a case for change.
    Then a manager makes a bid for glory with a project which is way larger than the problem needs. This automatically aligns the company into for and against.
    To diffuse the situation, an external guru is brought in, at enormous expense.
    Then follows several more months as he produces a report and everyone vies for influence. He brings in his favourite suppliers, all with risk-averse governance coming out of their ears and their eyes full of dollar signs.
    Months later a project is born. The battle lines are drawn. And IT takes a business brief and turns it into an 18 month software development project.
    Finally this delivers back something almost entirely unrecognisable – the original problem is long since forgotten in the wrangle for dominance in “the project”.
    Of course the IT thing doesn’t work. Many months later, compromises made, everyone basically gives up and the project moves along the line of least resistance to deliver probably 10% of the possible improvement.

    Now that is the way big business works. Not just in projects.

    But to tar process improvement with the same brush is wrong.

    Process improvement is about watching what happens and finding small, iterative improvements. Testing new ideas. Getting everyone together, not just a project team. Sweating the small stuff – taking the biggest problem and working on it until it isn’t biggest anymore. Taking the smallest processes and improving each of them until they make a significant improvement to the big process they are part of. Bottom up, not top down. And covering every process, not just the headline ones which managers can pin their ego to.

    Process improvement is not just for big business. It is how little businesses outcompete big business.

    • Peter –
      I’m not sure if you’re agreeing or disagreeing with me 🙂 or neither 🙂
      I completely agree that process improvement can be an instrument of competitive edge for small companies, as they grow. As they grow, they encounter issues with scaling (see the WPEngine example). And those issues, are also opportunities to not only address the issues but to invest in making things more reliable and repeatable going forward.

      I’ve seen the straw man you presented play out, but to play devil’s advocate, we’ve had many successful projects with large companies, who are receptive to change at least in the areas we’re working with them on. When they aren’t receptive to the changes that process improvement and/or BPM implementations require, we will exit stage left.

      The point of my post was that, as you grow, you’ll endlessly find new opportunities to improve- because things that weren’t worth investing in at small scale will become worth it at a larger scale. It doesn’t mean that the big business will be more efficient or outcompete the little business – it just means that they have a good ROI for investing in the improvements.