Great article by Keith Swenson explaining why large and complex organizations are difficult to model using “scientific management”.

I think Keith makes a couple of great points that are worth letting soak in:

• Complex systems are unpredictable
• Organizations that “Learn” can adapt when the environment becomes less predictable.
• If you divide your organization between brains and brawn, you are failing to reach your organization’s full potential.
• Learn from *near* mistakes – not just the *actual* mistakes.

Of his takeaways for BPM- my favorite is transparency:  people involved in the process need to understand the part they play and the context of the process if they are to help the organization learn and improve upon it.

Worth the read and there’s a good comment-stream as well.

On a tangent relating this to political science…

For very complex organizations, it may well be worth applying some theories from political science and international relations to your thought process. One of the most useful political science theories is “Realism” – the notion that nations act in their own best self-interest, in a rational and calculating way.  My first exposure to this was in Dr. Krasner’s course on International Politics.  Usually deviations from what appears to be self-interest can be explained by imperfect information or miscalculation – but in general realism probably most closely describes how most states will act at critical moments of international relations.  In large, complex organizations, it helps to assume that departments, teams, and divisions will apply a healthy dose of Realism to the way they behave.  Part of how you deal with overly complex systems is by having useful abstractions that give you a shortcut to understanding, not ALL of the minutiae of cause and effect, but rather the SUM TOTAL of these effects.  It is much like the difference between micro and macro economics. Realism does not deny the other effects, but it argues that they are secondary effects rather than the primary correlating explanation.

Keith put together a pretty interesting chart reflecting business process technologies over the last few decades, showing how they relate on a spectrum from predictable to unpredictable, as well as how firm the consensus about how to use the technology has become.

Keith Swenson, Thoughts on Collaborative Planning, "4 Process Trends & 1 Gap"

As usual, Keith presents a good thought model.  But of course there’s a gap that may or may not be real – it is Keith’s perception of the gap between manual (email) processes and workflow-based processes.  I think there are actually a range of technologies inbetween these two approaches-  ticketing systems, case management, and wikis are all examples of technologies that address the space described by the gap.  Still, I always find this kind of graph thought-provoking: it organizes your thoughts in support or in contrast to a particular point of  view.

Keith Swenson has a few posts on the frustrations of email these days – and after some time experimenting and a friendly nudge from me, he’s come around and tried OtherInbox.

The problem?  Too much spam, which makes email nearly useless.

The proposed solution (my take):  use an email provider that has a REALLY good spam filter.  Like Gmail.  I have a very old personal address that gets 99% spam but the occasional personal note comes through it.  I forward it through a Gmail account and as a result I see the 10 or so useful emails that show up on that email address per year.

But, Gmail only effectively filters spam.  It doesn’t help you filter merchant emails that you would like to receive, but perhaps not in the quantities that you receive them.  Only some merchants are nice enough to let you state how often you want to hear from them.  There are lots of opt-in situations where you are required to provide an email address.  You could use a single “spam email” to opt-in to all these confirmations, etc.  But then you can’t tell who gave your email address to the spammers when it does go bad, and you can’t turn it off selectively.

OtherInbox solves that problem by letting you give each merchant a unique email address that you can later turn off if it becomes an avenue for spam.  You can also use it to sign up for group mailing lists, and the like, which are more likely to get compromised than some commercial accounts.

I think between Gmail and OtherInbox, you can largely whip the spam problem.  For now…

Keith is consistently writing the clearest thoughts on the subject of “Process Reification” vs. “Model Preservation”.

In this particular article on the subject, Keith focuses on a very common misconception in the BPM community (even by some well known names in the field):

I have met so many people that think that BPM is just a scripting language for SOA services.  These people will argue at length that “BPM is a part of SOA because SOA is useful without BPM, while BPM is useless without SOA”

(Notably, he also references a good article by JP Morganthal which sounds controversial – keeping your SOA and BPM initiatives separate, but which actually makes some very good points about why SOA and BPM initiatives are motivated by different organizational and business imperatives).

But BPM is not just a scripting language for your SOA services.  To quote from Keith’s blog: “SOA is a tool we use, while BPM is what we do.”   I think this is a really important dichotomy to understand, and explains precisely why so many companies that purport to be in “BPM” still don’t really get it yet.

Having just referenced Connie Moore’s thoughts on collaborative BPM, it seems a propos to then turn our attention to “the process wiki” – which I learned about through Keith Swenson’s blog post (“Rise of the Process Wiki”) on the subject (and I’ve since seen a couple of additional references).

If something like the process wiki takes off, it could really change some things in the BPM world.  I’m still a little underwhelmed by what is already in the wiki, but I think that is partly because the vendor-specific collaboration forums offer more interaction with the models directly.  But, where I can see real value is in promoting discussion of the processes that are posted, trade-offs in design, etc.  That’s where it would play to a wiki’s strengths, in my opinion.

You can find a pretty good discussion of the site on Keith’s blog, as it dovetails nicely with his interest in model portability and XPDL as a vehicle for such.

“Window Dressing” can kill your project.  It can even kill your BPM project.  Keith Swenson wrote a humorous but educational piece on an in-house BPM project gone awry.  Anyone who has deployed an internal application can relate to his story, but if you’ve deployed your own technology (or your own BPM technology) it hits especially close to home!

My favorite point made in the post is that flashy or pretty UIs don’t add value in-and-of-themselves.  That the focus should start with function and value, and apply the Keep It Simple Stupid (KISS) principle.  The extra UI “features” are break-even, if not value-detracting, from a value proposition.

Interestingly, Keith’s very next post was about the same kind of window dressing… but this time on the “back end” of integration/services:

I was asked “What common mistake do people make that causes unnecessary delay in BPM projects?”  The answer: Many projects have a goal to implement too much at once.  Some projects attempt to turn a manual process into a completely automated “straight-through” processes where there is no human interaction at all.

It is true that the more you can automate a process, the lower the cost or running the process.  But the cost of automation can vary greatly from activity to activity.

As he points out, integration is expensive.  Wrapping that legacy application as a service, is expensive.  It often makes sense to start with more easily attainable goals as a fallback position – by enabling the process without integrations first.  A technique that we’ve found effective on our projects:

• Start by modeling the process flow (which includes steps such as process mapping, inputs/outputs, etc.)
• Get the right level of logical abstraction in the second pass (re-organizing the original process)
• Get rudimentary UIs in place to allow running the process with as few integrations as possible – fewer integrations even than what the project requirements are.  Make sure the happy path works.
• Build test services to mimic the behavior of the integrations where possible – this allows you to test the various outcomes, exceptions, etc.
• Now prioritize and build the integrations that improve the process, based on the value of the improvement as compared to the cost of the improvement.

In fact, a process can be deployed with substantially all of the integrations yet to be completed, if the business users and IT professionals can be pragmatic rather than idealistic.  As Keith himself states:

There is an emotional side.  Some people are shocked when I suggest that we simply assign a task to a user, and have that user manually update the information in the third party system.  It just seems wrong to automate the process so that people can manually do a task.  This group-think often pushes a project to an all-or-nothing stance in process automation.  Striving to automate that last 5% causes most of the delay and cost of a process automation project.

This is best-practice process improvement as enabled by BPM.  It adheres to some of the principles that folks in the Lean school of that would espouse: namely, that you don’t let technology get in the way of your process.  Figure out the process, then use technology to improve it as the process stabilizes.

(As an aside, on one of my recent projects, rather than wrap batch scripts as synchronous transactional web services, we just continued to trigger them the old fashioned way:  by inserting records into their job-scheduling tables and polling for results.  It isn’t a particularly exciting method of integration, but these batch jobs already supported it, and it gave us better velocity toward production deployment. )

Just giving a shout-out to Keith Swenson, who has changed his blog’s title from “Go Flow” to “Thoughts on Collaborative Planning”, and he explains why in this post.  I think its a bit like when you start writing a book, a page at a time, and at some point you realize you have the wrong title, or you can think of a better one.  Keith’s blog is more about quality than quantity, and every time he posts I’ve found it to be a valuable, thoughtful discourse, and maybe more importantly, he’s been thought-provoking.  Here’s hoping there’s at least another 3 years and 90 more posts on your blog, Keith!

A couple of posts on this subject recently, one of them by my good friend John Reynolds (Process Manager Interoperability).  Keith Swenson made a reference to the Wf-XML spec as being relatively little-known and he wasn’t kidding – neither John nor I had heard of it previously and no doubt part of that is because the name says nothing about what it can accomplish for you.  The letters “Wf-XML” just don’t say anything about what this spec will do for you (allow you to start, monitor, and react to the completion of a process run in another system/environment).

As John points out, it seems higher probability that consumers of BPM technologies will benefit more from technologies that help with interoperability between those BPM tools, rather than from the more complicated task of actually migrating those implementations across BPM tools.  Of course, the SOA camp ca well claim that they solve this problem – but it isn’t the service-oriented nature of the interface that is the problem – its having a purpose-built standard for the design pattern of two independent process platforms communicating about the state of a process instance (or subprocess instance).

Keith recently listed 21 questions in his blog that he recommends a prospective buyer should ask BPM vendors when comparing products.  Not all the questions are about interoperability exclusively, but the first few are definitely focused on interoperability.  However, you have to weight questions based on how important they are to you.  You can get a highly inter-operable product that doesn’t functionally *do* what you need it to do.  You can get a product that functionally hits all your needs but doesn’t support interchange formats… and you can find data points inbetween.

Unfortunately we’re still very early in the world of interchanging business process models.  I would be hesitant to place interchange high on my list for differentiating my vendors.  I would place more emphasis on functional requirements, total cost of ownership, and adherence to my IT approach.  After those needs are met, then I’d start sorting by other things (interchange of BPM models being one).  One of the things I’ve noticed is that the general assumption seems to be that if I export a business process from one tool to XPDL and import it to another, it will just run in the new tool.  That isn’t necessarily the case.  Different BPM tools provide a different level of support for self-contained implementation specifications (implementation of those activities or services), and generally speaking those implementation details won’t translate, because they aren’t explicitly part of any of the BPM specs.

Having said that, Keith has a few questions to ask that have an impact on interoperability or interchange, but are also key differentiators of product capability, including questions about nested processes and how deep the nesting can go, data tracking, event posting, supporting wf-xml, exposing processes as webservices, etc.  Check out his list for more in-depth discussion of each one!

Keith has updated the community with a Model Portability Landmark:  WfMC’s announcement of a BPMN Model portability test.

This is a great step forward, and Keith does a great job summing up the results.  Unfortunately, from my perspective, the portability standard uses XPDL, rather than the as-yet-to-be-approved BPMN 2.0.  I have nothing against XPDL, I just wish some of the vendors I work with supported it!  Without that support, I’m tempted to rain on the parade of this announcement of portability.  For example, missing from this list on WfMC’s website are Lombardi, Savvion, Appian, Intalio, Pega.  Well, Pega and Lombardi (arguably market leaders based on Analyst reports) were recently placed prominently in Gartner’s Magic Quadrant for BPM, and they don’t support XPDL.

However, there is a silver lining for those of us who don’t use products that support XPDL.  Whether BPMN portability can be accomplished or not is not longer a theory.  It is a fact.  It can be done, and it has been done.  Now the question is whether the BPMN-2.0 supporters will get to the altar of portability sooner than later… one can hope!

Keith has put out a series of articles (possibly more to come) that lay down a great marker for how to evaluate Modeling strategies and how these approaches compare (strengths and weaknesses of each, across several dimensions).

It started with this post, which set the table for further discussion.

Following that were posts on Analytics, Round-tripping, Performance, and Simulation.

I recommend reading them all if you want a better understanding of some of the BPMN – BPEL debates (without even mentioning those terms), and understanding implicit tradeoffs in a product that the vendor may not (or may not be able to) articulate explicitly.

A long-standing debate in the BPM world has centered around whether or not to preserve the model.  It sounds like an esoteric topic, and Keith Swenson’s post: Model Strategy: Preserving vs. Transforming, takes a pretty clinical angle on this debate, and I think lays out the two camps and their respective arguments pretty well.

I like his definition of the common ground components:

• business process enactment: the business process itself has a dynamic component as it moves from the beginning to the end of handling a single case. The process definition does not normally change here, only the process instance or context that records the state of a particular case changes.
• business process lifecycle: these are the changes that a business process goes through from initial concept, to modeling, to integration, and finally to deployment into an enactment environment.
• business process improvement: this is the change to a business process that occurs over time through repeated use of the business process lifecycle followed by analysis of how well that version of the business process worked. This is the TQM or continual improvement aspect of BPM.

The two camps then divide themselves into Model Transformers, and Model Preservers.  The Transformer camp, as Keith argues, has roots in long traditions and proven success in software engineering (and related engineering tasks).  For example, 4G tools, UML, and even VLSI chip design applications are good examples – where the picture translates into code through transformation to a new form.  However, the analogy to these prior tools and content areas isn’t perfect.  For one thing, the author of the “picture” was likely the same person who was the consumer of the picture – the engineer (or developer).  It might not literally be the same person, but the same kind of audience.

Keith gives a good example of how transformation might apply to a Business Process model:

Coming back to a BPM example: the business analyst might put an activity describing a document review in the business view. This would be transformed to: a activity which sends email informing the person; an activity to check the document out of a repository; an activity to send a reminder message if the response is getting late, and an activity to wait for the response to come back. The programmer might extend this to include notation to handle exceptions and other timeouts that the business person did not include. Later this might be transformed again in order to get the executable code.

This is a pretty good example, though in my experience the business usually thinks of things like email, repository checkout, and reminder messages… However, the programmer very well might have exceptions to handle and other technical details that don’t direclty concern the business person creating the original document.  I think the key thing to note here though, is that to the model transformers, it is okay, and even expected, that by the time the technical folks are done adding their fit-and-finish and details to the model, that it will be unrecognizable to the business (Keith has a great image of simple model transformed to complicated model transformed to XML… surely a format the business won’t relish).  There is great discussion in this camp about something that is a hard problem – the “roundtripping” problem.  Because people can model things that are hard to round-trip.  And they can edit the transformed content, which is not guaranteed to have exactly the same freedoms and restrictions as the modeling environment.  So you have what I would call a “lossy” transformation.  It is not the same as compiling to bytecode and executing, because the expectations is that a developer or engineer will EDIT the “bytecode”… Anyone who has decompiled optimized java bytecode has seen that the compiler makes changes, and it no longer looks like the original code.  If you also edited that bytecode, the resemblance to the original would be even more tenuous.

Suffice to say, Round-Tripping is a really-hard-problem for the folks in the Transformation camp.  The best argument I’ve heard from this camp is that you should *not* round-trip, but instead, treat the transformation as a one-way journey.  I think that would be fine, so long as the developer cannot make further changes to the result of the transform. (this is, by the way, what Ismael Ghalimi recommends)

The second camp, the Model Preservers, believe the model is the model is the model.  While developers or engineers can ADD to the model, they must do so in a way that preserves the model.  So, integrations, exceptions, etc. can be layered onto that model, but they are layered in context- and without losing the original context of the model.  The concept of Round-Tripping is trivial – the visual and storage and execution are expected to be 100% in-synch, and any difference would be treated as a defect to be fixed, and all three are designed with BPMN as the starting and ending point…

So why is preserving the model important?  Because if you embark on the transformation approach, the model that the business creates eventually becomes a piece of paper or image on the screen that isn’t truly relevant to what is running in production.  In traditional software development there might be a very important phase where the business validates that the software developed meets “the spec” as represented in a generally large Word document… But this step can be eliminated if you do model preservation – instead, the business is testing whether the BUSINESS captured the process effectively and correctly, and whether the technical bells and whistles adequately support the process, and no longer testing adherence to a 6-month old specification of requirements.

The power of being able to execute the model the business understands is huge.  The power of being able to tie back the analytics to that model (trivially even) is also huge.

However, software vendors in either camp still have to execute well on their designs and software development to get good results… And neither approach is easy.  But then, that’s why we buy the software! (or the support contract)