I am quite uncomfortable with the term “unpredictable”. If you reason in terms of resources and lifecycles (states and transitions), every activity becomes “predictable”. The resulting flow of activities may be mapped or unmapped. The only processes that are unpredictable are the ones where resources and their lifecycles are not completely mapped. What give the impression that something is unpredictable (i.e. not well mapped by BPMN…) is that some transitions allow a large number of activities to be performed and you have difficulty to represent a particular path.

The problem, you see, it to abstract the resources and their lifecycle from the processes. The moment you do that, you come up with this kind of classification and you believe that there are indeed different kinds of processes, including “unpredictable” ones, when in fact the underlying mechanism is common to all these “types” of processes.

A document under review has states, a marketing proposal being developed has states (often unmapped), a request for quote has state and a financial transaction has states as well. They all have a lifecycle and they all allow (or disallow) human or automatic activities to be performed at different states of their lifecycles.

I am not sure when the punditocracy will eventually come to the realization that indeed, resources and lifecycles cannot be ignored, but until then, the pundits will continue to come up with “gaps” and “unpredictability” and marvel at JavaScript as the “glue” that keeps processes executable.

Just my 2c

JJ-

Well, I think Keith is referring to processes that are not well-understood in advance – perhaps not yet mapped (but mappable), or perhaps processes that are perceived (by their users/owners) as being “unstructured” or relying too much on human judgment to be easily abstracted into a structured process.

The same oversimplification (or over-complication) can happen with respect to entities and their lifecycles as well if you pick the wrong amount of abstraction – too much abstraction and you haven’t captured an important state – too little abstraction and you’re capturing too many meaningless or undifferentiated states.

I guess in the projects I work on we usually do talk about the states of the entities we’re dealing with and what the touch-points with the process are so I don’t see the disconnect that you do – I just see different groups of pundits talking about it from different perspectives – one group that’s very focused on organizing the data and states, and another that’s focused on the business process model perspective.

As I said in my post, I’m not sure the gap that Keith perceives is really there- if you consider all the tech available to you, and not just the stuff that appears to be in the BPM market per se, I think there is actually quite a bit of software available to manage these processes for which email isn’t enough structure and workflow products appear to be too much structure (on the human activities, irrespective of the structure required for the data elements). Wikis and ticketing systems are good examples – they don’t put a lot of constraint on the user behavior, but the data definitely has states it is marshaled through that mean something to users. (as you can see if you mark something “resolved” when the ticket-submitter disagrees :)

Jean-Jacques,

I just wrote a piece about “blindness” to unpredictable processes: http://kswenson.wordpress.com/2010/01/08/it-is-all-taylors-fault/
There I talk about the fact that we are so used to using principles of Scientific Management we tend to only see things that can be analyzed this way. You might see this as a definitional problem: e.g. a “process” has to be predictable or it is not a “process”.

But I am talking about truly unpredictable process. Let me give you some examples:

(1) The US is currently entering Strategic Arms Reduction Treaty negotiations. It is a certainly that this negotiation will not go according to any fixed, regular plan. It is impossible at this point to predict the course of events.

(2) A patient arrives at a hospital. It is impossible to know the course of treatments that that patient will experience over the course the stay and the follow up. It is part of the job of the doctor to first diagnose the problem, and then prescribe the treatment, but that treatment is unpredictable (given the information known at the time of admittance).

This is not a matter of simply the process hasn’t been modeled yet.

It is also a rejection of the notion that there exists a “super process” in which we all live, with an immense number of branch nodes which accounts for all the possible variations we see in real life.

Instead, we observe a sort of “process butterfly effect” which eliminates the possibility of mapping certain work process in advance.

I have a degree in Physics, and when I started there was a belief that if you could measure the preconditions well enough, you could predict the weather arbitrarily far into the future. It would simply be a matter of measuring the initial conditions to sufficient precision. This belief is based on the assumption that small errors in measure will become less significant in time. Then along came chaos theory, which introduces the concept of “sensitive dependence upon initial conditions”. The idea that a measurement error as small as the flapping of a butterfly wing, might built in importance to the point that weather a few weeks into the future becomes unpredictable.

For process I call this “sensitive dependence on EXTERNAL conditions”. Some processes are sensitive in this way. The patient, for instance, could have a condition for which a treatment has just been discovered, and could never have been anticipated by a process. More likely, the doctor just found out about it, or maybe was persuaded based on other cases he personally has seen. No matter how good the doctor’s diagnosis is, it can’t be known perfectly, and it is not unusual for treatment to be cancelled or changed mid course due to “complicating factor” which were not predictable. Even when the diagnosis is perfect, such as infection by a particular germ, the microbes/viruses themselves mutate and produce unpredictable effects.

I hope I have persuaded you to see that there are such things as unpredictable processes and that this is simply not a matter of being unmapped. They are, in any sense, not mappable ahead of time. (They can, however, be mapped while the process going along, and that is the point of the gap).

Scott,

A shorter note: the gap exists as a gap in “major trends”. You are right, there are technologies and techniques that fit in there, but I believe they can yet be considered a trend.

Particularly you mention “Case Management” which I believe is at the heart of this trend. But, if you talk to the key vendors, you will find that they believe that “Case Management” is an application of their BPM technology. Savvion, Pega, others ship “Case Management Solutions” but I don’t believe that these solutions meet the need for “unpredictable processes”.

Only the future will tell whether this is a new trend or not, but you are right in thinking that some of the technology is available today … if you know where to look.

Keith-
good comments, on both counts. Regarding the first – although in principle I can agree that there are such unpredictable processes, in practice I think you can model something meaningful by not trying to model the specific treatments and decisions per se, but to model the phases/stages/states and having the doctor’s judgment/decisions to drive movement from one stage to another (not necessarily a linear progression). In practice, whenever someone gives an example of the unpredictable process, I can imagine the process being mapped and executed, but with a lighter hand, if that makes sense.

I’m still waiting for the concrete example that I really can’t even imagine the states in advance… this gets pretty abstract :) The physics examples I can relate to – it explains my belief in luck pretty well.

Keith:

I also have a degree in Physics and the fundamental concept in physics is “state”. State even defines “time” if you look at Prigogine’s work. Time only exists because the universe cannot revert to a given state. State transitions are irreversible. This is the biologic time. Einstein’s time is based on the definition of speed (and not the opposite).

All the examples that you are giving are easily mapped into a set of resources and their lifecycles. Ignoring that a process’s activities transition arbitrary resources from state to state in BPM amounts to trying flying airplanes pretending gravity does not count.

We talked many times in the past, you work for a vendor, people like Scott or Bruce are loosely associated with vendors. I appreciate that you can never say “as a vendor we forgot X” (imagine if Boeing were to admit they forgot to account for gravity in their latest air plane design…). So this discussion will never be closed until new vendors appear (IMHO) and the only good thing about this latest wave of acquisition is that we are opening the door for these new vendors to emerge. They will create a vacuum big enough for these vendors to exist. I concur with Bruce that, this wave of acquisition marks the end of this phase of BPM. I repeat what I said before, if these vendors had found the right foundation for BPM, they would be buying IBM, because this is as big as the BPM market should be. Thinking that a “top” BPM vendor is worth $50M (even in a bad economy) proves that this vision of BPM has failed completely. A successful BPM vendor should be at least selling 20-30 Billion of BPM licenses and services per year.

So, how do we get out of this discussion? I challenge you, Scott and Bruce to give me your most sophisticated process definition, something you feel is hard core BPM, possibly impossible to model with BPMN, and I will spend the time to decompose these processes into resources and their lifecycles. Then we’ll talk again.

My email is jdubray/gmail.com. I work for MomentumSI, a small consulting organization. We are vendor agnostic. We work with our customers to find the best solution for them, and yes, they do understand very, very well the concepts of resources and resource lifecycles. Recently the chief architect of one of our customers told me that he had learned more about their own business in one hour (explaining the relationship between lifecycles, SOA and BPM) than since he had joined the company. This is just a sample of the reactions we get.

If we could make the results of this challenge public, that would be best, but if you want to keep it private, I am still ready to spend the time. Because I know that progress means convincing people like you, Scott or Bruce to adopt these ideas.

JJ –
first, I never claimed that I could give such an example, so I can’t accept your challenge :) I can admit that theoretically such an example exists, but yet, I haven’t heard one yet that I can’t imagine modeling via modeling the states of the entities or even the states of the process, without trying to model all the possible decisions (in other words, treat the decisions as data rather than flow lines, and then you can take a vast model and make it simpler (potentially)).

I’m definitely not going to argue with two physicists about physics :) Turns out I had my own brush with physics writing code for the high energy physics researchers at SLAC, but that work had little to do with physics and much more to do with exposing statistical information (n-tuples) in ways that people could visualize more easily.

Sorry to disappoint you, JJ :) Its just that, you and I aren’t really disagreeing vis-a-vis the challenge so I’m not going to take up the argument just for argument’s sake.

Now, as to your comments of, if the existing vendors had really solved BPM they’d be buying IBM… well, you and I both know that that is specious. There is so much more to software than just writing a good product and solving a real problem – even if you have the right idea, it might require millions in R&D; that you have to raise money for. Even if you have the right product, it requires building a sales channel that can effectively close business. Even if you have all of this, you are fighting an uphill battle to establish new relationships with customers while the incumbents (e.g. IBM) already have relationships in these customers and big technology profiles. Even if you have all this, someone may offer you present cash value that makes it economically worth it to sell. So, my point is not that it isn’t possible, just that it isn’t as simple as you proclaim. However, I agree with you about how big the BPM market should/could be – hopefully either some vendor will come up with the right answers or IBM or one of the other big guys can leverage their buys to get us there.

As for MomentumSI, I didn’t realize you worked for one of our hometown (Austin) firms. Good for them to have someone of your experience on the team. Momentum once upon a time was a Lombardi partner and even helped build a piece of the product back in 2002-3. Your anecdote about the chief architect sounds very familiar to me – when we, as consultants, do our job well, we get this kind of feedback. If you get these kind of reactions, Momentum is sure to grow and prosper.
Best of luck,
Scott

JJ: I have always had the highest impression of your work. But I am surprised at your position on this. I would like to discuss further.

It is true …. you can always model anything as the following process:

1) assess the situation
2) decide what to do
3) do it
4) repeat ad-infinitum.

I am sure there are many variants of this. This model is not useful for any purpose as a model. It is a reasonable strategy if you have nothing else to go on, but it really is pointless as a model. It communicates nothing about what is really going to be done. There is nothing that anyone else can gain from this that tells them how to prepare for what is coming. You can’t make any predictions from this.

The generic doctor process might be only slightly different from this. You could, of course, have a process branch for all known treatments, and a big branch node before that, but then you get into a problem: my scenario includes the situation where a treatment was never known before (to the doctor or the hospital). There again, you could have the “do treatment not previously known” step in the process.

Scott touched on “useful level of abstraction”. For a model to be useful, it needs to include a certain level of detail. That detail is necessary for it to (1) provide a benefit in guiding people, (2) allow others to anticipate what is going to happen and coordinate to that, (3) …? What is a model good for? We need enough detail for effective coordination.

Note: when I say a process is unpredictable, I do not mean that it is entirely ad-hoc and unstructured. The structure will appear, it just can’t be known ahead of time. As work continues, fragments of the process may be brought in which are well formed. Thus the doctor says “do a liver biopsy” then there is a set process fragment that is well known for doing that, which yields a result. But the process beyond that is unknown until the result of the biopsy are received.

The entire process does not remain unpredictable over the entire life of the process. There is an “unfolding” of events that becomes clear as you go along. So when I say “unpredictable” I mean it is so at the time that the process is started.

As for your challenge: I stick with my patient/hospital example. There are thousands of new treatments being invented every day. Illnesses can be combined in rare patterns, making the typical treatment unusable. New drugs are invented, and old drugs are used in new ways. A doctor will specialize in a particular field, and be knowledgeable about only a fraction of what is known in a field. Different communities of patients have different patterns of illness, driving doctors to specialize in their community as well. Patients themselves will do research on their condition and ask their doctor to try something.

If all you are saying is that this can be approached with the assess-decide-do process, then we are in agreement. But as a model, I would say that process is not useful for anything.

The reason I am arguing that unpredictable processes exist, is that Scientific Management leads us to believe that all processes could be defined and that the workplace could be a huge factory where all the processes are known in advance. We simply need to do the work to write down the process in detail. My point is that that approach will not work for unpredictable processes, and is also will not work for predictable processes which are so rare that the cost of recording the process is more than the benefit. Thus a different approach is needed.

Keith, Scott:

thanks for your answers. There are many companies that grew out of nothing when the market is big enough: Microsoft, Google, SalesForce,… We would know by now if only 50+ year old companies were the only one that can succeed. So I’ll stand on my argument that this vision of BPM has failed and the current acquisitions simply puts an end to that vision. IBM and Progress with use a couple of things from their acquisition, keep the good engineers and milk the customer base (upsell they say). Let’s stop pretending that BPM has succeeded in any way, because it has not. Keeping pretending that only delays the day where BPM will be successful.

Keith, I am glad you want to continue the discussion. I do think this is very important and it can finally unlock the value of BPM. You are basically making my point. When you say:

1) assess the situation
2) decide what to do
3) do it
4) repeat ad-infinitum.

You really mean:
1) figure out which state(s) the entities participating the process are in
2) look up the activities that are enabled in these states
3) pick one (or more)
4) go back to 1)

>> It communicates nothing about what is really going to be done.
we agree on that, but I think, what you don’t realize is that this is effectively what is happening -ALL- the time in a process of any kind. Some process involves fixed resources (say a bank loan) or the context of the process can be more dynamic and accept or eliminate resources. For instance, a prescription can be dynamically added to the on going interaction between the patient and its physician. You can even have prescription interacting with each other. The patient itself is a composite state machine (heart, blood, other organs… all have well defined states, including the one based on measurements -HDL too high, …).

I think what you don’ t see is that the actions of the physician are totally controlled by the states of the patient. It is possible that the physician does not assess the states properly, but all its actions (translated into prescriptions) are bounded by the state of the patient.

What’s a bit unusual here, is that you don’t really have a system (yet) surrounding the patient preventing incorrect activities to happen and states are not known with a great precision (specially in the US), but the exact same mechanisms as the one involved in processing a purchase order are at play. This is actually how the entire universe works … If an electron is in a particular state on an atom, there are only so many actions that can change its state. If a molecule is in a certain state (attached to a catalyst, above a certain critical temperature, pressure…) then only specific actions can be performed.

This is why you can’t define all the processes in the universe (or the enterprise for that matter). It is far easier to specify the lifecycle of the (business) entities and observe which processes can happen (using BPMN to document these processes is perfectly ok). But thinking that you are “in control” because you write a BPMN definition and you somehow can “execute” it, does not reflect at all the reality in the enterprise, in biology, physics or any system, whatever it is. This is the fallacy that has prevented BPM to thrive in the past 10 years. Industrial Process Control (the world where I come from) on the other hand has been building state based formalisms since the 70s (at least) and there has never been any problem there. Manufacturing automation is a huge market compared to BPM…

But what do I know?

Again, I would prefer a BPMN example from one of you, or somebody else, something real, if possible.

“here are many companies that grew out of nothing when the market is big enough: Microsoft, Google, SalesForce,… We would know by now if only 50+ year old companies were the only one that can succeed.” Your definition of success is only commercial success for a vendor. You’re defining the rules, and which data points are allowed to make the point.

Look, Google was willing to sell to yahoo at one point and yahoo passed. Does that makes search suddenly not interesting of Google had sold instead? Lots of search companies came before Google and didn’t survive independently (altavista anyone?) – in fact, by the time Google came along everyone thought Search was kind of a waste of time for a commercial enterprise…

And then there is technology that has no successful commercial outcome – like browsers. Mozilla makes less than the biggest BPM players, and all that revenue is from ads, not selling browsers. Netscape is toast. Opera is vanishingly small. So, are browsers a failure or a success? How about HTML? Is there a firm that represents the success of HTML? Would it be a bad thing if BPM concepts became as ubiquitous as browsers? :)

The way I measure success is different than the way you measure it. I measure success by whether what we’re doing has value for the customer – real ROI. If we can produce it , then we’re successful. I know from previous discussion that you dislike this bottom-up thinking, but really it doesn’t make too much sense to lose sleep over things that are outside your zones of control or influence, and to focus your energies on what you can affect.

Moreover, I’ve seen focus on entity lifecycles to the exclusion of process run amok (as well as exclusive focus on process ignoring entities gone wrong). To the point that you lose sight of the fact that all this state / entity management is really to serve a purpose – fulfilling a customer request or order – and the measurements of success need to be around THAT and not around how fast you can update state or get a task completed. There’s no holy grail, there are only tools and brains to use it.

The military has an expression – equip the man, don’t man the equipment – and I think that is exactly the right mindset for producing value in an organization. Depending on the situation, you use a combination of tools to solve the problem.

I’ll leave the discussion of the unpredictable processes to you and keith :)

>> Moreover, I’ve seen focus on entity lifecycles to the exclusion of process
>> run amok (as well as exclusive focus on process ignoring entities gone
>> wrong)

Scott, this is precisely my point, it is just like when building an airplane you can neither ignore gravity nor fluid mechanics. The problem I see, for a decade now, is that people conveniently pretend that processes can be designed in complete abstraction of business entity lifecycles. Few people would argue the other way around, but it would be just as bad.

Could we now make progress and all work towards advancing the state of BPM?

yes, by all means :)