Sandy has posted a pretty good presentation on mixing rules and process, which pretty well captures how I feel about the subject.  I’ve never understood why the rules-vendors out there try to model the process using rules.  On the flip side, pure BPMS vendors sometimes fall into the trap of feeling they have to claim to have rules engines because the rules folks will try to claim that they have BPM.  I think customers who are interested in both BPM and BR functionality could do themselves a favor by telling vendors up front that they are using separate packages for this functionality - they’re likely to get more candid answers from the sales folks from both companies, as it allows the vendors to play to their respective strengths.

I’ve deployed several projects that integrated a BPM platform with a rules product, and its just easy to do via webservices or an API call in all but the most extreme cases.  Anyway, enough from me, here’s a link to Sandy’s post.

Disclaimer:  I used to work for a company that did a lot of work in the configuration space, which has a pretty big overlap with rules.  We did heuristic search, constraint satisfaction, resource allocation and pooling, spatial constraints, containment, and we even did massive rule systems that were super fast.  Intellectually it was a very interesting field because you take really hard problems (in some cases, problems that you could demonstrate were NP Hard problems) and finding “reasonably optimal” solutions in a very finite amount of time.  As I said, intellectually very stimulating.  In other cases, it was coming up with very creative ways to use simple rule-based systems to compute very user-friendly user-interfaces in millisecond time against very large rule bases.  But one thing I learned for sure: thinking about the world as a set of configuration logic or rules is a different way of thinking, and it just isn’t intended for the average Bear.  This is why I don’t see representing “everything as rules” as being a terribly useful way of approaching the world when it comes to involving your organization in the process of business process management or improvement.  I consider myself a reformed rules guy, and now, tongue-in-cheek, I see everything as a process!

Update: On a (somewhat) related note, a somewhat humorous post from Jim Sinur on how rules might have initiated the real-estate meltdown.

A business unit commissions a solution that involves people, decision points, multiple paths and real-time visibility. Sounds like a business process, right? Sounds like a perfect fit for Business Process Management; more specifically, a BPMN rendering and a BPMS tool to implement and execute the vision.

Some key aspects in this solution are:

• People
• Business Decisions
• Multiple paths sometimes in parallel
• Point to point orchestration
• Metrics

These are all aspects that support and imply a BPM solution.

Now, what if I needed a similar solution, but minus the people? What if I had a batch or back-office solution that did not involve people other than an occasional exception path? What if this batch solution was expected to process thousands of transactions daily? Would this still be a BPM solution or would it be a software program designed specifically for this purpose? I have heard many, even in the BPM field; argue that such a batch solution may not be suited for a BPMS implementation. That BPMS tools are not geared for this type of high-volume fast paced execution required of such a batch oriented solution. However, to that I say why not?

You see, just because “People” are not necessarily involved in activity steps within the process, the process is no less a business artifact than those which do involve people. Now surely I’m not advocating that every program written could otherwise be a BPM solution, but I am saying that many of these so-called batch solutions do indeed involve business decisions, multiple paths, and multiple point-to-point system-to-system communications; all of which require sophisticated orchestration and most certainly should have adequate real-time metrics. These solutions are often augmented and evaluated for efficiency and accuracy just like that of people oriented BPM solutions. Often these batch solutions do need to handle exception paths which may involve people. Should the program kick-out on exception to some other process solution? And if so, how do the solution owners manage the big picture of the process flow? More times than not, these batch solutions must maintain state in some manner. They must be prepared to handle system failures and contingencies. Is this all to be written rigidly into a program somewhere with very little visibility?

I also find that many BPM enthusiasts tout automation as one of the goals of BPM in a march towards efficiency. I agree with that sentiment, but once a process has been improved upon to the point of automation, is it no longer a process? Should it then be re-factored into a compiled program of sorts? I think not; for automation left unattended is a dangerous thing. An automated solution still requires checks and balances, reported metrics, and proof that the efficiency gains expected are realized; and what about improvements, new products that adjust the automation, new regulatory requirements that incur change in the solution? Even when fully automated, the business process still remains in the fore front.

Software programs and compiled, efficient code certainly has its purpose, and after all, these BPMS tools wouldn’t run without it, but when it comes to business visibility, orchestration, state management and multiple touch points; whether people are involved or not, Process Management solutions are quite applicable. And as these program interfaces become more and more modular through an SOA discipline, the orchestration and process management of those modules and services at run-time becomes increasingly important. So I challenge the BPMS vendors to stay focused on the process vision, but don’t stop at Mortgage Apps, Claims Adjudication, and Book Order solutions …… think further, faster, and with greater depth than ever before; I want to manage all my business solutions through a well defined, visible, and deployed process!

Thanks to Sandy’s blog post, I’ve once again been pointed to an interesting post in a blog I didn’t have in my Google reader (I’ve now added it!  thanks again Sandy!), in which Ismael Ghalimi gives unsolicited advice to the BPMS competition out there.  I can sum it up as:  “Use our BPEL engine!”

I want to discuss this theme in three parts:  The Case for Change, The Proposed Solution, The Proposed Benefits.

The Case For Change. The economy is hurting, and logically software vendors selling perpetual licenses will be commensurately hurt by reduced purchasing plans.  It isn’t a bad premise, though I will point out a couple of things.  During a slowdown in the early 90’s, software that had good ROI sold at accelerated rates, while software without large ROI stagnated or didn’t sell at all.  Products that extracted cost from expensive operations enjoyed great success.  On the other hand, during the 2001-2003 slowdown, I saw something else happen.  Instead of buying software to make complex processes and operations more efficient, I saw companies dramatically simplify their processes or products to eliminate the need for software (for example, PC and Server configurations became dramatically simpler, and fewer configurations were offered.  The same thing happened in telecommunications.  Complicated sales processes got streamlined).  I can’t say whether the current economic environment will spur more software sales in BPM or less… I have visibility to the deployment side, and companies that already own BPM are looking to get more ROI out of that software $.  But for those without BPM software, I’m not sure that they can effect the same kinds of rapid ROI that BPM platforms provide, and therefore I’m not sure how many of them will back off of plans to acquire BPMS packages.  Ismael asserts that many of their purchase plans will change.  I’m not so sure. If I look to 2003 and 2004 as examples, IT departments dramatically cut spending in the 4th quarter to preserve earnings for the current year (and yet, it didn’t stop BPMS vendors from selling in those quarters).  In the new year, however, priorities were reassessed and projects that were deemed to have a large probability of a large ROI were started. I can’t say for sure how this year will play out yet for BPMS vendors.

So, I’m not sure I buy the economic argument. But I do buy that replacing proprietary solutions with open source solutions or de facto standard solutions makes sense. Why? well, you focus your engineering efforts on the parts that differentiate you, and you rely on the Open Source community to provide for the plumbing elements.  If you find a bug, you fix it, build it, submit it back to the community, and benefit from gradually increasing quality of the overall solution.  Ismael uses the Database example, but I’d say this analogy isn’t perfect for his purposes, because the “winners” in the database space have been primarily licensed software vendors. A better example for BPMS solutions would be the J2EE stack.  Once upon a time it would have been hard to avoid using Weblogic or Websphere, but JBOSS has become a viable solution, and it doesn’t have additional licensing costs (though, the support contracts are actually quite expensive at present).  And you certainly wouldn’t want to write your own J2EE or LAMP stack for your BPM product - you want to focus on BPM and let someone else solve those interesting platform problems for you.

The Proposed Solution. Ismael proposes embedding (OEMing) Intalio’s BPEL engine in competitor products. In terms of his Blog post, it looks like “BPEL Engine” and process engine are being equated.  However, I think that several of the vendors use something closer tied to the parallel processing notation BPMN, rather than to the XML notation BPEL (there is a brief reference in his post that BPEL’s pi-calculus is better than Petri-nets, but without any supporting data directly in the blog post, since I think that was a bit of a tangent… and certainly doesn’t fit in this post!).  So while they may use a conversion of BPMN to BPEL and then in turn use a BPEL engine, it isn’t necessarily the case (they may, in fact, have a BPMN engine using some other technology).  This introduces some friction for vendors to adopt Intalio’s BPEL engine as a literal replacement for their own processing engines.  Nevertheless, I’m sure some vendors would benefit from getting a BPEL engine embedded even if it isn’t their main processing engine… (and therefore, worth it for Intalio to pursue such agreements where possible)

Looking at his database analogy… This isn’t the same as deciding not to build your own DB and “outsourcing” this technology to Oracle, Sybase, or MySQL… Its actually the equivalent of Oracle deciding to replace their Database Engine with MySQL or PostgreSQL on the grounds that they are cheaper and scalable and times are tough so Oracle could save some money by doing so… but read on for why that may not be their decision…

The Proposed Benefits. No more investment in “stack” technology, pick up a scalable BPEL engine.  The key here is, does the BPEL engine Intalio produces eliminate any competitive advantage or perceived advantage, that the other BPMS vendors have vis-a-vis the competition due to their own respective process engines?  I can’t answer that question for the BPMS vendors, but if we go back to the Oracle analogy, I think Oracle has done quite well (so far) by sticking to their guns and their own DB engine.  There are switching costs that can’t be underestimated too badly - its costly to switch someone from one Database to another… not to mention the monumental task Oracle would have of switching their own software stack to run on an open source database.  Similar friction would exist for the BPMS vendors to switch their process engines out for a BPEL engine.

Conclusion. So, it isn’t that I think this is a bad idea, I just think the bar may be really high, and I’m not (yet) in a position to judge if Intalio passes the bar (or if the other vendor solutions don’t set the bar high enough).  Ismael and Intalio should keep pushing this angle though, with the BPM vendors.  I don’t think any of them will be picking up the phone to call Intalio, as Ismael suggests, but I would suggest he approach them about co-development and OEM agreements.  Having the industry solidify around a common process engine would be a good thing, assuming that process engine is truly superior to all the engines out there today, and might help accelerate adoption of certain standards and innovations.  It would make it easier to expose useful APIs to the developer world (Ismael has some posts about building for developers that are pretty good too), it would make it easier to write solutions that will work across more than one BPMS vendor.  And it would make it easier to push BPMS education down into the college levels at some point.  Still, it looks like a tough hill to climb from here.  Keep fighting the good fight, Ismael.

The case for modeling without thought of execution…

I recently came across a blog entry from IDS Scheer on their Aris BPM Blog. Thanks to Sandy Kemsley for pointing me to it from her blog. Upon first read of the article by Sebastian Stein, I was struck by the difference in perspective between those who implement processes and those who model them.

For those who model (Modelers), the Model is the chief output and goal. Having a Model that will survive the test of time is the goal. You can see that bias throughout the post. In fact, the core philosophy is embodied right here:

“A business process model, depicted in one of the popular notations like BPMN or EPC, should not contain any technical details. If the underlying IT infrastructure or implementation technology changes, the business process model should remain stable. Your warning bells should ring if you have to change your business process just because you changed the implementation technology used.”

The two key points:

1. No technical details
2. stable with respect to technology changes

Something Overlooked by a Model-only Perspective…

But there are some problems with this… First, all the BPMN/BPMS tools that I have worked with support layering of processes. This layering allows the user to create a model that reflects Business sensibilities at the top layer, and if needed, several layers down in detail. So, if your need is to model something without “any technical details” you are not prevented from doing so in the BPMN-oriented tools that I’ve used.

Second, when you get to a certain level of detail, the process design should be informed by Technology. How so? It is important to understand if a transition is a manual or an automated one. Is it a non-value-added manual step? Then generally we want to automate it, or ideally remove it. Value-added manual step? Then generally we want to optimize around its constraints, but automation won’t be the goal. However, we may want to use technology to reduce errors, to improve time-to-execute, etc. In the posting, Sebastian doesn’t go into detail as to what he considers a “technical detail”, but it does beg the question: what is too technical? How about input and output data from a step in the process? These are critical process design considerations (if you know that a piece of data is required as an input, but you’re not sure where it comes from, you have a problem to resolve in your process design. And those inputs and outputs help define the “contract” of an activity or subprocess (or even of the entire process).

Third, Modeling tools today make it exceedingly easy to change a Model to adapt to Process changes. While it seems like a good idea to have a Model that is “stable” with respect to technology changes - the fact is, business processes change faster and more often than the technologies and systems that support them. The real problem isn’t keeping the Process consistent across technology changes - the problem is that the underlying technology may not be flexible enough to adapt to the new process model! At the least, the technology layer is often not agile enough to do so at a sufficiently affordable price and on a sufficiently short timeline (unless of course, that process technology layer is a good BPMS).

Fourth, the resilience that one needs, truly, is with respect to performance data. Performance data analysis is what will drive my process improvement activities, or identifying a process operating outside control limits. I need to be able to compare the performance of my process now to the performance of my process next year, to the performance of the process last year… If my process changes dramatically, how do I do that? Note: I’m not saying the technology changed. The process changed. So what I need is a way to track data that will make sense even in the face of relatively substantial changes in my process. BPMS tools can provide this facility, either baked in or via smart modeling practices, by taking snapshots of data at key milestones in the process that are not likely to change, semantically, even while the syntax (specific steps) of the process may change. To this end, even though the order entry portion of the process may change dramatically, you can still track information around the # of orders in, the value of those orders, the time it takes to process them, etc. even though the order entry process may go from highly manual to highly automated to web-self-service (or may yet encompass all three).

How do we Sum it up?

So the argument is that a modeling-only tool buys you a benefit (stability against technical change) that you don’t need, while not providing a benefit (technical agility with respect to business process changes) that you do need… yet still doesn’t address the key stability need -that of the measured process performance data. Moreover, the integration from most modeling tools to an actual functioning BPMS is, for the most part, non-existent from a practical perspective. Even when that integration exists, it is usually lacking process execution sensibilities in the model. There is a difference between drawing a model that represents the business needs and drawing one that can NOT be executed because of ambiguities and inconsistencies. For the best integrations I’ve seen so far, the products and the integration are all written by one vendor. (I’m definitely interested in seeing examples of this kind of tooling and integration and I’d be happy to write up reviews for such)

I’ve actually written an import to a BPMS suite using an Aris model as a starting point - and its hard!  There is a ton of non-relevant data in the export - positioning information, for example - and other information you need is difficult to lay hands on (roles/ownership).  To be fair, this wasn’t a BPMN diagram in Aris, but it WAS a diagram of a process, in a very unstructured environment.  It wasn’t any easier than parsing it out of Visio vdx files.  My recommendation, is that if you are given a process modeled in a modeling only tool - your first instinct should be to redraw that process in your execution modeling environment rather than try to import it (unless the importer ships with your product, in which case, give it a try!).  You’ll be surprised how fast you can recreate the model in your execution environment.

Now what?  Does an Execution-Oriented Model still make sense?

Okay. Given the arguments Sebastian presents, it seems he is suggesting that if you don’t know what product you will use to implement, you should use Aris to model your process (in fairness, if you don’t know what execution environment you will use, paper, visio, and Aris are all good options). And that, because it is “agnostic” with respect to the implementation tool you use, there is some derived benefit (this is really the point I disagree with). However, if you are going to build your solution in a completely different toolset, and you accept my premise that exports out of Aris (and other modeling tools) into execution BPMS suites leave a great deal to be desired, then you come to an interesting crossroads. Is he suggesting that once given an Aris model we should just write BPEL xml or some Java code to implement the process? or that we should then use a BPMN-oriented modeling suite to re-model and then implement the process?

In our experience, just “writing code” to codify a process in a modeling tool is a mistake. For one, how can the business determine if you have faithfully reproduced the process in your code? Extensive usability / UAT testing might reveal an answer, but it is a very expensive way to find out, and it only happens after all the code is written - and any mistakes will be very expensive to fix at this point because they could be simple mistakes or they could be conception or foundational mistakes. An Agile development process can help, but many organizations have trouble carrying off this approach with traditional software tools. If the technical team uses a BPMN execution environment (a BPMS) to build that process, then the business will be able to see the process in BPMN, a language (drawing) that they can understand, and understand the semantics thereof. By visually inspecting the design, the business can eliminate the greatest proportion of future defects at the earliest part of the design phase. And the technical team will implement each portion of the process in context of the business process at that point. And that is critical for providing useful business context to the technical team at the time they most need it.

Which Model is the Master?

And finally, now that your Process is implemented in an execution-oriented BPMS, as well as modeled in your modeling-only environment… which Model is the “Master”? Of course, you can make either answer work.  But let’s be clear about the choice you make :

Option 1:  The Model as drawn by the business in the modeling tool is the master.  it does NOT reflect what is actually happening in the business, or within IT, but it does show what the business was hoping the process would look like when the project started.  (Optionally, it may have even been revised and updated at the end to reflect some of the changes that implementation and testing revealed needed to be made).

Option 2:  The Model that works as agreed to by IT and the Business, drawn and executed in the BPMS environment.  This is the model that was actually tested by business users in UAT, by Unit Testing in IT, and system testing in IT.  This is the model that is actually running your business process in production, and it reflects reality.

Is it important that your original Model is resilient to technology change in this context?  Is it relevant that your model doesn’t have any technical details in it?

Or does it seem to be more interesting that there is now a BPMN model that represents what actually runs in your business every day, that can be measured and analyzed over time.  Does it matter that this BPMS is resilient to back-end technology changes (activities provide abstraction to what type of integration, and each integration can provide abstraction as to what specific systems are being tapped)?  Does it matter that this BPMS can support relatively rapid changes in process to adapt to your real business?  Does it matter that you can map the data you are tracking to your Model, to generate heat maps and highlight problem areas?

Well, you can guess where our heads are at.  Modeling is important, but Execution makes it relevant to the bottom line, and makes the Model itself more valuable.  If you want help turning your models into reality, we can help.

Like everyone else, I’m reading a lot of press about the economy, and most of it is negative. But in our little bubble of BPM, things are cooking along pretty well. Sure there are ups and downs but the trend-lines are up.

As if to underscore that, we’re starting to see the 1H2008 press releases from BPMS vendors. Lombardi’s is here (not to mention that it was well-covered by Bruce Silver’s blog, Sandy’s Blog, and the BPM in Action blog) and Savvion just released theirs here (if other vendors posted similar announcements I haven’t seen them yet). We’ve also seen some interesting investment news from Appian (a $10M investment round). Lombardi points to some impressive growth, especially regarding the license bookings (85% growth year over year). All indications are that the number could actually accelerate next year since they have seeded a much larger user base with Blueprint (their SaaS process mapping tool)- users that are likely to have positive impressions of the software and that may translate into more buys for the execution software (Teamworks).

Savvion also points out some nice gains- most profitable quarter yet (operating profit). However, in comparison to the Lombardi release it lacks a lot of detail. We don’t know how much profit grew year over year or by how much it exceeds previous profit numbers. And we don’t know how they’re doing on key metrics that might point the path toward growth (license growth, maintenance revenue renewals, consulting $, etc.). Not that either of these companies are required to disclose numbers, as they’re private firms, but the amount of disclosure from Lombardi is clearly greater so far. That speaks volumes about their confidence vis-a-vis the other pure-play vendors which are not putting out the same level of detail. And it puts pressure on the other guys to start putting up the same details. Hopefully with the Metastorm IPO filing, we’ll start getting at least two sets of more complete numbers to look at for trends.

Regardless of the shakeout in BPMS vendors, we believe the BPM segment is growing - the need for Process-oriented services is growing.  And we’re well-equipped to help customers with process-oriented solutions.  However, as an independent consulting firm, we keep our eyes open for this kind of trending data because we want to be playing good defense - we want to be where the ball is likely to go. That’s where the work will be, and that’s where we’ll make our investments. As a consultancy, we’d like to see all of these vendors grow at the kind of rates Lombardi is growing at, but we don’t have the numbers yet to know if Lombardi is typical, or the standout.  I think we have the most experienced Lombardi BPMS implementation team on the planet right now, but that’s another subject, and another post!