I think Paul Vincent’s blog on CEP is where it’s at – my favorite place to keep tabs on this interesting space.  It isn’t BPM, but it is related to BPM, and vice versa.  Not only are the markets complementary, sometimes the concepts in one space directly make sense in the other.  A side note: often when having trouble designing a process, I think about the important business (or even system) events that might occur – I may not know the actions that happen inbetween, but the events become useful markers around which to design a process.

A recent post from Paul caught my attention: “Do Events Have Duration“.

This might seem an odd question, but there are 2 schools of thought here:

• An event is a point in time.
• An event is an activity that can take a period of time.

Taking the latter view, we can say things like:

Earthquake MMM took place from date-time D1 to date-time D2
Fraudulant act on account NNN took place from date-time D3 to date-time D4
Flight QQ123 was delayed in the period date-time D3 to date-time D4, when it completed disembarkation time T1 late
World War 2, an historical event, took place from 1 September 1939 to 2 September 1945.

Lest this make your head hurt, Paul makes a compelling argument that events should be considered moments in time, and an “event” like World War II is really more of a “state” of being in World War II with a start and an end and many other events in between.

I think the principle of Occam’s razor applies here: the simplest explanation is usually best.

Another side note: my favorite comment on his blog post is one that almost sounds like a snippet from a Star Trek script: “…’temporal semantics of event processing’. In short — events always have durations, but in many cases, the time granularity that we are interested in a certain application is large enough so that we can approximate the duration to a time point (note that a ‘second’ is also an interval and not a point, since time is not discrete).”  I was waiting for something about reconfiguring the deflector shield…

Ok ok, what could possibly be interesting about Hadoop-based systems and Mathematics?

Well, it sounds fancier to say Hadoop-based system… but really this basic math applies to any batch-oriented system and those of us who have been writing batch processing solutions now-and-then for the last 20 years should at least be intuitively aware of the math presented in this article, if not consciously thinking about it at design time.

The key equation:

Runtime = Overhead / (1 – {Time to process one hour of data})

Or, stated differently:

Runtime = Overhead + {Time to process one hour of data} * {Hours of data}

Where hours of data and runtime are equal.  These equations help explain why a perfectly healthy batch processing system can suddenly fall tragically behind – if the time to process an hour’s worth of new information is greater than an hour, you have a problem – and the problem will just keep getting worse until you:

• Improve the runtime of the algorithm
• Apply more resources to your server / cluster.
• Filter the incoming data better (if possible) to improve your signal to noise ratio and thereby eliminate unnecessary data processing

In the BPM and CEP worlds, often that third bullet is a key element to improving performance – it doesn’t require more hardware – it just requires you to move your filter “upstream” from your BPM infrastructure to your EAI infrastructure or your ESB infrastructure… or from your EAI/ESB infrastructure to the source of the noise… Some would say this is squeezing the balloon, moving the bottleneck elsewhere, but actually, filtering better up stream may make those systems more efficient as well (if generating the payload and calling out to a webservice or ESB requires cycles, then not doing it as a result of an efficient filter may well reduce processing cycles… )

At any rate, its a good read.  Andrew Paier, figured you especially would get a kick out of this article, given our experience back in 2003…