Manage Learn to apply best practices and optimize your operations.

Event stream processing & business agility

Mark Palmer discusses a new class of software called event stream processing (ESP) and why it is hailed as the "next big thing" by both software vendors and analysts.

The quest for agility has spurred the recent rise of the adoption of service oriented architecture (SOA) and the face of modern IT integration architecture is changing. Technology stovepipes of the past are now being connected by enterprise service bus (ESB) technology, which provides the backbone for networking, communication, mediation, and service container management needed to support an SOA. Every integration software vendor provides some form of ESB within their products and the ESB has risen to the status of de facto standard for integration of service-oriented applications. But what's the next step in the evolution of the IT integration fabric?

The next step is a new class of software called event stream processing (ESP). ESP has been hailed as the "next big thing" by both software vendors and analysts because it helps an SOA integration fabric become intelligent and responsive. ESP enables a business to think differently about its operations and IT infrastructure because it can understand the state of a business in the now, rather than just in the past.

But aren't ESB's already emitting events and aren't all SOA infrastructure elements capable of emitting and carrying events? The answer is "yes," ESB's are event-capable today. However, they don't prescribe what to do with the events that are emitted by their services. That is the critical value that ESP brings.

ESP enables an event-driven SOA to decipher event patterns (if A is followed by B followed by C), temporal (within 4 seconds), and spatial (within 10 feet) relationships among events – and can do so in real time. This allows a business to continuously analyze key performance indicators in real-time, identify threat and opportunity in real-time, and act instantly. These capabilities require a new style of computing - stream computing – that can deliver the missing link between an event-driven SOA and real-time business insight.

ESP – Finding ESB Event Patterns Using Causality and Temporal Constraints

To illustrate how the ESB and ESP can work together, let's examine a specific example: credit card fraud detection. The goal is to monitor purchase activity within the system and capture authorization requests that can be analyzed in real time to detect fraudulent activity. Such operations fully express the three stage event processing paradigm of 1) monitor, 2) analyze, and 3) act.

First, we need electronic access to the events on the ESB. Source event streams, shown below as "Merchant A" and "Merchant B" represent purchase activity events delivered via messages on the ESB to the Event Engine.

Second, we need rules that act upon those events. Since ESP engines process events asynchronously, events can come from anywhere, be of any type, and be received in any order. An event processing language (EPL) uses event attributes, their time of occurrence, and any inferred causal relationship among events, as its fundamental elements, rather than the structured data and relational algebra of SQL. An EPL processes queries "partially" – for example when A and B are true, then if C occurs within N seconds, take some action. Detecting event patterns in real-time help applications identify threats or opportunities that arise in a business instantaneously, like opportunities to buy and sell stocks in real-time, to automate a manufacturing shop floor, or to detect credit card fraud. An example of an event-based rule is shown below.

This EPL code begins with an event filter "ON credit_payment (user)". This statement instructs the engine to monitor events on the ESB that represent charges to a credit card. As events flow through the ESB, this event pattern is partially satisfied. Next, the "FOLLOWED-BY" statement instructs the ESP engine to monitor subsequent credit_payment events against the same account "user". If three charges occur within two minutes, then the code has identified a potentially fraudulent activity. Though not depicted above, an example of inclusion of "spatial" logic might be to flag any charges that fall outside a pattern of purchases geographically or flag a subsequent charge if the geographic distance between the purchase locations suggests the legitimacy of one of the charges is improbable, if not impossible. These examples illustrate the first central concept of ESP: the inference of causality. ESP has inferred from the relationship amongst the charges the possibility that one or more of the charges were caused by fraud.

The "WITHIN" statement illustrates another key concept in ESP: time. In this example, if the third credit_payment event is not detected within two minutes of the first charge, the activity is not flagged as potentially fraudulent, and the scenario completes. In stream computing, as in the real-time, agile enterprise, the significance of any individual event – in terms of business importance – quickly depreciates. The window of opportunity to act upon an event is often brief and transient. Unless the event processing architecture can rapidly discern the significance and respond, the opportunity to exploit the situation will quickly pass, with circumstances altered by subsequent events or other factors.

Finally, we see the third key element of ESP: action. Automated systems like a credit fraud detection application often invoke event-driven action once a pattern has been detected. In this example, the current request for a charge is declined and the account is flagged for fraud management action by sending a new derived event on the ESB.


As the enterprise service bus continues to become the backbone of the enterprise IT integration infrastructure, it provides a stream of events that make real-time insight a reality. Stream computing, and the ESP tools that enables it, provides the ability to detect time, cause, and spatial-based patterns among events in the ESB stream. By combining ESP rules with an ESB-normalized integration fabric, the enterprise can become truly agile.

Dig Deeper on Topics Archive

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.