Why, when to use messaging middleware

Messaging middleware has shifted from integration with legacy systems to integration between any two systems.

What are the primary types of messaging middleware and use cases for them?

Todd BiskeTodd Biske

Of the many technology areas, middleware tends to be the kitchen sink. As the name suggests, it was originally used to describe technology that was inserted "in the middle" -- between end users and legacy/back office (typically mainframe) systems. Messaging middleware was a sub-area, strictly focused on the movement of information between systems, and not on the processing of that information by the source and receiving systems.

Over time, more things have been lumped into the middleware category, although the view shifted from being one of integration with legacy systems to a view of integration between any two systems. As such, the middleware space began to grow more sophisticated. Rather than simply providing dumb pipes, the technology expanded to perform processing on those messages between the two systems. So, let's try to provide taxonomy around the middleware that's out there today.

Communications and messaging middleware

I expanded this title because we need a category that works not just for topic-queue-based interactions, but for normal TCP/IP- and Hypertext Transfer Protocol-based messaging, as well. The reason for this is there is an increasing number of hardware-based products that can perform more intelligent actions, graying the lines between traditional networking infrastructure and software-based messaging systems. Additionally, software-based products have also moved toward the infrastructure space with increased use of virtualization and software-based networking. So, it's beneficial to have a common group that covers this space, rather than risk having your networking teams and your middleware teams driving to competing strategies.

Within this space, we have traditional messaging middleware, which includes IBM WebSphere MQ, Tibco's messaging products and any number of Java Messaging Service providers. These are normally topic- and queue-based interactions, very well-suited for asynchronous interactions, although they can also handle synchronous request-response-style interactions. These systems work well with mainframes and legacy systems that still don't play very well in the modern TCP/IP world. Asynchronous processing can play a key role in business process orchestrations that involve manual and automated activities, so messaging middleware is really a foundational element for most enterprises.

On the TCP/IP side of things lie communications middleware. I'm only going to include the more intelligent networking aspects of this. This includes load balancers, network optimization appliances and firewalls. These products can frequently look deeper than just the networking headers and apply more intelligent decisions to the messaging traffic.

Workflow and integration middleware

Application servers grew in popularity with the Web and are used today, although maybe not with the same formality that existed in the J2EE (now Java EE) heydays. Simply put, these are general-purpose systems for building almost any kind of application. Many of these systems include libraries that allow users to build systems that act as an integration layer between other systems, but their main purpose is to build new things.

Of many of the technology areas, middleware tends to be the kitchen sink.

Next, we have the more targeted integration platforms. The metaphor for development is different than with the general-purpose platforms. Here, the intent is to work from adapters that know how to talk to a variety of systems, and focus on building the glue that connects adapters together.

While core messaging middleware established pipes for communication, these systems are intended to deal with the information aspects of integration: transforming between messaging formats, aggregating data from multiple sources, and orchestrating interactions between multiple systems. These systems are more transactional in nature, largely dealing with stateless interactions and likely originated by the end user. Technologies like enterprise application integration tools, enterprise service bus, Business Process Execution Language engines and even integration appliances, fall into this category.

A more specialized area is workflow systems. The primary add-on is that workflow systems deal with state. These are long-running processes involving interactions with many systems (and likely people).

There can be overlap with workflow systems and traditional batch-scheduling systems. These systems' core focus is orchestration and keeping track of where things are in a long running process. The workflow system may not perform any processing itself, but instead will act as the master delegator to the people, integration platforms or general purpose computing platforms that handle the units of work required.

Data middleware

In any large enterprise there is a need to move large amounts of data between different data repositories, and likely between partner companies. While it would be great to have a single location for data that served all purposes, that's simply not the case. In the data middleware space, two areas I want to call out are extract, transform, load (ETL) technologies and change data capture (CDC) technologies.

ETL technologies are about bulk data movement between two data systems, which consist of three steps:

  1. Extracting the data from the source system;
  2. Transforming data into the format for the destination system; and
  3. Loading the data into the destination system.

There's certainly overlap with the integration middleware space, especially when dealing with more real-time integration than batch integration. Many integration services simply take data in one format, transform it to another, and pass it to the destination system for processing. That certainly sounds a lot like ETL, doesn't it? The common distinction is that ETL tools are normally dealing with data repositories on the two sides of the integration equation, when integration middleware deals with processing systems.

CDC technologies are about the steps before ETL. In order to know what data to extract and load into the destination system, you need to know what has changed. CDC tools capture this information over the course of a time period.

Hopefully this has served as a good overview and potential taxonomy for you to use when organizing the technologies used at your company. Keep in mind that vendors sell their products based on features, and the products out there may not cleanly fit into one bucket. What's important is to organize things in a way that makes sense for your organization and can help guide you to the right portfolio for your needs.

About the author:
Todd Biske is an enterprise architect with a global publishing company, working out of their office in the St. Louis Metro Area. He writes a blog about BPM and SOA and is the author of SOA Governance from PackT publishing.

Follow us on Twitter @SearchSOA and like us on Facebook. 

Dig Deeper on Enterprise application integration