ESBs and integration – Looking back on 2011 and forward to 2012

2011 was a year of growth and development for the enterprise service bus (ESB). probed some players to get a sense of the year in ESBs.

For enterprise application integration, the enterprise service bus (ESB) is something that has clearly ‘arrived.’ The trend has been afoot for awhile, and shows few signs of slowing. According to IDC analyst Maureen Fleming, enterprise spending on ESBs grew 17% in 2010. That is “robust compared with almost all other classes of enterprise software,” she said. The ESB is gaining traction with SOA architects concentrating on application integration.

Although many ESB integration implementations are tactical and used to connect applications, the use of Web services with ESBs is also very common, according to Fleming. 2011 was also a year of growth and development for ESB technology, as open source versions came more into play, vendors added features including improved data integration capabilities and enterprise architects used ESBs more as a part of REST and mobile deployments.

“There is a realization that we shouldn’t wait for data anymore and that is having a positive effect on the adoption of ESBs.”
Ross MasonCTO and founder of MuleSoft

What are pertinent trends? probed some players to get a sense of the year in ESB/ integration. They saw mobile applications driving ESBs, ESBs taking a role in transforming cloud services, SOA gateways taking on ESB characteristics and ESBs moving more aggressively to the perimeters of the enterprise infrastructure.  

Looking back on the year, WSO2 Co-Founder & CTO, Paul Fremantle first describes a 2011 that saw a substantial shift to incorporate REST and JSON in the ESB.

“This is something we have seen driven by mobile applications. In many cases it is a mobile application that is making a simple JSON/REST call to the ESB, which is then doing lightweight orchestration of internal services,” including SOAP calls and legacy services, he explains. Freemantle also remarks on the growth in SOAP and XML traffic through the ESB, but says the rate of growth of JSON and REST has been still greater.

XML and SOAP are widely considered overkill for mobile clients, while JSON and REST seem right at home in that emerging space. Still, mobile clients make calls that ultimately invoke SOAP and XML on the backend.

“As a side effect of this, we've seen a lot of people looking at the traffic shaping capabilities of ESBs, including throttling, caching, metering and priority execution,” Fremantle says.

Additionally, Freemantle says there has been a move to offer and use the ESB in cloud computing settings. He describes seeing ''a strong interest in running integration in the cloud in a multi-tenant, elastic deployment'' and sees the financial sector leading the charge. “I believe that ESBs will need to support inherent multi-tenancy to work well in PaaS environments," he said. "I also expect to see a lot of companies focusing on internal and external APIs and using the ESB to manage, monitor and meter Web-based APIs,” he adds.

Finally, he says, the performance of ESBs has become more important. “At WSO2 and in Apache Synapse we've seen a lot of focus on performance of streaming and routing, and the asynchronous non-blocking model has been seen as key. This is because ESBs are being used more widely and traffic rates are increasing.''

Agile data source handling in the ESB/integration market

Jaime Ryan, partner solutions architect at Layer 7 Technologies, says he has seen two recent trends in the ESB market, both driven by the need for agility in the manner in which data sources and applications are connected and exposed. He says the driving factor is the explosion in social networking, user-generated content and an insatiable consumer demand for new and better ways to interact with a digital world.

That is encouraging businesses of all sizes to adapt and expand the reach of their applications. “This has manifested itself in two different ways, depending upon the size and requirements of the businesses in question: open source ESB technologies and SOA gateways as ESBs,” he explains.  As a result, according to Ryan, startups and tech-savvy small businesses have begun to recognize the power and flexibility of SOA in general and the ESB paradigm specifically. “They need to quickly roll out new interfaces and applications without a huge investment in technology or consulting, and are turning to open source ESB solutions to fill the gap,” he says.

"These small businesses are more likely to need simple standards-based integration with SaaS technologies than they need support for monolithic packaged applications and legacy mainframe formats,” he says.

For his part, Ken Johnson, middleware product manager at Red Hat, says, while the spotlight has been focused on ''big data'' and ''cloud" in 2011,  the ESB integration problem “has not magically solved itself.”

Indeed, ESB integration challenges abound and the need to do more with less is driving increased attention on integration. “The market is more mature and customers are more savvy than a few years back,” he says. For instance, SOA is now well understood and being widely used. Along with other integration techniques like EAI, EDA and process orchestration, SOA is being used by a more mainstream user. SOA is no longer just for the early adopter or “financially-advantaged” organizations, suggests Johnson.

According to Johnson, customers are now looking at integration problems more holistically, understanding that one class of products or technologies isn't necessarily enough to meet needs. “It's not about just using an ESB or just using Web services for example,” he says.

From the vantage point of FuseSource, CTO Rob Davies says, in 2011, the flexibility provided by ESBs has enabled people to take integration to the edges of their organization. “It is really about integration everywhere. We have reduced the cost barrier to reaching the edges, especially for retail businesses,” he says

“Open source is a key enabler for this because you are taking down the cost barrier and you have the flexibility,” he says. In particular Davies cites the example of Apache Camel because of its large community and available components.

ESBs and middleware meet real-time operations

Rounding out the discussion, Ross Mason, CTO and founder of MuleSoft says ESBs have arisen along with a focus on adopting middleware that is more lightweight and Agile.

He says another trend evident in 2011 has been a growing desire to process data in real-time. “We have seen a shift to a demand for processing large quantities of data that might have previously been done with an ETL tool,” he said. Rather than performing intra-day batch processing, people want to process as it happens, either because they want to react or because they simply want to monitor data, he suggests.

“There is a realization that we shouldn’t wait for data anymore and that is having a positive effect on the adoption of ESBs,” he adds.

“In 2012 we see that continuing. I see the line between ESBs and ETL blurring, the demands on either camp - the requirements - are becoming very similar.” he adds.

Looking ahead to 2012, Freemantle says he expects to see a much stronger focus on ESBs within cloud - especially in platform-as-a-service (PaaS) deployments, whether public or private.

Large enterprise ESB/integration factors

For larger enterprises, the move toward more agile ESB technologies is based on four driving factors, according to Jamie Ryan, partner solutions architect at Layer 7 Technologies. These are:

  • Cross-divisional integration of heterogeneous technologies due to mergers and corporate silos;
  • Secure partner interaction over the Internet;
  • Connectivity with applications and services deployed in the cloud; and
  • Mobile initiatives for extending business reach. 

“Traditional internal-only ESB deployments are no longer sufficient, as more data and applications need to be available beyond the four walls of the data center,” he says. And, the same basic ESB requirements still apply – intelligent routing, message format transformation, protocol mediation – but the security and governance of these newly-exposed external APIs become paramount.  “For this scenario, more enterprises are turning toward SOA gateways – technologies that have a proven track record of secure, managed integrations over unsecured channels,” he says.

Deployed as hardware, virtual or software appliances, they provide the same simple installation and quick time-to-market that open source ESB technologies offer, but with the security certifications required by the government, financial services, and retail customers. Furthermore, he adds, SOA gateway technologies have now “expanded the traditional ESB model to include the extended architecture of today’s modern enterprise; internal, partner, cloud and mobile are exposed through the same robust, policy-based interfaces and securely integrated to meet IT and business demands.”

Don't forget SOA integration

Fleming at IDC points out that the adoption of really sophisticated services-oriented integration utilities for standardized orchestration and composition is not hugely common but is growing, particularly in larger enterprises. More and more, orchestration is a part of the ESB equation.

Architecture design is important for overall throughput, she notes. The style of messaging should be associated with the goals and requirements of the architecture.

Event-driven architectures that enable event-driven processing -- that are services-oriented -- tend to provide good scalability and throughput but are not [as] broadly adopted as they actually should be. “Application infrastructure is needlessly expensive because of that,” she adds.

Similarly, Robin Bloor, Ph.D., Chief Analyst & Co-founder, The Bloor Group and founder, Bloor Research, points out that, when ESBs are used for enterprise application integration (EAI) it is often because a point-to-point approach has been determined to be insufficient. The choice of using an ESB often comes, he says, when the architecture begins to resemble a hub and spoke scheme. A final reason to take the ESB path: reuse of applications and services is tough to do without one, Bloor notes.

More on this topic

Dig Deeper on Topics Archive