How to design a B2B system for integration and the cloud

Just when EAI seemed like the perfect solution for B2B management, integration and the cloud brought on a whole slew of challenges. Here we explore how business can adapt.

Enterprise application integration sought to bridge internal data silos, but B2B integration and the cloud bring...

challenges. Here we explore how today's B2B systems need to be designed in response to these challenges.

Recent changes in application and infrastructure design and deployment mean traditional enterprise applications are required to publish data in ways that can be consumed by a myriad of applications running both on premises and in-cloud. The resulting integration pain points can be addressed with a cloud-based, modular, publish-subscribe B2B system.

An unfortunate consequence of the post-mainframe era of client-server computing was the proliferation of data silos as enterprise applications migrated to dedicated systems that were often incapable of sharing data and typically by equally siloed IT teams. The fact that each system created information that could prove useful to other business applications spawned the enterprise application integration (EAI) business. Moving applications -- and their associated generation of and need for data to cloud infrastructure, whether private or public -- exacerbates the data sharing problem as applications become less monolithic and modular designs become the norm on virtual systems.

Meanwhile, in the modern B2B system, new, cloud-native applications often adopt a mashup design that combines code modules, network services and data sources. Design and deployment mean traditional enterprise applications are required to publish data in ways that can be consumed by a myriad of applications running both on premises and in-cloud.

Whether applications themselves move to the cloud or not, cloud services are a convenient way to solve most B2B integration problems.

In a sense, application data is becoming virtualized, existing as software-defined resources that can be programmatically accessed, collected and redistributed by other cloud services. Yet implementing a data virtualization strategy requires some sort of information broker that can access, interpret and republish the myriad sources and formats of enterprise data in ways that other applications can consume. It's a grandiose vision, but one already achievable using cloud-savvy data integration services designed to bridge the gaps between legacy, proprietary applications and their data formats and the world of cloud services and application program interfaces.

Data integration software or services in B2B settings generally include the following features:

  • Data source connectivity to databases, applications, Web services or files.
  • Data parsing and transformation, which typically involves turning source data into some intermediate format like XML and then rewriting it for a target format.
  • Metadata extraction and analysis that can discover data relationships, enable data mining and reporting, and facilitate more complex data modeling.
  • Design and modeling to allow building arbitrarily complex data transformations and support new applications.
  • B2B system management via a central console for usage, error and security reporting and system status.

Systems support one or more deployment options: on premises; in-cloud, user operated on infrastructure as a service (IaaS) or vendor-operated software as a service (SaaS); or hybrid.

Cloud-ready B2B integration services

Whether applications themselves move to the cloud or not, cloud services are a convenient way to solve most B2B integration problems. As data sources become more distributed, increasingly spanning internal and third-party applications and data providers, unifying data integration in the cloud is more efficient than knitting together disparate sources using private networks, virtual private networks and customized software. In these situations, a cloud approach can improve data integration processes and lower B2B system costs.

There are a multitude of vendors providing data integration products, and most incorporate the ability to both access cloud services and data and run on cloud infrastructure. The following are a selection of products and key features from a mix of mature, general-purpose EAI leaders and newer cloud-native specialists:

  • Denodo Platform: Offers what it calls a day that provides data connection, combination and publishing services for structured and unstructured data sources, both in-cloud and on premises.
  • Informatica: A leader in the EAI market, Informatica offers a number of cloud integration products that augment its on-premises software with connectors to popular SaaS products, like Salesforce and Workday, and IaaS services from AWS and Azure. The Informatica Platform is a complex suite of products so buyers need to do some homework to find the right mix.
  • IBM InfoSphere: Another data management platform that builds on a foundation of information integration features with modules for data transformation and refinement, replication, warehousing and big data (Hadoop, streaming data). Befitting Big Blue, there's probably no data integration problem it can't handle if you can master the complex portfolio and stomach the price.
  • Jitterbit Harmony: Provides cloud and application connectors for common services and enterprise applications that can be deployed as SaaS, on premises or hybrid. It also supports custom connectors to any SOAP or REST service and Open Database Connectivity and Java Database Connectivity (ODBC and JDBC) databases.
  • Oracle Data Integrator (ODI): A good choice for ingesting data into Oracle databases and applications with a full set of ETL (extract, transform, load), CDC (change data capture) and data replication features. ODI tightly integrates with other Oracle products including the GoldenGate environment real-time streaming data.
  • Snaplogic, Elastic Integration Pipeline: Befitting its name, Snaplogic decomposes the data integration problems into logical chunks, called Snaps, which can be mixed and matched using a graphical editor to create arbitrarily complex integration processes. Snaps come in four categories: file readers, XML parsers, XML formatters and file writers. Its system uses a software-defined network-like architecture that decouples command and control (control plane) from data access and processing (data plane). Snaplogic includes a large snap library for common applications and systems like Active Directory, Salesforce, SAP, Microsoft Dynamics, NetSuite and others.
  • Talend Open Studio for Data Integration: Includes more than 900 connectors and components that can be used in cloud-to-cloud (e.g. from SaaS apps like Salesforce or Box to the Talend cloud) or hybrid scenarios. Open source platform is both less expensive and easier for developers to test and extend than many alternatives.

Action plan

B2B integration products provide a rich and complex set of features that can solve most application needs; however, they are often overkill. The lesson then is don't overbuy and always look for the simplest, and cheapest, feasible B2B system. Remember: Don't use a food processor when a simple knife will do. For example, AWS services like Redshift and EBS can ingest SQL data using basic ODBC or JDBC commands.

For more complex needs, the choice will often be driven by existing internal systems; that is, use what you already have. Large enterprises with existing Oracle, IBM or Informatica systems should first try modules for these legacy EAI products that can extend them to cloud applications. Smaller organizations or greenfield projects should investigate newer, cloud-native data integration products that offer free trials and usage-based SaaS offerings.           

Next Steps

How big a role does EAI still play in a modern B2B system?

Access our CIO-centric guide to B2B integration

Explore the expanding role of REST in B2B integration

Dig Deeper on B2B integration