Evaluate Weigh the pros and cons of technologies, products and projects you are considering.

Pushing the SOA boundaries: Including batch-driven applications in real-time integration projects

This tip examines an often neglected SOA adoption issue: handling integration with existing systems that can only be communicated with in a non-interactive, non-XML based manner.

The benefits of enterprise-wide SOA adoption have been widely discussed and understood. The challenge is how to reach this 'promised land' and meeting this challenge requires the usual combination of realism about how far and fast things should change, and the ability to map these changes back to business gains and problems solved.

In this article, I will focus on a specific but often neglected issue: handling integration with existing systems that can only be communicated with in a non-interactive, non-XML based manner, through batch files or even human requests delivered as Excel spreadsheets. Failure to take account of such systems will severely limit the scope of SOA adoption, leaving isolated 'islands of integration' -- and in doing so yet another integration technology that was intended to solve the issues of disconnected systems becomes a part of the problem.

Limiting the span of SOA
To date, there has been a single standard response to the difficulties associated with bringing existing technologies into the SOA world - use one of the web service enablement tools available in order to create a new interface to the existing system. However, this may not always be possible or desirable, usually for one of two reasons:
  • The application does not expose such interfaces or data, being instead batch driven and creating a report at specific time intervals.
  • The application belongs to another organization, and there is no justification for that organization to take the risk and cost of changing to an SOA. This last problem is typical in any B2B service provision scenario, when for example a service provider cannot mandate a single message format to its clients.

When we consider existing transport protocols, and particularly the business processes they support, we are confronted with a similar situation. The reality is that many of these are still batch based, even to the extent of using FTP or email to distribute non-XML documents such as Excel files. Rather than disrupting the entire organization, SOA implementations must accommodate and integrate - without necessarily enforcing change on the business processes involved.

Batch and beyond
Most corporate information is not yet Web service enabled – the most common route to SOA. Nobody would contest this point, but nevertheless the 'routes to enablement' favoured by the majority of vendors are the easy wins, such as the Web service enablement of Java applications. When the information that needs to be integrated within an SOA is actually stored in a csv file sitting in a directory, the solution is less straightforward. Unfortunately, this type of requirement is common, as is apparent from any appraisal of the range of integration projects undertaken within any enterprise.

This is not simply a data format issue: whilst the SOA vision often assumes that information will flow over HTTP or some form of reliable messaging based on JMS or the WS standards, in many real-world cases data is transported by ftp or even left in directories for collection. To put it another way, many business processes can involve data feeds delivered in ftp'd update files, for example a 'latest parts' list output from a legacy manufacturing system, that needs to be checked against the order before the order is submitted into a SOA friendly RPC style service exposing the ERP system.

Even when the legacy systems are interactive and not batch, they may be considered 'untouchable' – too important or fragile to be fiddled with for the purposes of Web service enablement. For instance, attaching a web service to a screen-driven application can have unforeseen consequences, as load dramatically increases through an application expecting human input – not real-time data feeds. Recognizing these risk factors, the benefits of SOA may not be enough to convince CIOs to change these existing systems.

However, even though we are faced with these challenges, we should not throw our hands in the air and give up – SOAs are perfectly capable of bringing these various systems and technologies under their wing – but it is important that the right approach is taken to SOA development and deployment.

Why bother?
It may seem bizarre to attempt to incorporate such types of data and batch driven processing into a SOA. But the need reflects a general principle that should always be uppermost in the minds of architects and CIOs – that the migration to SOA must be driven by business requirements, rather than technology for its own sake or a desire to deliver a 'cohesive' architecture.

At the same time, it should be understood that this does not undermine the essentially service-oriented nature of the SOA. Instead, in each case appropriate technology must be put in place to present existing application data in a way that can be used by these architectures, without changing the existing systems themselves.

Firstly, success will depend on how effectively the SOA can work alongside existing technologies. This means that the SOA must be capable of accessing whatever information, in whatever format, the application can expose. So while XML remains the "common format", and we can continue to leverage its unique properties to handle orchestration and mediation, SOA implementations must make the translation between non-XML and XML data formats as seamless as possible.

Secondly, the automation of business processes must be capable of including steps that are not Web services based. If business processes can only be automated when each step is represented by a neatly defined Web service, SOA implementations will never move beyond the prototype stage to tackle real world problems. Truly effective SOAs must incorporate capabilities such as directory polling, ftp and data format translation techniques to automatically pull data from relevant directories and convert it to the appropriate formats – without requiring any change in technology of business process on the other side of this transaction.

Building a real-time faÇade to batch
Returning to our specific issue, bridging between the non-XML, non-interactive systems and the real-time XML world of SOA is not, in fact, such a daunting problem. The core requirements, beyond the ability to create and deliver a service, are:
  • An ability to interact using many transport mechanisms - including directory polling, FTP and email
  • An ability to map between the original formats and the required SOA formats.

Ideally, this should be achievable with minimal programming, through configuration and with a degree of flexibility in terms of the effort required to support a new format.


Any new technologies and architectures must inevitably operate within the limitations imposed upon them by existing infrastructure and systems. A new architecture must do more than promise long-term gain only after shorter-term pain – it must be capable of encompassing the full range of interactions and data formats. The batch/non-XML example happens to be the most common case in point.

Once we are confident that the SOA can be extended across the enterprise, it is then possible to adopt SOA incrementally -- gradually broadening the SOA 'net' -- and delivering in a way that makes that co-existence with these existing technologies as simple and painless as possible.

Ronan Bradley is CEO of PolarLake

Dig Deeper on Topics Archive

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.