Manage Learn to apply best practices and optimize your operations.

How to solve operational BI problems with SOA middleware

From focusing to the problem at hand to thinking in terms of events, there are ways to use middleware to facilitate fixing operational BI glitches.

While traditional business intelligence (BI) focused on aggregation and improved historical data analysis, operational BI attempts to improve the decision-making process for frontline workers. This requires a different focus on taking data from real-time systems to help improve this process. A good service-oriented architecture infrastructure can help bring data together in a way that support s this as new insights arise.

puzzle piece fitting in

"The operational BI system has to use the data about the customer and the context to give good advice to the frontline worker," said James Taylor, principal consultant of decision management solutions, an analytics consultancy. "It can provide massive leverage, because any decision you make can be leveraged across employees."

In the operational context, training is lower and these workers don't have the same ability to make good judgments as an analyst. The value is even higher when you start targeting these workers. "You want a new hire on the phone with a customer to be able to respond appropriately," Taylor noted.

Focus on the problem

The first thing to remember is that operations analytics is not a wide open exploration of data; there is a focus on a business problem. Operations analytics provides a benefit to a specific issue that can lead to smarter procedures or more efficient use of resources.

In traditional BI, the focus is often on setting up a big data warehouse to answer any question. Operational BI, however, is about finding an answer to a specific business problem, usually for a nontechnical user. This means you don't give an analyst a tool to try out. You have to think about giving the nontechnical worker an answer, and make sure that answer is relevant and worth giving as a response on a regular basis.

Think in terms of events

With operational BI, the infrastructure is more event-oriented and needs to be able to pick up on real-time transactions. The infrastructure required is more like application integration than traditional data integration, according to Jake Freivald, vice president of corporate marketing for Information Builders. "We have seen a convergence between operational BI and traditional BI. We do see them coming together where historical BI is used to create a baseline for operational BI," he said. "A key part of this is the use of middleware that incorporates BI on top."

An enterprise architect has to begin with the end user in mind instead of the analyst.

Freivald sees more organizations willing to forego data warehouses than in the past. There is also more openness to the idea that you will not necessarily have a data warehouse. This means less focus on the tools and more on the SOA infrastructure required for supporting this style of integration.

Traditional BI is more concerned with loading the data warehouse, whereas operational BI is about creating process flows that add intelligence into operations for a particular business problem. Operational BI zeros in on one transaction and is concerned with data flow, rather than the information being instantiated in a data warehouse. After the transaction, this data can be dumped.

Real-time cleansing

Organizations need to make sure they are working with good data if they want to arrive at useful decisions. "Data quality used to be something that would happen on the way to the data warehouse," Freivald said. "But if you are not using a data warehouse, it needs to be cleaned in real time to be relevant to the operational systems."

A single transaction is not enough to make decisions on. There needs to be an enrichment process to make the data useful. With ETL (extract, transform, load), the enrichment process is easy. The system gathers results, marries them and sends them to the data warehouse.

With operational BI, the enrichment process happens in real time. For example, an application may have to reach into the systems to see if a particular transaction is fraudulent, such as comparing how many have occurred in a given week versus the customer's history. It has to evaluate how this compares with the predictive model to determine if a certain transaction is likely to be fraudulent, such as a customer relationship management or trade clearing, or you may need to go to a cloud-based resource.

The concept of a data quality firewall can help address this. "In the same way that a security firewall protects corporate systems from hackers, a data quality firewall protects operational BI systems from dirty data," Freivald said. The data quality firewall creates an infrastructure to clean, correlate and enrich transactions on the way into high-value systems. This makes is possible to do operational BI without worrying about the quality of the data to make decisions.

Focus on messaging instead of files

Another key element to building successful operational BI systems lies in thinking about messaging, rather than files. Enterprise architects need to think about middleware instead of ETL and data integration. "This requires different tools with a different mindset," Freivald said.

For example, Freivald talked to a bank recently that wanted to do fraud analysis of ATM transactions. The bank kept asking how to scrape the VSAM files used for storing the data. In this case, it made more sense to put a listener on the wire for the messages as they came in, rather than trying to access the files. "They were not thinking about messages since they were used to thinking about managing information using files," Freivald explained.

Making it work on the front line

More on business intelligence

BI success needs data quality strategy

How BI programs add business value

Mastering the presentation of BI data

An enterprise architect has to begin with the end user in mind instead of the analyst. The system needs to be simple and straightforward enough that a nontechnical worker can benefit from straightforward decisions delivered on a regular basis.

Freivald said this also involves integrating the information into the existing workflow. If a user has to leave the main application screen, it is unlikely they will look for an answer or that they will trust it. This means integrating the operational BI capabilities into their day-to-day Web application, CRM or the mobile device.

Expanded governance

It is important to think about the potential adverse impact of more information in the hands of a wider group of employees. It is one thing to give a trusted group of analysts' access to richer platforms for analyzing data, but that could create problems when the data is dispersed too widely.

"If I choose to present more data, I have to be careful about what is shown compared to if I only show it to 10 executives," Taylor said. People using that environment don't need to see the data, just the results of analyzing it. You don't need to include personally identifiable information in those models.

Be ready for change

An enterprise architect has to look at the business process first and find things in the process that could be improved by adding intelligence. These need to be processes worth the investment of setting up the operational BI. You need to have analysts look at the problem because they understand the big picture, and you need business people looking at the answers the analysts are seeing.

Setting up an operational BI system tends to be iterative. Don't expect your first version to be awesome. You might find the operational BI system does not do what you want but could be improved with some additional data or predictive analytics running in the background. "This is an iterative process, and the business benefits won't come from a single waterfall-style development cycle," Freivald said.

You also need flexible people and middleware. There won't be a single-use ETL product to get from one database to another. It will call upon other applications and other BI models. "The main best practice is to focus on improving the business process and iterate on that with the right combination of business people," Freivald said.

About the author:
George Lawton is a journalist based near San Francisco. Over the last 15 years, he's written more than 2,000 articles on computers, communications, business and other topics. Find out more at

Follow us on Twitter at @SearchSOA and like us on Facebook.

Dig Deeper on Enterprise application integration

Join the conversation


Send me notifications when other members comment.

Please create a username to comment.

Have you tried to solve operational BI problems with SOA middleware?
in-line data transformation capability of ESBs is perfect for transforming log data and database extracts into structured data that is easier to store and query. We are avoiding the EDW though through a enterprise data hub architecture based on Hadoop.
Some messages are bigger than the JVM size, and it is easily to be manipulated by the appliance like IBM datapower xI50