Evaluate Weigh the pros and cons of technologies, products and projects you are considering.

Q&A: Dave Linthicum talks about commercial Web data services and distributed computing

The cloud keeps growing and we keep finding new uses for it. Cloud computing and SOA expert David Linthicum describes commercial Web data services that are taking advantage of the processing power of the cloud.


David Linthicum

Dave Linthicum, the CTO of Bick Group, is a well known cloud computing and SOA expert. He played a role in developing concepts involved with distributed computing including EAI, B2B Application Integration, and SOA. David Linthicum has authored 10 books, including "Enterprise Application Integration" and "B2B Application Integration," and is an active blogger.

SearchSOA.com: Is there a specific area of distributed computing that you are focusing on these days?

David Linthicum: In the cloud, we are moving toward what I call Big Data. The idea is that because we can get access to an almost unlimited number of servers at the same time -- thousands of instances, which is far more than you could have on premises -- we can use things like MapReduce, the software framework introduced by Google, Yahoo, and Facebook, to support distributed computing.

So, in short, we in IT have been messing with this idea of distributed computing and data for the last 20 years but now we have access to resources that we didn't have in the past.
Dave Linthicum, CTO, Bick Group

 MapReduce takes a query from a mass of data and will distribute it to any number of servers and run the query in little pieces. It is like giving 20 people a book to read and then reassembling and reducing to get the results back. That addresses a couple of problems. A lot of my clients, for instance, are waiting three days to get a traditional query back. Sequential queries and relational databases are the problems. By contrast, the cloud is a highly scalable and elastic way of doing that so queries that took days can now be done in minutes.

Apache Hadoop is another thing that helps. It arose out of work by Yahoo, Facebook, and Google. They had a massive information processing problem and were using MapReduce. The inventor of Hadoop made a software framework distributed application that can work with thousands of nodes.

So, in short, we in IT have been messing with this idea of distributed computing and data for the last 20 years but now we have access to resources that we didn't have in the past -- and it has been driven by the real business needs of organizations like Yahoo, Google, and Facebook.

SearchSOA.com: How about the somewhat more conventional world of commercial Web data services, what can application developers and others do to more effectively harness these external resources for their internal needs?

David Linthicum: Well, there certainly are services out there that provide this -- say Xignite, which provides financial data on demand. Most cloud providers such as Amazon and the US government provide feeds and the value of [those feeds] is that you and your organization can leverage the information cost effectively. These services are typically RESTful Web services that are coupled into applications to serve a particular purpose. In this case, the purpose is providing information. A stock ticker example is very typical, but we can also leverage news services from another API and customer validation from another, all leveraged by the same composite application.

I would say a typical app of this type is risk analytics, where you would actually model a stock trade. Suppose you are trading millions in assets to try to get a better financial position. The information you need to do the risk analytics come from the web. You populate it through these APIs, which work through backend services, and it ends up providing you with, say, a 1-5 rating. Instead of having to maintain information out of Wall Street for a stock trading operation, you can now leverage third-party provider information and the API they are going to provide. This capability has been around for a long time, but people didn't really get into it until the cloud took off.

Another related trend is people moving from just having information as a service to cloud providing additional capabilities around that information. For example, Amazon provides some aspects of information-as-a-service, by providing some information using a Web services interface. So, why subscribe to another separate service when Amazon provides some and Google provides others? They are the aggregators now. In fact, they may end up buying up some of the older service providers.

SearchSOA.com: A few months ago you blogged enthusiastically about a new "consortium" that you were participating in. What is it and has it lived up to your expectations?

David Linthicum: That was something called the SOA Data Integration Architecture Community. It exists on LinkedIn and it provides a place for people to discuss the ins and outs of SOA. It is a place where people suggest, for example, different standards and ways to work with current data integration standards. It is a very loosely coupled group -- open and free -- but I think it is proving to be very valuable to people who want to create agile architectures such as SOA. Members come from many disciplines, including enterprise and IT architects, as well as IT managers.

Dig Deeper on Topics Archive

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.