News Stay informed about the latest enterprise technology news and product updates.

Testing, testing, testing

CBDi Forum addresses testing and certification, the next big issue for Web service deployment. This article provides a simple framework for thinking about what type of testing is required at each stage of the life cycle and comment on Mindreef, a start-up company that has just launched this week and is starting to address many of the issues that CBDi has identified.

Market Analysis

Testing, testing, testing

In July we identified testing and certification as being the next big roadblock just round the corner that needs to be addressed. We said: "The big issue is will the service work correctly every time when I need it? As yet few are thinking about the issues of testing and certification. We suggest that testing and certification of Web services is not business as usual and that new solutions are needed to provide assurance that services can really be trusted."

Radical change in testing requirements
In our commentary we identified how Web services place new demands on testing activity. We emphasized the fundamental differences between black box and white box reuse, and that whilst we have all paid lip service to black box with components, we are forced to operate with full transparency between provider and consumer in the Services world. Another important conclusion we drew is that Testing seems too narrow an expression because traditionally it is limited to activities that take place in the development process, and not during the execution phase, and certainly not prior to every business event. We need some form of active certification that the task has completed correctly with no unwanted effects. This goes beyond simply checking return codes and provides confirmation that the business transaction has completed correctly, which may of course involve multiple parties to validate this.

Framework for testing
The need for dynamic testing suggests we need to look at each part of the life cycle of a service and identify appropriate testing and certification at each stage. This requires a new framework for testing which might include:

  • Static Testing - execution of a service prior to deployment
  • Static Certification - warranty by some certifying body that a service is fit for purpose prior to deployment
  • Dynamic testing - execution of a dummy or test service, that is used as part of pre-conditional activity to ensure the service provides the correct functionality.
  • Dynamic certification - execution of post conditional services that verify the service has executed to specification.

The dynamic aspects of testing XML Web services require the introduction of "testing" service(s) in addition to the "functional" service. These are pre-condition and post-condition checks before and after each functional interface, and could include technical and business oriented testing services. A technical testing Service may check that the WSDL complies with prior agreements. A business testing Service would ensure transactional integrity. For example if a bank provides a Web service function call "fundTransfer(fromAcct, toAcct)", then there may be two additional Services required:

"checkAccts(fromAcct)" which checks the fromAcct has enough funds, and
"checkBalance(fromAcct, toAcct)" which verifies the sum of the two accounts is unchanged.

These additional services are specifically checking pre and post conditional states, and designed to validate the correct operation of the core business service. A benefit of separate classification of these services would be to make it easy to manage (the entire life cycle of) these services quite separately from the core business transactions.

Diagnostics from Mindreef
This week, we report on a new start-up company, Mindreef, that has just announced its presence and intent to focus on Web services diagnostics. Mindreef has been founded by escapees from NuMega, (now part of Compuware) who we have reported on over the years as a company that understood how to test and certify components.

The Mindreef Web Services Diagnostics systems collect information about SOAP transactions and use the information to provide visibility and insight into Web services communications through features such as logging, debugging, reporting, profiling, analytics and collaboration. The diagnostic services operate between the endpoints and are completely non-invasive. In their announcement Mindreef says that "the additional complexity inherent in distributed applications is leading to problems that standard debugging and performance tools are not designed to handle. A new breed of comprehensive diagnostic tools is required that focuses on the connections between loosely coupled components rather than the code in the components." We couldn't have said it better ourselves. Read more on this also with a short Q & A with chief architect and founder in our report, referenced below.

Web services don't work?
On a not entirely unrelated matter, we have noticed in the last couple of weeks that some industry analysts are taking a rather negative view of Web services. No sooner than one analyst publishes comments such as, "it's all marketing hype" then another comments that "we will wait for 10 years for Web services".

One wonders where these people have been for the last three years, or indeed which lagging vendor is feeding them this line. Readers of our weekly commentary and our monthly CBDI Journal know that Services concepts are a natural evolution of many solid technologies and practices and are being adopted by the entire industry and its customers. One can only assume that these analysts are so desperate that they have sacrificed technical and analytical integrity in their quest for visibility. The following extracts from the Reader Feedback Page on one of their sites suggests that the negative coverage might not be working:

"I would love to see the analysis for this kind of report. As a company we are using XMLWS today to integrate CRM and Manufacturing processes together and this one of a number I know that have similar solutions. "
"With this piece of "research" .... have demonstrated just how much they really understand about this important topic. Being controversial requires real knowledge and insight which they patently lack. "
"The transition to Service based systems is happening slowly mostly in private, and yes often in legacy integration. But the key point is the introduction of real separation of concerns that delivers the adaptability that organizations have been striving for over many years using technologies such as DCE, Corba etc. "
"This type of comment is nothing new. However, it is misdirected. "
"The basic SOAP and WSDL technologies are a major step forward and we should put them to good use and keep up the good momentum on the rest of the standards needed for a wider range of apps."

Note: These extracts from Reader Feedback Pages were clearly marked as "content over which the publisher takes no responsibility".

All this makes us wonder where the stimulus for such negative comments comes from. Perhaps some lagging vendor, disappointed they have been left behind in the Web services market is doing negative briefing to the more gullible industry analysts, in an attempt to spread fear, uncertainty and doubt. However it's obvious to the more serious minded amongst us that the Service based revolution is unstoppable because it makes sense. Of course it will take time to mature, but this is why we need serious debate on key business and technical issues, not unsubstantiated gossip. As they say in the trade: Name and Address supplied.

Please let us have your feedback and comment to

CBDI Newswire Thu 04 Jul: Testing Web services
(You will be asked to login or register as a Bronze member to see this report as it is a prior month item)

CBDI Newswire Wed 30 Oct: Mindreef Focuses on WS Diagnostics

Thanks to CISCO for contributions to thinking on WS testing strategies

Copyright CBDi Forum Limited 2002. The CBDi Forum is an analysis firm and think tank, providing insight on component and web service technologies, processes and practices for the software industry and its customers. To register for the weekly newswire click here.

For more information:

  • Looking for free research? Browse our comprehensive White Papers section by topic, author or keyword.
  • Are you tired of technospeak? The Web Services Advisor column uses plain talk and avoids the hype.
  • For insightful opinion and commentary from today's industry leaders, read our Guest Commentary columns.
  • Hey Codeheads! Start benefiting from these time-saving XML Developer Tips and .NET Developer Tips.

  • Visit our huge Best Web Links for Web Services collection for the freshest editor-selected resources.
  • Visit Ask the Experts for answers to your Web services, SOAP, WSDL, XML, .NET, Java and EAI questions.
  • Couldn't attend one of our Webcasts? Don't miss out. Visit our archive to watch at your own convenience.
  • Choking on the alphabet soup of industry acronyms? Visit our helpful Glossary for the latest lingo.
  • Discuss this article, voice your opinion or talk with your peers in the SearchWebServices Discussion Forums.

Dig Deeper on Topics Archive

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.