In the context of a service-oriented architecture, the notion of testing Web services, and which tools to use, is evolving.
"[Most customers] haven't gotten to the point of maturity to know what they need," said Jason Bloomberg, a senior analyst with Waltham, Mass.-based ZapThink LLC. "And there is still a real lack of solutions for doing any real testing of Web services in the context of SOA."
Different vendors offer different pieces of the puzzle, Bloomberg said, such as diagnostics, load and performance testing, asynchronous capabilities, and orchestration and versioning. "But nobody is looking at the whole picture from the perspective of a SOA; larger vendors are looking at some aspects, but there are still a lot of pieces to fall in place," he said.
User organizations aren't looking at the whole picture yet, either, said Carey Schwaber, an analyst at Cambridge, Mass.-based Forrester Research Inc. "The end-to-end, actual testing of Web services -- user companies are less likely to do -- and they are almost always focused on the user interface. They rarely dig under the UI, where most robust testing comes in."
Both young Web services testing companies, as well as mainstream testing vendors, are targeting this market, Schwaber said. Certain vendors "focus on testing a wide range of Web services," she said, "but most [vendors] are just building it in as a checklist in terms of what they support." At this stage of the market, "we don't know how often users need support for Web services testing; it may be they've just heard the buzzword."
For Web services in their simplest form -- software with a Web services wrapper -- today's testing tools "are adequate," Bloomberg said. "But for SOA implementations, testing is essentially being done in a custom way."
Developing test programs in-house to generate SOAP requests, however, "becomes very painful very quickly to just maintain those interfaces as the schema and business processes change," said Narendra Patil, president and CEO of Optimyz Software, a tool vendor in Santa Clara, Calif.
Optimyz's WebServiceTester is an end-to-end product offering automatic test generation; functional, regression, and load testing; conformance testing against WS-I Profiles, BPEL-based orchestration testing; secure Web services testing; and debugging and diagnostics.
Patil stresses the importance of orchestration, or testing the business process as a whole. "When you have Web services, and then each Web service is representing some part of the business process, you quickly realize users of my application are not going to use each Web service individually, but as a business process, and they have no tools to simulate and test the business process as whole. People are concentrating on functional testing, then deploying the service. But with a lot of consumer applications, the traffic is very high and you start seeing performance issues."
Long-time tool vendor Mercury also targets the "end to end" of Web services testing in the form of three offerings: LoadRunner, QuickTest Professional and Business Process Testing, its newest tool that sits on top of LoadRunner.
Following is a brief look at some of the other vendors targeting Web services testing:
- Empirix Inc. -- e-TEST: e-Manager Enterprise, test management; e-Tester, functional testing; e-Load, scalability testing.
- Mindreef Inc. -- SOAPscope, Web services diagnostics; Share-It, collaborative online Web services diagnostics software that allows Web services professionals to publish and share evidence related to a Web services problem online via the company's Web site.
- Parasoft -- SOAPtest, WSDL validation, unit and functional testing of the client and server, performance testing.
- PushToTest -- TestMaker, open-source framework and utility to build intelligent test agents that drive services.
- Solstice Software Inc. -- Integra Enterprise, testing, simulation and validation software for message-based systems.
- Spirent plc -- Avalanche, load testing.
- IBM Rational Software Co. -- TestStudio, unit, functionality, performance, and load testing; PurifyPlus, runtime analysis tool for detects memory and performance bottlenecks early in the development cycle.
Curtis Williams, developer at SOA consulting firm TeamSOA, said there is value to using a niche Web services testing tool like Mindreef's SOAPscope, along with other development tools in an organization's arsenal. He said "[SOAPscope is] like a safety blanket. It's a way to give third-party auditor's eye to a service or application; it will verify XML for reliability, it will analyze XML outside of what the vendor says is good XML."