Welcome!

Open Source Cloud Authors: Elizabeth White, Pat Romanski, Valeriia Timokhina, Shelly Palmer, Stackify Blog

Related Topics: Microservices Expo, Open Source Cloud

Microservices Expo: Article

Hidden Cost of Open Source SOA Testing

There are many aspects of service testing that contribute to a comprehensive solution across the SOA life cycle

In this article, I will discuss some considerations and potential limitations of adopting an open source SOA testing solution for a long term SOA strategy . Open Source has become an essential and popular resource for many tools and platforms used in SOA deployments. From operating systems such as Linux, to databases such as MySQL, and browsers such as Firefox, open source has a proven track record for cost-effective applications and tools.   SOA testing involves the ability to test SOAP, XML, and REST based messaging patterns against a service endpoint in order to assess the robustness, reliability, and resilience of the service. Comprehensive testing of a service involves 4 primary areas: Functional, Performance, Interoperability, and Security.

Functional testing provides the ability to verify the proper behavior of services and build regression test suites to automate testing and baseline expected behavior of services to quickly assess and validate functionality through the lifecycle of service revisions.

Performance testing via a concurrent, simultaneous loading agent framework can determine throughput and capacity statistics of the service across the range of input and client load variances to validate Service Level Agreement rates and well as identify bottlenecks and potential architectural weaknesses and performance dependencies.

Interoperability testing maximizes interoperability by measuring both the design characteristics of a service as well as the run-time adherence to standards and best-practices. Isolating potential interoperability issues early on in the lifecyle can significantly optimize efforts of integration when exposing the service to trading partners and clients which may be build on a varying array of  technologies and platforms.

Security testing assesses the risk posture and robustness of a service with regard to vulnerability, data-leakage, data privacy, and data integrity. Each service is unique based on the schema and message patterns which defines the input and response message structure of how to communicate with the service.  In the case of SOAP services for example, using the WSDL schema as the source, security tests can be built to create boundary condition tests for the service which then identify the robustness of the service handling inputs outside the range of expectation.

These 4 areas of testing together provide the comprehensive analysis and understanding of the resilience, reliability, and robustness of the service.

Open Source SOA Testing
The open source tools available today for SOA testing focus primarily on the functional testing aspects of a service. Since functional testing is often first in the SOA development lifecycle, and adopted early-on in the development and implementation phase, free open source tools become widely adopted by development teams both because they are free, and also because the use cases are often limited to simple unit testing of service messaging.

However, as services mature and move down the SOA lifecycle phases to system testing, integration, pre-production analysis and validation, the other perspectives of SOA testing need to play a role in the comprehensive assessment of the quality, robustness, and capabilities of a published service.  It is in these areas where the open source SOA testing tools can fall short.

SOA Functional Testing Limitations
Generally the functional testing capabilities of an open source testing tool are adequate for the simple type of SOA deployments which do not have complex WSDLs, Schemas, or message patterns. Once the deployments gain more complexity however, the challenges of functional testing move from single request-response testing to scenario testing where the functional behavior is measured not by one request-response, but rather several transactions each dependent on the other as a business functional unit. Having the ability to test these types of functional scenarios effectively requires the ability to maintain state between one test result and the next.  

SOA Performance Testing Limitations
While there are several open source performance testing products on the market, these are primarily tools used in the static web testing paradigms. When dealing with SOAP and XML -based transactions, the static data testing behaviors of these performance harnesses does not allow for the unique-wire signature requirements of the message patterns as they occur in actual service transactions. When running performance tests with a web-based testing platform, the results are that the end-points become inundated with static messages which are not characteristic of actual traffic patterns. In fact, in many cases, the service endpoint itself is supposed to reject these static messages as replay-attacks on the service.

Another consideration of performance testing is the level of security and identity provisions that messages may be required to carry in order to access the service. Static open source performance testing harnesses do not provide solutions for message security and message identity requirements.

SOA Interoperability Testing Limitations
The promise of SOA provides a open, reusable architecture which lowers cost through the reuse ROI factor. The challenge of SOA however is the ability to widely inter-operate with other technologies that also communicate via SOAP, XML, and REST. Interoperability testing involves both design-time analysis of service characteristics, such as WSDL and schema, as well as run-time assessment of a service robustness in terms of consuming and handling message patterns that may fall outside the expected structures.  Many open source toolkits leverage the available WS-I analysis framework to provide a means to assess the design-time characteristics of a WSDL and schema according to published profiles, and also provides some run-time analysis reporting of message patterns. However, the Open Source toolkits do not provide the ability to generate messages that fall outside these expected patterns, which is the key to measuring the actual posture of the run-time service. In fact, it is testing of messages that are not expected where the true measure of a service interoperability posture can be determined.

SOA Security Testing Limitations.
Security testing falls across many areas. From a threat perspective, security testing involves integrity and structure of messages with injection attacks at the parameter and data structure levels in order to assess the behavior and resilience of the service endpoint when faced with data values and message structures outside of the expected format.  From a trust perspective, security testing involves PKI with encryption, signatures, and identity tokens. This requires testing frameworks that understand the various emerging standards from W3C and OASIS in order to support the wide range of security message formats and also requires the means to retrieve and utility X509 and Private Keys from a variety of sources including Windows Keystore, Java Keystore, SmartCards, PKCS12 files, etc.  Many open source testing tools are designed for general testing and message creation, but lack the in-depth security and identity features to be considered viable for this type of testing.

Conclusion
Adopting a free open source tool for SOA testing seems the simplest, most cost effective choice for developers and testers early on. However, you should plan and consider the implications of a longer term strategy with an open source testing tool. The many other aspects of service testing which contribute to a comprehensive testing solution across the entire SOA lifecycle that go beyond simple testing paradigms.

Before adopting a tool, consider whether down the road you may need something more.  You may find that paying for a solution ends up costing much less than not paying for one.

More Stories By Jason Macy

Jason Macy is the CTO at Crosscheck Networks responsible for implementation and product strategy of the SOA Web Services based technologies. As co-founder of Crosscheck Networks, Jason has pioneered the field of web service testing and simulation with over 40,000 product installation worldwide. Jason previously served as VP Engineering for the wholly-owned subsidiary Forum Systems where he is responsible for the software development lifecycle of the industry's only FIPS certified hardware security gateway for SOA web services. Before moving into the XML web services realm, Jason worked as the lead architect for Raytheon responsible for testing and successful commissioning of the Air Traffic Control system at Schipol Airport in Amsterdam, Holland. Jason has extensive experience in XML, Web Services, Networking, and Security and and provides education and training of SOA web services implementation, testing, and security including speaking engagements at industry conferences such as Gartner and StarEAST. Jason holds dual-degrees in both Computer Science and Computer Engineering.

@ThingsExpo Stories
Elon Musk is among the notable industry figures who worries about the power of AI to destroy rather than help society. Mark Zuckerberg, on the other hand, embraces all that is going on. AI is most powerful when deployed across the vast networks being built for Internets of Things in the manufacturing, transportation and logistics, retail, healthcare, government and other sectors. Is AI transforming IoT for the good or the bad? Do we need to worry about its potential destructive power? Or will we...
SYS-CON Events announced today that Daiya Industry will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Daiya Industry specializes in orthotic support systems and assistive devices with pneumatic artificial muscles in order to contribute to an extended healthy life expectancy. For more information, please visit https://www.daiyak...
SYS-CON Events announced today that SIGMA Corporation will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. uLaser flow inspection device from the Japanese top share to Global Standard! Then, make the best use of data to flip to next page. For more information, visit http://www.sigma-k.co.jp/en/.
SYS-CON Events announced today that B2Cloud will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. B2Cloud specializes in IoT devices for preventive and predictive maintenance in any kind of equipment retrieving data like Energy consumption, working time, temperature, humidity, pressure, etc.
SYS-CON Events announced today that NetApp has been named “Bronze Sponsor” of SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. NetApp is the data authority for hybrid cloud. NetApp provides a full range of hybrid cloud data services that simplify management of applications and data across cloud and on-premises environments to accelerate digital transformation. Together with their partners, NetApp em...
SYS-CON Events announced today that Interface Corporation will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Interface Corporation is a company developing, manufacturing and marketing high quality and wide variety of industrial computers and interface modules such as PCIs and PCI express. For more information, visit http://www.i...
SYS-CON Events announced today that Massive Networks, that helps your business operate seamlessly with fast, reliable, and secure internet and network solutions, has been named "Exhibitor" of SYS-CON's 21st International Cloud Expo ®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. As a premier telecommunications provider, Massive Networks is headquartered out of Louisville, Colorado. With years of experience under their belt, their team of...
SYS-CON Events announced today that Mobile Create USA will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Mobile Create USA Inc. is an MVNO-based business model that uses portable communication devices and cellular-based infrastructure in the development, sales, operation and mobile communications systems incorporating GPS capabi...
SYS-CON Events announced today that Nihon Micron will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Nihon Micron Co., Ltd. strives for technological innovation to establish high-density, high-precision processing technology for providing printed circuit board and metal mount RFID tags used for communication devices. For more inf...
SYS-CON Events announced today that N3N will exhibit at SYS-CON's @ThingsExpo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. N3N’s solutions increase the effectiveness of operations and control centers, increase the value of IoT investments, and facilitate real-time operational decision making. N3N enables operations teams with a four dimensional digital “big board” that consolidates real-time live video feeds alongside IoT sensor data a...
While some developers care passionately about how data centers and clouds are architected, for most, it is only the end result that matters. To the majority of companies, technology exists to solve a business problem, and only delivers value when it is solving that problem. 2017 brings the mainstream adoption of containers for production workloads. In his session at 21st Cloud Expo, Ben McCormack, VP of Operations at Evernote, will discuss how data centers of the future will be managed, how th...
SYS-CON Events announced today that Suzuki Inc. will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Suzuki Inc. is a semiconductor-related business, including sales of consuming parts, parts repair, and maintenance for semiconductor manufacturing machines, etc. It is also a health care business providing experimental research for...
SYS-CON Events announced today that Ryobi Systems will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Ryobi Systems Co., Ltd., as an information service company, specialized in business support for local governments and medical industry. We are challenging to achive the precision farming with AI. For more information, visit http:...
SYS-CON Events announced today that MIRAI Inc. will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. MIRAI Inc. are IT consultants from the public sector whose mission is to solve social issues by technology and innovation and to create a meaningful future for people.
SYS-CON Events announced today that mruby Forum will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. mruby is the lightweight implementation of the Ruby language. We introduce mruby and the mruby IoT framework that enhances development productivity. For more information, visit http://forum.mruby.org/.
SYS-CON Events announced today that Fusic will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Fusic Co. provides mocks as virtual IoT devices. You can customize mocks, and get any amount of data at any time in your test. For more information, visit https://fusic.co.jp/english/.
In his session at @ThingsExpo, Greg Gorman is the Director, IoT Developer Ecosystem, Watson IoT, will provide a short tutorial on Node-RED, a Node.js-based programming tool for wiring together hardware devices, APIs and online services in new and interesting ways. It provides a browser-based editor that makes it easy to wire together flows using a wide range of nodes in the palette that can be deployed to its runtime in a single-click. There is a large library of contributed nodes that help so...
SYS-CON Events announced today that Enroute Lab will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Enroute Lab is an industrial design, research and development company of unmanned robotic vehicle system. For more information, please visit http://elab.co.jp/.
Agile has finally jumped the technology shark, expanding outside the software world. Enterprises are now increasingly adopting Agile practices across their organizations in order to successfully navigate the disruptive waters that threaten to drown them. In our quest for establishing change as a core competency in our organizations, this business-centric notion of Agile is an essential component of Agile Digital Transformation. In the years since the publication of the Agile Manifesto, the conn...
Real IoT production deployments running at scale are collecting sensor data from hundreds / thousands / millions of devices. The goal is to take business-critical actions on the real-time data and find insights from stored datasets. In his session at @ThingsExpo, John Walicki, Watson IoT Developer Advocate at IBM Cloud, will provide a fast-paced developer journey that follows the IoT sensor data from generation, to edge gateway, to edge analytics, to encryption, to the IBM Bluemix cloud, to Wa...