Welcome!

Open Source Cloud Authors: Elizabeth White, Derek Weeks, Stackify Blog, Liz McMillan, Dynatrace Blog

Related Topics: Microservices Expo, Open Source Cloud

Microservices Expo: Article

Hidden Cost of Open Source SOA Testing

There are many aspects of service testing that contribute to a comprehensive solution across the SOA life cycle

In this article, I will discuss some considerations and potential limitations of adopting an open source SOA testing solution for a long term SOA strategy . Open Source has become an essential and popular resource for many tools and platforms used in SOA deployments. From operating systems such as Linux, to databases such as MySQL, and browsers such as Firefox, open source has a proven track record for cost-effective applications and tools.   SOA testing involves the ability to test SOAP, XML, and REST based messaging patterns against a service endpoint in order to assess the robustness, reliability, and resilience of the service. Comprehensive testing of a service involves 4 primary areas: Functional, Performance, Interoperability, and Security.

Functional testing provides the ability to verify the proper behavior of services and build regression test suites to automate testing and baseline expected behavior of services to quickly assess and validate functionality through the lifecycle of service revisions.

Performance testing via a concurrent, simultaneous loading agent framework can determine throughput and capacity statistics of the service across the range of input and client load variances to validate Service Level Agreement rates and well as identify bottlenecks and potential architectural weaknesses and performance dependencies.

Interoperability testing maximizes interoperability by measuring both the design characteristics of a service as well as the run-time adherence to standards and best-practices. Isolating potential interoperability issues early on in the lifecyle can significantly optimize efforts of integration when exposing the service to trading partners and clients which may be build on a varying array of  technologies and platforms.

Security testing assesses the risk posture and robustness of a service with regard to vulnerability, data-leakage, data privacy, and data integrity. Each service is unique based on the schema and message patterns which defines the input and response message structure of how to communicate with the service.  In the case of SOAP services for example, using the WSDL schema as the source, security tests can be built to create boundary condition tests for the service which then identify the robustness of the service handling inputs outside the range of expectation.

These 4 areas of testing together provide the comprehensive analysis and understanding of the resilience, reliability, and robustness of the service.

Open Source SOA Testing
The open source tools available today for SOA testing focus primarily on the functional testing aspects of a service. Since functional testing is often first in the SOA development lifecycle, and adopted early-on in the development and implementation phase, free open source tools become widely adopted by development teams both because they are free, and also because the use cases are often limited to simple unit testing of service messaging.

However, as services mature and move down the SOA lifecycle phases to system testing, integration, pre-production analysis and validation, the other perspectives of SOA testing need to play a role in the comprehensive assessment of the quality, robustness, and capabilities of a published service.  It is in these areas where the open source SOA testing tools can fall short.

SOA Functional Testing Limitations
Generally the functional testing capabilities of an open source testing tool are adequate for the simple type of SOA deployments which do not have complex WSDLs, Schemas, or message patterns. Once the deployments gain more complexity however, the challenges of functional testing move from single request-response testing to scenario testing where the functional behavior is measured not by one request-response, but rather several transactions each dependent on the other as a business functional unit. Having the ability to test these types of functional scenarios effectively requires the ability to maintain state between one test result and the next.  

SOA Performance Testing Limitations
While there are several open source performance testing products on the market, these are primarily tools used in the static web testing paradigms. When dealing with SOAP and XML -based transactions, the static data testing behaviors of these performance harnesses does not allow for the unique-wire signature requirements of the message patterns as they occur in actual service transactions. When running performance tests with a web-based testing platform, the results are that the end-points become inundated with static messages which are not characteristic of actual traffic patterns. In fact, in many cases, the service endpoint itself is supposed to reject these static messages as replay-attacks on the service.

Another consideration of performance testing is the level of security and identity provisions that messages may be required to carry in order to access the service. Static open source performance testing harnesses do not provide solutions for message security and message identity requirements.

SOA Interoperability Testing Limitations
The promise of SOA provides a open, reusable architecture which lowers cost through the reuse ROI factor. The challenge of SOA however is the ability to widely inter-operate with other technologies that also communicate via SOAP, XML, and REST. Interoperability testing involves both design-time analysis of service characteristics, such as WSDL and schema, as well as run-time assessment of a service robustness in terms of consuming and handling message patterns that may fall outside the expected structures.  Many open source toolkits leverage the available WS-I analysis framework to provide a means to assess the design-time characteristics of a WSDL and schema according to published profiles, and also provides some run-time analysis reporting of message patterns. However, the Open Source toolkits do not provide the ability to generate messages that fall outside these expected patterns, which is the key to measuring the actual posture of the run-time service. In fact, it is testing of messages that are not expected where the true measure of a service interoperability posture can be determined.

SOA Security Testing Limitations.
Security testing falls across many areas. From a threat perspective, security testing involves integrity and structure of messages with injection attacks at the parameter and data structure levels in order to assess the behavior and resilience of the service endpoint when faced with data values and message structures outside of the expected format.  From a trust perspective, security testing involves PKI with encryption, signatures, and identity tokens. This requires testing frameworks that understand the various emerging standards from W3C and OASIS in order to support the wide range of security message formats and also requires the means to retrieve and utility X509 and Private Keys from a variety of sources including Windows Keystore, Java Keystore, SmartCards, PKCS12 files, etc.  Many open source testing tools are designed for general testing and message creation, but lack the in-depth security and identity features to be considered viable for this type of testing.

Conclusion
Adopting a free open source tool for SOA testing seems the simplest, most cost effective choice for developers and testers early on. However, you should plan and consider the implications of a longer term strategy with an open source testing tool. The many other aspects of service testing which contribute to a comprehensive testing solution across the entire SOA lifecycle that go beyond simple testing paradigms.

Before adopting a tool, consider whether down the road you may need something more.  You may find that paying for a solution ends up costing much less than not paying for one.

More Stories By Jason Macy

Jason Macy is the CTO at Crosscheck Networks responsible for implementation and product strategy of the SOA Web Services based technologies. As co-founder of Crosscheck Networks, Jason has pioneered the field of web service testing and simulation with over 40,000 product installation worldwide. Jason previously served as VP Engineering for the wholly-owned subsidiary Forum Systems where he is responsible for the software development lifecycle of the industry's only FIPS certified hardware security gateway for SOA web services. Before moving into the XML web services realm, Jason worked as the lead architect for Raytheon responsible for testing and successful commissioning of the Air Traffic Control system at Schipol Airport in Amsterdam, Holland. Jason has extensive experience in XML, Web Services, Networking, and Security and and provides education and training of SOA web services implementation, testing, and security including speaking engagements at industry conferences such as Gartner and StarEAST. Jason holds dual-degrees in both Computer Science and Computer Engineering.

@ThingsExpo Stories
SYS-CON Events announced today that Cloudistics, an on-premises cloud computing company, has been named “Bronze Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Cloudistics delivers a complete public cloud experience with composable on-premises infrastructures to medium and large enterprises. Its software-defined technology natively converges network, storage, compute, virtualization, and management into a ...
SYS-CON Events announced today that SD Times | BZ Media has been named “Media Sponsor” of SYS-CON's 20th International Cloud Expo, which will take place on June 6–8, 2017, at the Javits Center in New York City, NY. BZ Media LLC is a high-tech media company that produces technical conferences and expositions, and publishes a magazine, newsletters and websites in the software development, SharePoint, mobile development and commercial UAV markets.
"I think that everyone recognizes that for IoT to really realize its full potential and value that it is about creating ecosystems and marketplaces and that no single vendor is able to support what is required," explained Esmeralda Swartz, VP, Marketing Enterprise and Cloud at Ericsson, in this SYS-CON.tv interview at @ThingsExpo, held June 7-9, 2016, at the Javits Center in New York City, NY.
Why do your mobile transformations need to happen today? Mobile is the strategy that enterprise transformation centers on to drive customer engagement. In his general session at @ThingsExpo, Roger Woods, Director, Mobile Product & Strategy – Adobe Marketing Cloud, covered key IoT and mobile trends that are forcing mobile transformation, key components of a solid mobile strategy and explored how brands are effectively driving mobile change throughout the enterprise.
My team embarked on building a data lake for our sales and marketing data to better understand customer journeys. This required building a hybrid data pipeline to connect our cloud CRM with the new Hadoop Data Lake. One challenge is that IT was not in a position to provide support until we proved value and marketing did not have the experience, so we embarked on the journey ourselves within the product marketing team for our line of business within Progress. In his session at @BigDataExpo, Sum...
Keeping pace with advancements in software delivery processes and tooling is taxing even for the most proficient organizations. Point tools, platforms, open source and the increasing adoption of private and public cloud services requires strong engineering rigor - all in the face of developer demands to use the tools of choice. As Agile has settled in as a mainstream practice, now DevOps has emerged as the next wave to improve software delivery speed and output. To make DevOps work, organization...
SYS-CON Events announced today that MobiDev, a client-oriented software development company, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place June 6-8, 2017, at the Javits Center in New York City, NY, and the 21st International Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. MobiDev is a software company that develops and delivers turn-key mobile apps, websites, web services, and complex softw...
DevOps is often described as a combination of technology and culture. Without both, DevOps isn't complete. However, applying the culture to outdated technology is a recipe for disaster; as response times grow and connections between teams are delayed by technology, the culture will die. A Nutanix Enterprise Cloud has many benefits that provide the needed base for a true DevOps paradigm.
What sort of WebRTC based applications can we expect to see over the next year and beyond? One way to predict development trends is to see what sorts of applications startups are building. In his session at @ThingsExpo, Arin Sime, founder of WebRTC.ventures, will discuss the current and likely future trends in WebRTC application development based on real requests for custom applications from real customers, as well as other public sources of information,
China Unicom exhibit at the 19th International Cloud Expo, which took place at the Santa Clara Convention Center in Santa Clara, CA, in November 2016. China United Network Communications Group Co. Ltd ("China Unicom") was officially established in 2009 on the basis of the merger of former China Netcom and former China Unicom. China Unicom mainly operates a full range of telecommunications services including mobile broadband (GSM, WCDMA, LTE FDD, TD-LTE), fixed-line broadband, ICT, data communica...
SYS-CON Events announced today that Ocean9will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Ocean9 provides cloud services for Backup, Disaster Recovery (DRaaS) and instant Innovation, and redefines enterprise infrastructure with its cloud native subscription offerings for mission critical SAP workloads.
Things are changing so quickly in IoT that it would take a wizard to predict which ecosystem will gain the most traction. In order for IoT to reach its potential, smart devices must be able to work together. Today, there are a slew of interoperability standards being promoted by big names to make this happen: HomeKit, Brillo and Alljoyn. In his session at @ThingsExpo, Adam Justice, vice president and general manager of Grid Connect, will review what happens when smart devices don’t work togethe...
SYS-CON Events announced today that SoftLayer, an IBM Company, has been named “Gold Sponsor” of SYS-CON's 18th Cloud Expo, which will take place on June 7-9, 2016, at the Javits Center in New York, New York. SoftLayer, an IBM Company, provides cloud infrastructure as a service from a growing number of data centers and network points of presence around the world. SoftLayer’s customers range from Web startups to global enterprises.
SYS-CON Events announced today that CA Technologies has been named “Platinum Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY, and the 21st International Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. CA Technologies helps customers succeed in a future where every business – from apparel to energy – is being rewritten by software. From ...
SYS-CON Events announced today that Technologic Systems Inc., an embedded systems solutions company, will exhibit at SYS-CON's @ThingsExpo, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Technologic Systems is an embedded systems company with headquarters in Fountain Hills, Arizona. They have been in business for 32 years, helping more than 8,000 OEM customers and building over a hundred COTS products that have never been discontinued. Technologic Systems’ pr...
SYS-CON Events announced today that Auditwerx will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Auditwerx specializes in SOC 1, SOC 2, and SOC 3 attestation services throughout the U.S. and Canada. As a division of Carr, Riggs & Ingram (CRI), one of the top 20 largest CPA firms nationally, you can expect the resources, skills, and experience of a much larger firm combined with the accessibility and attent...
SYS-CON Events announced today that HTBase will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. HTBase (Gartner 2016 Cool Vendor) delivers a Composable IT infrastructure solution architected for agility and increased efficiency. It turns compute, storage, and fabric into fluid pools of resources that are easily composed and re-composed to meet each application’s needs. With HTBase, companies can quickly prov...
SYS-CON Events announced today that Loom Systems will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Founded in 2015, Loom Systems delivers an advanced AI solution to predict and prevent problems in the digital business. Loom stands alone in the industry as an AI analysis platform requiring no prior math knowledge from operators, leveraging the existing staff to succeed in the digital era. With offices in S...
Buzzword alert: Microservices and IoT at a DevOps conference? What could possibly go wrong? In this Power Panel at DevOps Summit, moderated by Jason Bloomberg, the leading expert on architecting agility for the enterprise and president of Intellyx, panelists peeled away the buzz and discuss the important architectural principles behind implementing IoT solutions for the enterprise. As remote IoT devices and sensors become increasingly intelligent, they become part of our distributed cloud enviro...
SYS-CON Events announced today that T-Mobile will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. As America's Un-carrier, T-Mobile US, Inc., is redefining the way consumers and businesses buy wireless services through leading product and service innovation. The Company's advanced nationwide 4G LTE network delivers outstanding wireless experiences to 67.4 million customers who are unwilling to compromise on ...