Welcome!

Open Source Cloud Authors: Jyoti Bansal, Carmen Gonzalez, Jim Hansen, Shelly Palmer, Derek Weeks

Related Topics: FinTech Journal, Open Source Cloud, @CloudExpo

FinTech Journal: Blog Post

Tap-as-a-Service: Enabling Traffic Monitoring in OpenStack Clouds | @CloudExpo #API #Cloud

Tap-as-a-Service, a platform-oriented solution, designed to operate as an extension of Neutron, the OpenStack network service

It is no surprise that OpenStack has evolved into a widely adopted cloud management framework. As it hurtles on a trajectory of rapid growth, a new breed of demands are making themselves felt - demands that a mature platform of this nature and scope must satisfy. One such requirement is the ability to monitor traffic flowing in the myriad of virtual networks found in an OpenStack datacenter.

Conceptually speaking, the monitoring process involves placing tap devices at appropriate locations within the network infrastructure and attaching traffic analyzers to them. These analyzers can then see the same packets passing through those network segments, as if they were also in-line. A logical tap device can be easily constructed using the port-mirroring function of a network switching element, which makes it possible to have a copy of the packets traversing one or more switch ports delivered to another port on the same switch. This capability is supported by almost all modern physical and virtual switches. So, why is it [still] not possible to monitor the activity in OpenStack virtual networks?

The answer to this question lies in understanding two important architectural characteristics of cloud-based virtualization platforms, namely multi-tenancy and location independence. The former allows available resources and services to be shared among different groups of users. Each group - known as a tenant - is provided with an environment that is completely isolated from the others, to the extent where members of a group are oblivious of the fact that other groups may be co-existing with them. Multi-tenancy promotes delegation of control in a safe and secure manner. For example, tenants are permitted to create and administer their own private virtual networks. Location independence, on the other hand, is primarily concerned with hiding the identities of individual infrastructure components from the virtualized workloads. This has made it possible to relocate running virtual machines from one host to another. An equally important but perhaps less appreciated benefit of location independence is improved efficiency in resource allocation. Tenants are therefore unaware of the physical hosts on which their virtual machines are running. Furthermore, virtual machines belonging to different tenants may be placed on the same host. In such a shared ecosystem it makes sense that tenants are not given direct access to the underlying switch fabric, consisting of host-level virtual switches, top-of-rack switches, etc. This restriction avoids the possibility of any cross-tenant data leakage. Unfortunately, this means that the port-mirroring capability of those switches is also not available.

OpenStack is not alone when it comes to the lack of traffic monitoring support for its virtual networks. Other cloud solutions, including Amazon Web Services (AWS), also suffer from this limitation for the very reasons described above. There is one aspect of OpenStack, however, that makes it stand out from the rest. It is open source technology! This provides an opportunity for someone (actually anyone) who is able and willing to introduce new capabilities into the platform. At Gigamon we live and breathe network traffic visibility, so we decided to take it upon ourselves and become ‘that someone' who will push the needle forward. To our surprise and delight we soon learned of a group at Ericsson who had independently arrived at the same conclusion. Our goals matched perfectly and it seemed natural and fitting that we pool our resources together to solve this problem.

Our project is called Tap-as-a-Service. It is a platform-oriented solution, designed to operate as an extension of Neutron, the OpenStack network service. TaaS offers a simple API that will enable a tenant (or the cloud administrator) to monitor ports in Neutron provisioned networks. Since it is vital that tenant boundaries are not compromised, a tenant can only monitor its own ports, i.e., any port on one of its private virtual networks or a port created by it on a shared virtual network. The TaaS workflow begins with the creation of a tap-service instance that has a Neutron port serving as the destination side of a port-mirror session. A monitoring virtual machine is usually attached to this port to consume the mirrored traffic. Later, one or more tap-flows can be added to the tap-service instance. A tap-flow represents the association between a (source) port that is being monitored and a tap-service instance. TaaS allows a mirror session to span across multiple hosts, by virtue of remote port-mirroring, thereby ensuring that location independence is preserved.

A reference implementation of TaaS was completed earlier this year and the source code has been uploaded to Stackforge, the OpenStack incubator, where a dedicated GIT repository now exists for this project [1]. At the last OpenStack Summit in Vancouver (May 2015), we did a technical presentation on this work that included a live demonstration of traffic monitoring using TaaS [2]. The response has been very positive, with support pouring in from both the developer and user communities. We will continue to enhance TaaS; some of the planned features are integration with the OpenStack dashboard, support for virtual machine migration, pre-capture filtering and rate-limiting of mirrored traffic. At the same time, we are also continuing our discussions with the Neutron core team to have TaaS accepted as an integral part of future OpenStack releases.

Port-mirroring used to be a switch layer function. Tap-as-a-Service has effectively virtualized this capability and made it available for the users of Neutron provisioned networks. We see TaaS as the basic building block on top of which more complex traffic visibility solutions can be engineered for a diverse set of use cases, ranging from network administration and trouble-shooting to application/network security, data analytics and more.

References

  1. Tap-as-a-Service code repository.
  2. Tap-as-a-Service (TaaS): Port Monitoring for Neutron Networks. Alan Kavanagh, Anil Rao, Vinay Yadhav, OpenStack Summit, Vancouver, Canada, May 20, 2015.

More Stories By Anil Rao

Anil Rao is a Distinguished Engineer at Gigamon, leading research and development activities related to virtual machine traffic monitoring. His research interests include operating systems, distributed computing, virtual machine technology and software defined networking.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@ThingsExpo Stories
SYS-CON Events announced today that IoT Now has been named “Media Sponsor” of SYS-CON's 20th International Cloud Expo, which will take place on June 6–8, 2017, at the Javits Center in New York City, NY. IoT Now explores the evolving opportunities and challenges facing CSPs, and it passes on some lessons learned from those who have taken the first steps in next-gen IoT services.
SYS-CON Events announced today that Interoute, owner-operator of one of Europe's largest networks and a global cloud services platform, has been named “Bronze Sponsor” of SYS-CON's 20th Cloud Expo, which will take place on June 6-8, 2017 at the Javits Center in New York, New York. Interoute is the owner-operator of one of Europe's largest networks and a global cloud services platform which encompasses 12 data centers, 14 virtual data centers and 31 colocation centers, with connections to 195 add...
SYS-CON Events announced today that MobiDev, a client-oriented software development company, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place June 6-8, 2017, at the Javits Center in New York City, NY, and the 21st International Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. MobiDev is a software company that develops and delivers turn-key mobile apps, websites, web services, and complex softw...
DevOps is often described as a combination of technology and culture. Without both, DevOps isn't complete. However, applying the culture to outdated technology is a recipe for disaster; as response times grow and connections between teams are delayed by technology, the culture will die. A Nutanix Enterprise Cloud has many benefits that provide the needed base for a true DevOps paradigm.
What sort of WebRTC based applications can we expect to see over the next year and beyond? One way to predict development trends is to see what sorts of applications startups are building. In his session at @ThingsExpo, Arin Sime, founder of WebRTC.ventures, will discuss the current and likely future trends in WebRTC application development based on real requests for custom applications from real customers, as well as other public sources of information,
Keeping pace with advancements in software delivery processes and tooling is taxing even for the most proficient organizations. Point tools, platforms, open source and the increasing adoption of private and public cloud services requires strong engineering rigor - all in the face of developer demands to use the tools of choice. As Agile has settled in as a mainstream practice, now DevOps has emerged as the next wave to improve software delivery speed and output. To make DevOps work, organization...
As businesses adopt functionalities in cloud computing, it’s imperative that IT operations consistently ensure cloud systems work correctly – all of the time, and to their best capabilities. In his session at @BigDataExpo, Bernd Harzog, CEO and founder of OpsDataStore, will present an industry answer to the common question, “Are you running IT operations as efficiently and as cost effectively as you need to?” He will expound on the industry issues he frequently came up against as an analyst, and...
My team embarked on building a data lake for our sales and marketing data to better understand customer journeys. This required building a hybrid data pipeline to connect our cloud CRM with the new Hadoop Data Lake. One challenge is that IT was not in a position to provide support until we proved value and marketing did not have the experience, so we embarked on the journey ourselves within the product marketing team for our line of business within Progress. In his session at @BigDataExpo, Sum...
Apache Hadoop is emerging as a distributed platform for handling large and fast incoming streams of data. Predictive maintenance, supply chain optimization, and Internet-of-Things analysis are examples where Hadoop provides the scalable storage, processing, and analytics platform to gain meaningful insights from granular data that is typically only valuable from a large-scale, aggregate view. One architecture useful for capturing and analyzing streaming data is the Lambda Architecture, represent...
Things are changing so quickly in IoT that it would take a wizard to predict which ecosystem will gain the most traction. In order for IoT to reach its potential, smart devices must be able to work together. Today, there are a slew of interoperability standards being promoted by big names to make this happen: HomeKit, Brillo and Alljoyn. In his session at @ThingsExpo, Adam Justice, vice president and general manager of Grid Connect, will review what happens when smart devices don’t work togethe...
SYS-CON Events announced today that Ocean9will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Ocean9 provides cloud services for Backup, Disaster Recovery (DRaaS) and instant Innovation, and redefines enterprise infrastructure with its cloud native subscription offerings for mission critical SAP workloads.
In his session at @ThingsExpo, Eric Lachapelle, CEO of the Professional Evaluation and Certification Board (PECB), will provide an overview of various initiatives to certifiy the security of connected devices and future trends in ensuring public trust of IoT. Eric Lachapelle is the Chief Executive Officer of the Professional Evaluation and Certification Board (PECB), an international certification body. His role is to help companies and individuals to achieve professional, accredited and worldw...
SYS-CON Events announced today that Technologic Systems Inc., an embedded systems solutions company, will exhibit at SYS-CON's @ThingsExpo, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Technologic Systems is an embedded systems company with headquarters in Fountain Hills, Arizona. They have been in business for 32 years, helping more than 8,000 OEM customers and building over a hundred COTS products that have never been discontinued. Technologic Systems’ pr...
SYS-CON Events announced today that CA Technologies has been named “Platinum Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY, and the 21st International Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. CA Technologies helps customers succeed in a future where every business – from apparel to energy – is being rewritten by software. From ...
The taxi industry never saw Uber coming. Startups are a threat to incumbents like never before, and a major enabler for startups is that they are instantly “cloud ready.” If innovation moves at the pace of IT, then your company is in trouble. Why? Because your data center will not keep up with frenetic pace AWS, Microsoft and Google are rolling out new capabilities In his session at 20th Cloud Expo, Don Browning, VP of Cloud Architecture at Turner, will posit that disruption is inevitable for c...
SYS-CON Events announced today that Cloudistics, an on-premises cloud computing company, has been named “Bronze Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Cloudistics delivers a complete public cloud experience with composable on-premises infrastructures to medium and large enterprises. Its software-defined technology natively converges network, storage, compute, virtualization, and management into a ...
SYS-CON Events announced today that Loom Systems will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Founded in 2015, Loom Systems delivers an advanced AI solution to predict and prevent problems in the digital business. Loom stands alone in the industry as an AI analysis platform requiring no prior math knowledge from operators, leveraging the existing staff to succeed in the digital era. With offices in S...
The explosion of new web/cloud/IoT-based applications and the data they generate are transforming our world right before our eyes. In this rush to adopt these new technologies, organizations are often ignoring fundamental questions concerning who owns the data and failing to ask for permission to conduct invasive surveillance of their customers. Organizations that are not transparent about how their systems gather data telemetry without offering shared data ownership risk product rejection, regu...
SYS-CON Events announced today that CrowdReviews.com has been named “Media Sponsor” of SYS-CON's 20th International Cloud Expo, which will take place on June 6–8, 2017, at the Javits Center in New York City, NY. CrowdReviews.com is a transparent online platform for determining which products and services are the best based on the opinion of the crowd. The crowd consists of Internet users that have experienced products and services first-hand and have an interest in letting other potential buyers...
SYS-CON Events announced today that T-Mobile will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. As America's Un-carrier, T-Mobile US, Inc., is redefining the way consumers and businesses buy wireless services through leading product and service innovation. The Company's advanced nationwide 4G LTE network delivers outstanding wireless experiences to 67.4 million customers who are unwilling to compromise on ...