Welcome!

Open Source Cloud Authors: Liz McMillan, Vaibhaw Pandey, Stackify Blog, Pat Romanski, Wesley Coelho

Related Topics: Containers Expo Blog, Open Source Cloud, @CloudExpo

Containers Expo Blog: Blog Feed Post

SSP Failure to Cloud Storage Success

What a Difference a Decade Makes

Storability 10 year Reunion

Years ago, a few buddies and I started one of the first cloud storage providers. Of course, we didn’t call it cloud storage back then, but we merry band of brithers (and sisters), we first generation Storage Service Providers (1gSSPs) were cloud storage way before the cloud was cool.

All the 1gSSPs – StorageNetworks, ScaleEight, StorageWay, Sanrise, and others – failed. The core problem was and still is that renting raw capacity over the network is a lousy business model.

  • 1gSSPs couldn’t sustainably buy their storage cheaper than their retail customers (although over a beer I can share some great stories of how the early 1gSSP robber-barron’s ‘negotiated’ with the storage vendors during the boom).
  • SSPs couldn’t sustainably offer broad enough management efficiencies to generate profits.
  • SSPs couldn’t overcome a host of logistic and cultural issues (network performance/cost, stigma/liability of releasing core data, etc).

After the bust, and the 911 attacks, the entire business simply collapsed. Some of us – my company, Storability and others like Arsenal Digital - managed to flip over to providing managed storage services – running NOCs, and doing backups and restores for our customers. Wasn’t a great business, but we survived long enough to eventually be sold off.

For those interested in an unbiased history of the 1gSSP market, there is a thorough and thoughtful analysis from the National Center for Supercomputing Applications (NCSA), University of Illinois at Urbana-Champaign (UIUC) posted here.

Ten years later, things look different, and the same.

A whole host of storage service providers – nee’ cloud storage providers – has arisen, not so much from the ashes of the 1gSSPs, but certainly with their dust in the new CSP DNA. These folks have it a little easier than we did back then, and I think more than a few of them are going to make an honest living this time.

In addition to the obvious improvements in network connectivity, bandwidth, and reliability, I see three critical changes that I believe will mark the difference between the past failure of 1gSSPs and the future success of today’s Cloud Storage Providers – file systems, file virtualization, and file storage gateways.

File Systems

Data used to be stored as long strings of 1’s and 0’s, actually millions and billions of 1’s and 0’s we called megabytes, gigabytes, petabytes, etc.  Back in the 1gSSP days, applications like database management systems untangled those 1’s and 0’s and formed them into useful information like bank account records and social security numbers.

Today, data is still made up of 1's and 0's, but the fastest growing forms of data, from the pictures you upload with your cell phone, to the books you download to your Kindle, come packaged in a convenient format called a file.

Files matter – files have digital labels that convey information about the file package itself.  Raw strings of 1’s and 0’s don’t.  More importantly, files have business and human context - 1's and 0's not so much.

Context matters - with it, we can make decisions about how to treat data. With files, we can look at the metadata (the data about the data contained in the label or header attached to the file itself) and learn who created the file, how old it is, and even gain hints about its actual content (does the file contain a song or a spreadsheet?). With this information, we can make intelligent decisions about where to put the file, how many copies we should make, how often we should back it up for safekeeping, etc.

With raw megabytes – no context - we have no way of discerning what’s what, so we have to treat the entire string of 1’s and 0’s the same – in most cases that means treating it all as if it’s all vitally important.

1gSSPs got sort of a raw deal trying to build a business storing all that raw data.

  • They had to treat it all the same – backing it all up every night, for instance.
  • They had to connect it directly to live applications. The banking app needs instant anytime access to the entire database – no telling when you might make an ATM withdrawal – and apps don’t like to wait for data, so the connection has to be very high speed (laws of physics and economics apply here).
  • They had to have it all – because they couldn’t discern one cluster of 1’s and 0’s from another, customers had to trust the SSP with all their raw data.

Files make life easier for today's wannabe Cloud Storage Provider.

  • Customers can decide and control what files go to the cloud
  • CSPs can offer differentiated services for files based on metadata
  • Applications are not as dependent on instant and constant access to files - they've learned to be patient waiting for downloads, just like the rest of us.
  • Files can be uploaded and downloaded between users and CSPs with ease, so variability in persistence and performance of the connection is better tolerated

File Virtualization

So, the ability to decide and control file location is critical for the success of cloud storage, but it's not enough.

If we know that Sally’s MP3 file of Andrea Boccelli’s “Silent Night” is non-business-critical (albeit absolutely amazing and worth downloading today), we can decide to push Sally’s file to a cheap storage device, and not back it up, saving us money and effort. We might even decide to upload Sally's file to a Cloud Service Provider that offers essentially free storage capacity, and really save the company some dough.

BUT…how will Sally know where it is when she goes to download it next Christmastime? Whoops.

Important point - moving files and treating them differently based on metadata is great, but users and applications cannot be expected to keep track of constantly changing file locations. So cloud storage won’t fly as a business model if Sally or her apps need to keep track of what’s where in the cloud. 

Enter file virtualization, a technology which masks the file's physical location.

File virtualization matters – with a virtualized file structure, regardless of where it physically resides, Sally and Sally’s applications are tricked into thinking Sally’s file is on her network drive at G:/Sally/Music/SilentNight.MP3.  She never realizes, and does not need to know, that it’s been moved, thus the Cloud Storage business model becomes viable.

File Storage Gateways

OK, so now we can decide, move, and eliminate the disruption of moving. So far so good, but the Cloud Storage Business needs one more piece of connecting tissue to reach the tipping point.

If all we care about is Sally and her music, the cloud storage business is pretty simple and in fact a bunch of free or almost free services abound that do just that. Though I admit it is obviously possible (duh, Facebook) I don’t know how to make money off ‘free’ so I am leaving that model alone.

In order to have a successful enterprise oriented (paying customer) cloud storage business, CSPs need the rough equivalent of a set-top box they can provide to the customer. Today, most CSPs offer a programmatic interface to upload and download files, which is kludgy at best, and isn’t going to scale in a commercial environment.

  • No customer is going to want to be locked into a single CSP, or be forced to adapt their infrastructure or modify their applications to support one vendor's cloud model.
  • Latency is an issue - no matter what we do to reduce the performance imperative, we are eventually going to have to accept the logic that some subset of cloud resident files must reside at least temporarily at the customer premises (sort of like the difference between downloading and streaming movies).

File storage gateways matter – with a gateway in place the customer can treat the cloud just like another storage device. Sure, the vast majority of spinning disks are now located at the CSP, but to the customer the CSP (through the File Storage Gateway) appears to be just another NAS box - albeit a cheap one, that never fills up, and never needs to be backed up.

Up until recently, there have been a few FSG startups poking about, which has been useful for vetting and growing the concept.  Fortunately, for commercial CSPs, serious and trusted vendors are now releasing FSGs.

So now, I believe we finally have the necessary infrastructure and technology for Cloud Storage success.  It's now possible to decide what data can, and control what data will, be safely stored in the cloud.  Once separated and moved, it's possible to decide and control how data is treated when it gets to the cloud.  It's now possible to do all this without disrupting users and applications. Moving from one CSP to another is now simple and non-disruptive. The performance and persistence issues that plagued 1gSSPs are under control. Modifications to the files, user behavior, and application intelligence are no longer necessary to achieve the benefits of cloud storage.

To my mind, the combination of these three major changes in the storage landscape – massive reliance on file systems, commercialization of file virtualization, and emergence of viable file storage gateways have now combined to eliminate the barriers and challenges we faced in the 1gSSP days, and together provide the technical and process infrastructure necessary for cloud storage to finally reach its full potential.

With the technical and logistical hurdles out of the way, it will be up to the skill of the players to decide who wins.

All best wishes go out to the next generation of cloud storage entrepreneurs – as we brithers say, ladies and gentlemen, the ice is yours. Good curling!

Read the original blog entry...

More Stories By Kirby Wadsworth

Kirby is widely recognized throughout the storage industry for his expertise in marketing and business strategy.

A veteran of both startups and established storage vendors, Wadsworth was a founder of Storability and served as vice president of marketing prior to its sale to StorageTek. Earlier, as vice president and general manager of Compaq's Network Storage Services Business Unit, he envisioned and introduced Compaq's Enterprise Network Storage Architecture (ENSA) which is still widely recognized today.

As vice president of marketing for Digital's Storage Business Unit, Wadsworth launched Digital's StorageWorks product line into the open systems marketplace, and led the creation and introduction of the Enterprise Storage Array product family.

@ThingsExpo Stories
Widespread fragmentation is stalling the growth of the IIoT and making it difficult for partners to work together. The number of software platforms, apps, hardware and connectivity standards is creating paralysis among businesses that are afraid of being locked into a solution. EdgeX Foundry is unifying the community around a common IoT edge framework and an ecosystem of interoperable components.
"MobiDev is a software development company and we do complex, custom software development for everybody from entrepreneurs to large enterprises," explained Alan Winters, U.S. Head of Business Development at MobiDev, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
In his session at 21st Cloud Expo, Carl J. Levine, Senior Technical Evangelist for NS1, will objectively discuss how DNS is used to solve Digital Transformation challenges in large SaaS applications, CDNs, AdTech platforms, and other demanding use cases. Carl J. Levine is the Senior Technical Evangelist for NS1. A veteran of the Internet Infrastructure space, he has over a decade of experience with startups, networking protocols and Internet infrastructure, combined with the unique ability to it...
"Space Monkey by Vivent Smart Home is a product that is a distributed cloud-based edge storage network. Vivent Smart Home, our parent company, is a smart home provider that places a lot of hard drives across homes in North America," explained JT Olds, Director of Engineering, and Brandon Crowfeather, Product Manager, at Vivint Smart Home, in this SYS-CON.tv interview at @ThingsExpo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
"IBM is really all in on blockchain. We take a look at sort of the history of blockchain ledger technologies. It started out with bitcoin, Ethereum, and IBM evaluated these particular blockchain technologies and found they were anonymous and permissionless and that many companies were looking for permissioned blockchain," stated René Bostic, Technical VP of the IBM Cloud Unit in North America, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Conventi...
"Akvelon is a software development company and we also provide consultancy services to folks who are looking to scale or accelerate their engineering roadmaps," explained Jeremiah Mothersell, Marketing Manager at Akvelon, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
Coca-Cola’s Google powered digital signage system lays the groundwork for a more valuable connection between Coke and its customers. Digital signs pair software with high-resolution displays so that a message can be changed instantly based on what the operator wants to communicate or sell. In their Day 3 Keynote at 21st Cloud Expo, Greg Chambers, Global Group Director, Digital Innovation, Coca-Cola, and Vidya Nagarajan, a Senior Product Manager at Google, discussed how from store operations and ...
Large industrial manufacturing organizations are adopting the agile principles of cloud software companies. The industrial manufacturing development process has not scaled over time. Now that design CAD teams are geographically distributed, centralizing their work is key. With large multi-gigabyte projects, outdated tools have stifled industrial team agility, time-to-market milestones, and impacted P&L stakeholders.
Gemini is Yahoo’s native and search advertising platform. To ensure the quality of a complex distributed system that spans multiple products and components and across various desktop websites and mobile app and web experiences – both Yahoo owned and operated and third-party syndication (supply), with complex interaction with more than a billion users and numerous advertisers globally (demand) – it becomes imperative to automate a set of end-to-end tests 24x7 to detect bugs and regression. In th...
"Cloud Academy is an enterprise training platform for the cloud, specifically public clouds. We offer guided learning experiences on AWS, Azure, Google Cloud and all the surrounding methodologies and technologies that you need to know and your teams need to know in order to leverage the full benefits of the cloud," explained Alex Brower, VP of Marketing at Cloud Academy, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clar...
"There's plenty of bandwidth out there but it's never in the right place. So what Cedexis does is uses data to work out the best pathways to get data from the origin to the person who wants to get it," explained Simon Jones, Evangelist and Head of Marketing at Cedexis, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
SYS-CON Events announced today that CrowdReviews.com has been named “Media Sponsor” of SYS-CON's 22nd International Cloud Expo, which will take place on June 5–7, 2018, at the Javits Center in New York City, NY. CrowdReviews.com is a transparent online platform for determining which products and services are the best based on the opinion of the crowd. The crowd consists of Internet users that have experienced products and services first-hand and have an interest in letting other potential buye...
SYS-CON Events announced today that Telecom Reseller has been named “Media Sponsor” of SYS-CON's 22nd International Cloud Expo, which will take place on June 5-7, 2018, at the Javits Center in New York, NY. Telecom Reseller reports on Unified Communications, UCaaS, BPaaS for enterprise and SMBs. They report extensively on both customer premises based solutions such as IP-PBX as well as cloud based and hosted platforms.
It is of utmost importance for the future success of WebRTC to ensure that interoperability is operational between web browsers and any WebRTC-compliant client. To be guaranteed as operational and effective, interoperability must be tested extensively by establishing WebRTC data and media connections between different web browsers running on different devices and operating systems. In his session at WebRTC Summit at @ThingsExpo, Dr. Alex Gouaillard, CEO and Founder of CoSMo Software, presented ...
WebRTC is great technology to build your own communication tools. It will be even more exciting experience it with advanced devices, such as a 360 Camera, 360 microphone, and a depth sensor camera. In his session at @ThingsExpo, Masashi Ganeko, a manager at INFOCOM Corporation, introduced two experimental projects from his team and what they learned from them. "Shotoku Tamago" uses the robot audition software HARK to track speakers in 360 video of a remote party. "Virtual Teleport" uses a multip...
A strange thing is happening along the way to the Internet of Things, namely far too many devices to work with and manage. It has become clear that we'll need much higher efficiency user experiences that can allow us to more easily and scalably work with the thousands of devices that will soon be in each of our lives. Enter the conversational interface revolution, combining bots we can literally talk with, gesture to, and even direct with our thoughts, with embedded artificial intelligence, whic...
SYS-CON Events announced today that Evatronix will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Evatronix SA offers comprehensive solutions in the design and implementation of electronic systems, in CAD / CAM deployment, and also is a designer and manufacturer of advanced 3D scanners for professional applications.
Leading companies, from the Global Fortune 500 to the smallest companies, are adopting hybrid cloud as the path to business advantage. Hybrid cloud depends on cloud services and on-premises infrastructure working in unison. Successful implementations require new levels of data mobility, enabled by an automated and seamless flow across on-premises and cloud resources. In his general session at 21st Cloud Expo, Greg Tevis, an IBM Storage Software Technical Strategist and Customer Solution Architec...
To get the most out of their data, successful companies are not focusing on queries and data lakes, they are actively integrating analytics into their operations with a data-first application development approach. Real-time adjustments to improve revenues, reduce costs, or mitigate risk rely on applications that minimize latency on a variety of data sources. In his session at @BigDataExpo, Jack Norris, Senior Vice President, Data and Applications at MapR Technologies, reviewed best practices to ...
An increasing number of companies are creating products that combine data with analytical capabilities. Running interactive queries on Big Data requires complex architectures to store and query data effectively, typically involving data streams, an choosing efficient file format/database and multiple independent systems that are tied together through custom-engineered pipelines. In his session at @BigDataExpo at @ThingsExpo, Tomer Levi, a senior software engineer at Intel’s Advanced Analytics gr...