Open Source Cloud Authors: Pat Romanski, Elizabeth White, Rostyslav Demush, Yeshim Deniz, Harry Trott

Related Topics: Containers Expo Blog, Open Source Cloud, @CloudExpo

Containers Expo Blog: Blog Feed Post

SSP Failure to Cloud Storage Success

What a Difference a Decade Makes

Storability 10 year Reunion

Years ago, a few buddies and I started one of the first cloud storage providers. Of course, we didn’t call it cloud storage back then, but we merry band of brithers (and sisters), we first generation Storage Service Providers (1gSSPs) were cloud storage way before the cloud was cool.

All the 1gSSPs – StorageNetworks, ScaleEight, StorageWay, Sanrise, and others – failed. The core problem was and still is that renting raw capacity over the network is a lousy business model.

  • 1gSSPs couldn’t sustainably buy their storage cheaper than their retail customers (although over a beer I can share some great stories of how the early 1gSSP robber-barron’s ‘negotiated’ with the storage vendors during the boom).
  • SSPs couldn’t sustainably offer broad enough management efficiencies to generate profits.
  • SSPs couldn’t overcome a host of logistic and cultural issues (network performance/cost, stigma/liability of releasing core data, etc).

After the bust, and the 911 attacks, the entire business simply collapsed. Some of us – my company, Storability and others like Arsenal Digital - managed to flip over to providing managed storage services – running NOCs, and doing backups and restores for our customers. Wasn’t a great business, but we survived long enough to eventually be sold off.

For those interested in an unbiased history of the 1gSSP market, there is a thorough and thoughtful analysis from the National Center for Supercomputing Applications (NCSA), University of Illinois at Urbana-Champaign (UIUC) posted here.

Ten years later, things look different, and the same.

A whole host of storage service providers – nee’ cloud storage providers – has arisen, not so much from the ashes of the 1gSSPs, but certainly with their dust in the new CSP DNA. These folks have it a little easier than we did back then, and I think more than a few of them are going to make an honest living this time.

In addition to the obvious improvements in network connectivity, bandwidth, and reliability, I see three critical changes that I believe will mark the difference between the past failure of 1gSSPs and the future success of today’s Cloud Storage Providers – file systems, file virtualization, and file storage gateways.

File Systems

Data used to be stored as long strings of 1’s and 0’s, actually millions and billions of 1’s and 0’s we called megabytes, gigabytes, petabytes, etc.  Back in the 1gSSP days, applications like database management systems untangled those 1’s and 0’s and formed them into useful information like bank account records and social security numbers.

Today, data is still made up of 1's and 0's, but the fastest growing forms of data, from the pictures you upload with your cell phone, to the books you download to your Kindle, come packaged in a convenient format called a file.

Files matter – files have digital labels that convey information about the file package itself.  Raw strings of 1’s and 0’s don’t.  More importantly, files have business and human context - 1's and 0's not so much.

Context matters - with it, we can make decisions about how to treat data. With files, we can look at the metadata (the data about the data contained in the label or header attached to the file itself) and learn who created the file, how old it is, and even gain hints about its actual content (does the file contain a song or a spreadsheet?). With this information, we can make intelligent decisions about where to put the file, how many copies we should make, how often we should back it up for safekeeping, etc.

With raw megabytes – no context - we have no way of discerning what’s what, so we have to treat the entire string of 1’s and 0’s the same – in most cases that means treating it all as if it’s all vitally important.

1gSSPs got sort of a raw deal trying to build a business storing all that raw data.

  • They had to treat it all the same – backing it all up every night, for instance.
  • They had to connect it directly to live applications. The banking app needs instant anytime access to the entire database – no telling when you might make an ATM withdrawal – and apps don’t like to wait for data, so the connection has to be very high speed (laws of physics and economics apply here).
  • They had to have it all – because they couldn’t discern one cluster of 1’s and 0’s from another, customers had to trust the SSP with all their raw data.

Files make life easier for today's wannabe Cloud Storage Provider.

  • Customers can decide and control what files go to the cloud
  • CSPs can offer differentiated services for files based on metadata
  • Applications are not as dependent on instant and constant access to files - they've learned to be patient waiting for downloads, just like the rest of us.
  • Files can be uploaded and downloaded between users and CSPs with ease, so variability in persistence and performance of the connection is better tolerated

File Virtualization

So, the ability to decide and control file location is critical for the success of cloud storage, but it's not enough.

If we know that Sally’s MP3 file of Andrea Boccelli’s “Silent Night” is non-business-critical (albeit absolutely amazing and worth downloading today), we can decide to push Sally’s file to a cheap storage device, and not back it up, saving us money and effort. We might even decide to upload Sally's file to a Cloud Service Provider that offers essentially free storage capacity, and really save the company some dough.

BUT…how will Sally know where it is when she goes to download it next Christmastime? Whoops.

Important point - moving files and treating them differently based on metadata is great, but users and applications cannot be expected to keep track of constantly changing file locations. So cloud storage won’t fly as a business model if Sally or her apps need to keep track of what’s where in the cloud. 

Enter file virtualization, a technology which masks the file's physical location.

File virtualization matters – with a virtualized file structure, regardless of where it physically resides, Sally and Sally’s applications are tricked into thinking Sally’s file is on her network drive at G:/Sally/Music/SilentNight.MP3.  She never realizes, and does not need to know, that it’s been moved, thus the Cloud Storage business model becomes viable.

File Storage Gateways

OK, so now we can decide, move, and eliminate the disruption of moving. So far so good, but the Cloud Storage Business needs one more piece of connecting tissue to reach the tipping point.

If all we care about is Sally and her music, the cloud storage business is pretty simple and in fact a bunch of free or almost free services abound that do just that. Though I admit it is obviously possible (duh, Facebook) I don’t know how to make money off ‘free’ so I am leaving that model alone.

In order to have a successful enterprise oriented (paying customer) cloud storage business, CSPs need the rough equivalent of a set-top box they can provide to the customer. Today, most CSPs offer a programmatic interface to upload and download files, which is kludgy at best, and isn’t going to scale in a commercial environment.

  • No customer is going to want to be locked into a single CSP, or be forced to adapt their infrastructure or modify their applications to support one vendor's cloud model.
  • Latency is an issue - no matter what we do to reduce the performance imperative, we are eventually going to have to accept the logic that some subset of cloud resident files must reside at least temporarily at the customer premises (sort of like the difference between downloading and streaming movies).

File storage gateways matter – with a gateway in place the customer can treat the cloud just like another storage device. Sure, the vast majority of spinning disks are now located at the CSP, but to the customer the CSP (through the File Storage Gateway) appears to be just another NAS box - albeit a cheap one, that never fills up, and never needs to be backed up.

Up until recently, there have been a few FSG startups poking about, which has been useful for vetting and growing the concept.  Fortunately, for commercial CSPs, serious and trusted vendors are now releasing FSGs.

So now, I believe we finally have the necessary infrastructure and technology for Cloud Storage success.  It's now possible to decide what data can, and control what data will, be safely stored in the cloud.  Once separated and moved, it's possible to decide and control how data is treated when it gets to the cloud.  It's now possible to do all this without disrupting users and applications. Moving from one CSP to another is now simple and non-disruptive. The performance and persistence issues that plagued 1gSSPs are under control. Modifications to the files, user behavior, and application intelligence are no longer necessary to achieve the benefits of cloud storage.

To my mind, the combination of these three major changes in the storage landscape – massive reliance on file systems, commercialization of file virtualization, and emergence of viable file storage gateways have now combined to eliminate the barriers and challenges we faced in the 1gSSP days, and together provide the technical and process infrastructure necessary for cloud storage to finally reach its full potential.

With the technical and logistical hurdles out of the way, it will be up to the skill of the players to decide who wins.

All best wishes go out to the next generation of cloud storage entrepreneurs – as we brithers say, ladies and gentlemen, the ice is yours. Good curling!

Read the original blog entry...

More Stories By Kirby Wadsworth

Kirby is widely recognized throughout the storage industry for his expertise in marketing and business strategy.

A veteran of both startups and established storage vendors, Wadsworth was a founder of Storability and served as vice president of marketing prior to its sale to StorageTek. Earlier, as vice president and general manager of Compaq's Network Storage Services Business Unit, he envisioned and introduced Compaq's Enterprise Network Storage Architecture (ENSA) which is still widely recognized today.

As vice president of marketing for Digital's Storage Business Unit, Wadsworth launched Digital's StorageWorks product line into the open systems marketplace, and led the creation and introduction of the Enterprise Storage Array product family.

@ThingsExpo Stories
Charles Araujo is an industry analyst, internationally recognized authority on the Digital Enterprise and author of The Quantum Age of IT: Why Everything You Know About IT is About to Change. As Principal Analyst with Intellyx, he writes, speaks and advises organizations on how to navigate through this time of disruption. He is also the founder of The Institute for Digital Transformation and a sought after keynote speaker. He has been a regular contributor to both InformationWeek and CIO Insight...
DXWorldEXPO LLC, the producer of the world's most influential technology conferences and trade shows has announced the 22nd International CloudEXPO | DXWorldEXPO "Early Bird Registration" is now open. Register for Full Conference "Gold Pass" ▸ Here (Expo Hall ▸ Here)
Join IBM November 1 at 21st Cloud Expo at the Santa Clara Convention Center in Santa Clara, CA, and learn how IBM Watson can bring cognitive services and AI to intelligent, unmanned systems. Cognitive analysis impacts today’s systems with unparalleled ability that were previously available only to manned, back-end operations. Thanks to cloud processing, IBM Watson can bring cognitive services and AI to intelligent, unmanned systems. Imagine a robot vacuum that becomes your personal assistant tha...
"MobiDev is a software development company and we do complex, custom software development for everybody from entrepreneurs to large enterprises," explained Alan Winters, U.S. Head of Business Development at MobiDev, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
I think DevOps is now a rambunctious teenager - it's starting to get a mind of its own, wanting to get its own things but it still needs some adult supervision," explained Thomas Hooker, VP of marketing at CollabNet, in this SYS-CON.tv interview at DevOps Summit at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
Recently, WebRTC has a lot of eyes from market. The use cases of WebRTC are expanding - video chat, online education, online health care etc. Not only for human-to-human communication, but also IoT use cases such as machine to human use cases can be seen recently. One of the typical use-case is remote camera monitoring. With WebRTC, people can have interoperability and flexibility for deploying monitoring service. However, the benefit of WebRTC for IoT is not only its convenience and interopera...
Cloud-enabled transformation has evolved from cost saving measure to business innovation strategy -- one that combines the cloud with cognitive capabilities to drive market disruption. Learn how you can achieve the insight and agility you need to gain a competitive advantage. Industry-acclaimed CTO and cloud expert, Shankar Kalyana presents. Only the most exceptional IBMers are appointed with the rare distinction of IBM Fellow, the highest technical honor in the company. Shankar has also receive...
It is of utmost importance for the future success of WebRTC to ensure that interoperability is operational between web browsers and any WebRTC-compliant client. To be guaranteed as operational and effective, interoperability must be tested extensively by establishing WebRTC data and media connections between different web browsers running on different devices and operating systems. In his session at WebRTC Summit at @ThingsExpo, Dr. Alex Gouaillard, CEO and Founder of CoSMo Software, presented ...
WebRTC is great technology to build your own communication tools. It will be even more exciting experience it with advanced devices, such as a 360 Camera, 360 microphone, and a depth sensor camera. In his session at @ThingsExpo, Masashi Ganeko, a manager at INFOCOM Corporation, introduced two experimental projects from his team and what they learned from them. "Shotoku Tamago" uses the robot audition software HARK to track speakers in 360 video of a remote party. "Virtual Teleport" uses a multip...
Business professionals no longer wonder if they'll migrate to the cloud; it's now a matter of when. The cloud environment has proved to be a major force in transitioning to an agile business model that enables quick decisions and fast implementation that solidify customer relationships. And when the cloud is combined with the power of cognitive computing, it drives innovation and transformation that achieves astounding competitive advantage.
Data is the fuel that drives the machine learning algorithmic engines and ultimately provides the business value. In his session at Cloud Expo, Ed Featherston, a director and senior enterprise architect at Collaborative Consulting, discussed the key considerations around quality, volume, timeliness, and pedigree that must be dealt with in order to properly fuel that engine.
IoT is rapidly becoming mainstream as more and more investments are made into the platforms and technology. As this movement continues to expand and gain momentum it creates a massive wall of noise that can be difficult to sift through. Unfortunately, this inevitably makes IoT less approachable for people to get started with and can hamper efforts to integrate this key technology into your own portfolio. There are so many connected products already in place today with many hundreds more on the h...
When shopping for a new data processing platform for IoT solutions, many development teams want to be able to test-drive options before making a choice. Yet when evaluating an IoT solution, it’s simply not feasible to do so at scale with physical devices. Building a sensor simulator is the next best choice; however, generating a realistic simulation at very high TPS with ease of configurability is a formidable challenge. When dealing with multiple application or transport protocols, you would be...
Detecting internal user threats in the Big Data eco-system is challenging and cumbersome. Many organizations monitor internal usage of the Big Data eco-system using a set of alerts. This is not a scalable process given the increase in the number of alerts with the accelerating growth in data volume and user base. Organizations are increasingly leveraging machine learning to monitor only those data elements that are sensitive and critical, autonomously establish monitoring policies, and to detect...
In his keynote at 18th Cloud Expo, Andrew Keys, Co-Founder of ConsenSys Enterprise, provided an overview of the evolution of the Internet and the Database and the future of their combination – the Blockchain. Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life settl...
In his session at @ThingsExpo, Dr. Robert Cohen, an economist and senior fellow at the Economic Strategy Institute, presented the findings of a series of six detailed case studies of how large corporations are implementing IoT. The session explored how IoT has improved their economic performance, had major impacts on business models and resulted in impressive ROIs. The companies covered span manufacturing and services firms. He also explored servicification, how manufacturing firms shift from se...
DevOpsSummit New York 2018, colocated with CloudEXPO | DXWorldEXPO New York 2018 will be held November 11-13, 2018, in New York City. Digital Transformation (DX) is a major focus with the introduction of DXWorldEXPO within the program. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term. A total of 88% of Fortune 500 companies from a generation ago are now out of bus...
The Jevons Paradox suggests that when technological advances increase efficiency of a resource, it results in an overall increase in consumption. Writing on the increased use of coal as a result of technological improvements, 19th-century economist William Stanley Jevons found that these improvements led to the development of new ways to utilize coal. In his session at 19th Cloud Expo, Mark Thiele, Chief Strategy Officer for Apcera, compared the Jevons Paradox to modern-day enterprise IT, examin...
IoT solutions exploit operational data generated by Internet-connected smart “things” for the purpose of gaining operational insight and producing “better outcomes” (for example, create new business models, eliminate unscheduled maintenance, etc.). The explosive proliferation of IoT solutions will result in an exponential growth in the volume of IoT data, precipitating significant Information Governance issues: who owns the IoT data, what are the rights/duties of IoT solutions adopters towards t...
Amazon started as an online bookseller 20 years ago. Since then, it has evolved into a technology juggernaut that has disrupted multiple markets and industries and touches many aspects of our lives. It is a relentless technology and business model innovator driving disruption throughout numerous ecosystems. Amazon’s AWS revenues alone are approaching $16B a year making it one of the largest IT companies in the world. With dominant offerings in Cloud, IoT, eCommerce, Big Data, AI, Digital Assista...