Welcome!

Open Source Cloud Authors: Pat Romanski, Elizabeth White, Liz McMillan, Wesley Coelho, Rainer Ersch

Related Topics: Open Source Cloud, @CloudExpo

Open Source Cloud: Blog Feed Post

Moving to the Cloud: Key Considerations for Cloud Storage

Cloud storage can be simultaneously simple and complex – just like cloud computing in general

This post is part of a series examining the issues involved when moving applications between internal data centers and public clouds.

The true challenges in storage and data management in the cloud result from the diverse and often unfamiliar processes and infrastructures offered by the cloud providers, including: new provisioning methods, storage properties, data population and transfer, and systems for data management (snapshots, clones, replication, backup). The cloud providers define the relationship between servers and storage and often impose constraints on everything from allocation size limits to the ways in which storage is managed. These are just some of the things you’ll want to consider as you start to think about integrating cloud computing into your existing IT environments.

I’d like to focus in detail on the complexity and variability of cloud provisioning and storage properties. There are different models for storage in existing compute clouds, with the most common being an “inclusive” storage model. In this model, each server comes with a certain amount of storage attached to it. The storage is a fixed capacity that is provisioned when you create the server from the pre-existing templates.

For example, Rackspace gives you disk space that is proportional to the memory (RAM) size you select.  The smallest memory/disk combination is 256MB of memory with 10GB of disk. With each doubling of memory, the disk space is also doubled until you get to roughly 16GB of memory and 640GB of disk.  With the new Terremark vCloud Express, you get a system disk that is predefined for each “template” server you select.  For a standard Linux distribution, you get a 10GB system disk, for Windows 2K3 you get a 20GB disk and for W2K8, you get a 40GB disk. Terremark’s vCloud Express allows you to add additional storage as new disks, while others (like Rackspace) allow to “resize” your servers and storage to create a new server with a larger disk and copy your data into it.

Amazon offers several distinct types of storage within EC2. The default storage you get with each server you create in the cloud is called “ephemeral” storage. You then have the option of allocating and attaching Elastic Block Storage (EBS), and there is also an object store system called Simple Storage Service (S3).  Ephemeral and EBS are standard “block storage” devices – meaning they are viewed and used as disks attached to your server (/dev/sdg in Linux, D: in Windows) while S3 requires an API or other tools to integrate with your systems. The good part about the EC2 storage offerings is that you have some powerful options as you build for the cloud; the hard part is mapping the proper resources to your applications and integrating this with your existing processes. Specifically, the base storage is ephemeral, which means that if you power-off the server, or it has a hard fault, all the data on that storage is lost. This means that everything on these drives (boot parameters, application updates, user data, logs, etc.) is subject to loss when you power off the machine. There are several methods of handling this situation: 1) Build your servers every time you start them from a formula or other sources such that you don’t depend on the base storage being persistent; 2) Use Amazon or third party tool sets to periodically “bundle” your servers into S3 (effectively taking a snapshot of the server); or 3) Attach EBS storage to your image and store your important data on persistent storage.

Turning to granularity, we find a wide range in the units or increments of available storage in the various cloud providers. There is the “included” storage mentioned above that is often based on the size of the server and the requested OS type. To add storage, we find cloud providers (such as Amazon) allowing 1GB increments up to 1TB, and others (like Flexiscale) allowing only fixed increments of 50GB/100GB/250GB. For Rackspace, you can resize both the server and storage according to the defined fixed ratios, but these are bound to memory and CPU so there is no independent scaling of storage. The bare-metal cloud provider NewServers allows iSCSI storage to be attached to your servers in 250GB increments. In the cloud these varied increments really matter, because you are paying by the GB/month and if you need just a little more storage, you could end up having to purchase 10x more storage than you need, or having to pay for more memory and compute than you need.

The conclusion we can draw is that there are numerous storage configuration options in the cloud, and these options become linked to the server “flavors” defined by individual cloud providers. Because you don’t have the same control or even mechanisms in the cloud as you do in the local data center, the manner in which you allocate, populate and manage data in the cloud will be different. The work you do to understand and map your applications’ requirements into cloud-based storage requires changing your processes to match those of the cloud, and often this work is cloud-specific.

Beyond configuration issues, of course, there are many other concerns. For example, with data management, you have to determine how you will get your data into the system, how to grow your systems and how to protect your data. Right now most clouds use template servers that you have to build up from a pre-installed base operating system using update mechanisms and then re-installing the application components. As for protecting your data, there are also many cloud-specific options available – from RAID-protected EBS in Amazon, to data snapshots and cloning, to backup services offered by companies like Rackspace.

The bottom line is that cloud storage can be simultaneously simple and complex – just like cloud computing in general. It’s simple to use if you just want to try something new; complex if you want to integrate cloud storage into your existing processes and infrastructures.

Next: Networking in the Cloud

Read the original blog entry...

More Stories By Ellen Rubin

Ellen Rubin is the CEO and co-founder of ClearSky Data, an enterprise storage company that recently raised $27 million in a Series B investment round. She is an experienced entrepreneur with a record in leading strategy, market positioning and go-to- market efforts for fast-growing companies. Most recently, she was co-founder of CloudSwitch, a cloud enablement software company, acquired by Verizon in 2011. Prior to founding CloudSwitch, Ellen was the vice president of marketing at Netezza, where as a member of the early management team, she helped grow the company to more than $130 million in revenues and a successful IPO in 2007. Ellen holds an MBA from Harvard Business School and an undergraduate degree magna cum laude from Harvard University.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@ThingsExpo Stories
SYS-CON Events announced today that Massive Networks will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Massive Networks mission is simple. To help your business operate seamlessly with fast, reliable, and secure internet and network solutions. Improve your customer's experience with outstanding connections to your cloud.
DX World EXPO, LLC., a Lighthouse Point, Florida-based startup trade show producer and the creator of "DXWorldEXPO® - Digital Transformation Conference & Expo" has announced its executive management team. The team is headed by Levent Selamoglu, who has been named CEO. "Now is the time for a truly global DX event, to bring together the leading minds from the technology world in a conversation about Digital Transformation," he said in making the announcement.
Internet of @ThingsExpo, taking place October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 21st Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The Internet of Things (IoT) is the most profound change in personal and enterprise IT since the creation of the Worldwide Web more than 20 years ago. All major researchers estimate there will be tens of billions devic...
"The Striim platform is a full end-to-end streaming integration and analytics platform that is middleware that covers a lot of different use cases," explained Steve Wilkes, Founder and CTO at Striim, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
SYS-CON Events announced today that Calligo, an innovative cloud service provider offering mid-sized companies the highest levels of data privacy and security, has been named "Bronze Sponsor" of SYS-CON's 21st International Cloud Expo ®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Calligo offers unparalleled application performance guarantees, commercial flexibility and a personalised support service from its globally located cloud plat...
Everything run by electricity will eventually be connected to the Internet. Get ahead of the Internet of Things revolution and join Akvelon expert and IoT industry leader, Sergey Grebnov, in his session at @ThingsExpo, for an educational dive into the world of managing your home, workplace and all the devices they contain with the power of machine-based AI and intelligent Bot services for a completely streamlined experience.
"DX encompasses the continuing technology revolution, and is addressing society's most important issues throughout the entire $78 trillion 21st-century global economy," said Roger Strukhoff, Conference Chair. "DX World Expo has organized these issues along 10 tracks with more than 150 of the world's top speakers coming to Istanbul to help change the world."
SYS-CON Events announced today that DXWorldExpo has been named “Global Sponsor” of SYS-CON's 21st International Cloud Expo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Digital Transformation is the key issue driving the global enterprise IT business. Digital Transformation is most prominent among Global 2000 enterprises and government institutions.
SYS-CON Events announced today that Datera, that offers a radically new data management architecture, has been named "Exhibitor" of SYS-CON's 21st International Cloud Expo ®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Datera is transforming the traditional datacenter model through modern cloud simplicity. The technology industry is at another major inflection point. The rise of mobile, the Internet of Things, data storage and Big...
While the focus and objectives of IoT initiatives are many and diverse, they all share a few common attributes, and one of those is the network. Commonly, that network includes the Internet, over which there isn't any real control for performance and availability. Or is there? The current state of the art for Big Data analytics, as applied to network telemetry, offers new opportunities for improving and assuring operational integrity. In his session at @ThingsExpo, Jim Frey, Vice President of S...
"We provide IoT solutions. We provide the most compatible solutions for many applications. Our solutions are industry agnostic and also protocol agnostic," explained Richard Han, Head of Sales and Marketing and Engineering at Systena America, in this SYS-CON.tv interview at @ThingsExpo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"We are focused on SAP running in the clouds, to make this super easy because we believe in the tremendous value of those powerful worlds - SAP and the cloud," explained Frank Stienhans, CTO of Ocean9, Inc., in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"We've been engaging with a lot of customers including Panasonic, we've been involved with Cisco and now we're working with the U.S. government - the Department of Homeland Security," explained Peter Jung, Chief Product Officer at Pulzze Systems, in this SYS-CON.tv interview at @ThingsExpo, held June 6-8, 2017, at the Javits Center in New York City, NY.
The financial services market is one of the most data-driven industries in the world, yet it’s bogged down by legacy CPU technologies that simply can’t keep up with the task of querying and visualizing billions of records. In his session at 20th Cloud Expo, Karthik Lalithraj, a Principal Solutions Architect at Kinetica, discussed how the advent of advanced in-database analytics on the GPU makes it possible to run sophisticated data science workloads on the same database that is housing the rich...
"MobiDev is a Ukraine-based software development company. We do mobile development, and we're specialists in that. But we do full stack software development for entrepreneurs, for emerging companies, and for enterprise ventures," explained Alan Winters, U.S. Head of Business Development at MobiDev, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
SYS-CON Events announced today that DXWorldExpo has been named “Global Sponsor” of SYS-CON's 21st International Cloud Expo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Digital Transformation is the key issue driving the global enterprise IT business. Digital Transformation is most prominent among Global 2000 enterprises and government institutions.
In his opening keynote at 20th Cloud Expo, Michael Maximilien, Research Scientist, Architect, and Engineer at IBM, discussed the full potential of the cloud and social data requires artificial intelligence. By mixing Cloud Foundry and the rich set of Watson services, IBM's Bluemix is the best cloud operating system for enterprises today, providing rapid development and deployment of applications that can take advantage of the rich catalog of Watson services to help drive insights from the vast t...
SYS-CON Events announced today that EnterpriseTech has been named “Media Sponsor” of SYS-CON's 21st International Cloud Expo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. EnterpriseTech is a professional resource for news and intelligence covering the migration of high-end technologies into the enterprise and business-IT industry, with a special focus on high-tech solutions in new product development, workload management, increased effic...
SYS-CON Events announced today that Massive Networks, that helps your business operate seamlessly with fast, reliable, and secure internet and network solutions, has been named "Exhibitor" of SYS-CON's 21st International Cloud Expo ®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. As a premier telecommunications provider, Massive Networks is headquartered out of Louisville, Colorado. With years of experience under their belt, their team of...
SYS-CON Events announced today that Cloud Academy named "Bronze Sponsor" of 21st International Cloud Expo which will take place October 31 - November 2, 2017 at the Santa Clara Convention Center in Santa Clara, CA. Cloud Academy is the industry’s most innovative, vendor-neutral cloud technology training platform. Cloud Academy provides continuous learning solutions for individuals and enterprise teams for Amazon Web Services, Microsoft Azure, Google Cloud Platform, and the most popular cloud com...