Open Source Cloud Authors: Pat Romanski, Elizabeth White, John Mertic, Yeshim Deniz, Carmen Gonzalez

Related Topics: Containers Expo Blog, Java IoT, Linux Containers, Open Source Cloud

Containers Expo Blog: Blog Post

GridFS and MongoDB: Pros and Cons

Should I use GridFS for file storage with MongoDB

GridFS looks like a great idea on paper - a virtual filesystem held within MongoDB which allows for larger than 16MB files to be held, synced and replicated. It's very tempting when architecting your solutions to want to consider using GridFS. It appears to be able to take on the problem of storing many thousands or millions of files without consuming file-system resources where there are often hard limits on the number of file names. It also seems to allow for massive files to be stored without any obvious downsides.

It is important, though, to know what GridFS is under the hood. For any file being stored with GridFS, the file is chopped into 255KB chunks. Those chunks are saved in a bucket, called fs, and a collection in that bucket, fs.chunks. Metadata about the files is stored in another collection in the same bucket, fs.files, though you can have more buckets with different bucket names in the same database. An index makes retrieving the chunks quick. All this chunking and metadata management is not done by the MongoDB database though. It is a task performed by the client's driver which is then wrapped in a GridFS API for that driver.

When you put or retrieve a large file of size nKB, in its entirety, what is happening is the driver is retrieving all the relevant chunks, all nKB/255KB chunks, as documents, assembling them at the client end and writing them out to wherever they are needed. So a 16MB file is retrieved as 65 documents of 255K each. Consider what would happen if you did that regularly on your MongoDB database outside of GridFS; there would be severe competition for the servers RAM between those documents and the rest of the database.

The chunking with GridFS and the fact that it is done by the driver also means that large operations like replacing an entire file within GridFS are not atomic and there's no built in versioning to fall back on. This may, or may not, be a problem for some applications where files are concurrently accessible by many users or their applications. You can, though, work around this by layering on your own versioning scheme over GridFS, only making replacement files the latest version when they have completed writing.

There is an upside of to chunking though - it is remarkably cheap to access particular sections of files because they are broken up into manageable blocks during the chunking so if you need to access particular parts of large binary files, you won't be pushing the working set out of memory in the server. You can also adjust the chunk size so if your application would work better with a smaller or larger chunk, you can tune your GridFS usage by requesting a particular size of chunk. Smaller chunks would push fewer parts of the working set out of memory, larger chunks less so.

You can avoid the entire issue of contention with your working set of data by having another MongoDB server dedicated to GridFS storage and optimized towards your file storage use patterns. This also lets you focus on tuning the best performance out of your core database instance without having to look over your shoulder for the march of the GridFS files through your working set. With MongoHQ, creating a separate database instance is easier than ever and there'll be a plan to suit your needs.

If you are dealing with files less than 16MB and want to handle them as atomic entities, it is also worth considering whether you need to use GridFS at all because you can have MongoDB documents with 16MB fields. You will have to ensure that when reading the database, you only pull the large fields into memory when you need to, but this does give you the atomic replacement writes and architecturally simpler system you may desire. The down side is that access to binary ranges within the file will likely require downloading the file, modifying it and re-writing it, but it is all about balance and matching your file storage and use patterns.

So, as we said at the beginning, "it depends" if GridFS is a good fit for your application's file storage needs. There are some pitfalls which you can avoid at planning time with some estimates of what quantity of file data you want to store and how you want to access it. There are also many benefits to a MongoDB and GridFS solution, especially in terms of replication and synchronization.

More Stories By DJ Walker-Morgan

Content Curator at MongoHQ, Dj has been both a developer and writer since Apples came in ][ flavors and Commodores had Pets.

@ThingsExpo Stories
Web Real-Time Communication APIs have quickly revolutionized what browsers are capable of. In addition to video and audio streams, we can now bi-directionally send arbitrary data over WebRTC's PeerConnection Data Channels. With the advent of Progressive Web Apps and new hardware APIs such as WebBluetooh and WebUSB, we can finally enable users to stitch together the Internet of Things directly from their browsers while communicating privately and securely in a decentralized way.
Amazon has gradually rolled out parts of its IoT offerings, but these are just the tip of the iceberg. In addition to optimizing their backend AWS offerings, Amazon is laying the ground work to be a major force in IoT - especially in the connected home and office. In his session at @ThingsExpo, Chris Kocher, founder and managing director of Grey Heron, explained how Amazon is extending its reach to become a major force in IoT by building on its dominant cloud IoT platform, its Dash Button strat...
Two weeks ago (November 3-5), I attended the Cloud Expo Silicon Valley as a speaker, where I presented on the security and privacy due diligence requirements for cloud solutions. Cloud security is a topical issue for every CIO, CISO, and technology buyer. Decision-makers are always looking for insights on how to mitigate the security risks of implementing and using cloud solutions. Based on the presentation topics covered at the conference, as well as the general discussions heard between sessi...
For basic one-to-one voice or video calling solutions, WebRTC has proven to be a very powerful technology. Although WebRTC’s core functionality is to provide secure, real-time p2p media streaming, leveraging native platform features and server-side components brings up new communication capabilities for web and native mobile applications, allowing for advanced multi-user use cases such as video broadcasting, conferencing, and media recording.
Fifty billion connected devices and still no winning protocols standards. HTTP, WebSockets, MQTT, and CoAP seem to be leading in the IoT protocol race at the moment but many more protocols are getting introduced on a regular basis. Each protocol has its pros and cons depending on the nature of the communications. Does there really need to be only one protocol to rule them all? Of course not. In his session at @ThingsExpo, Chris Matthieu, co-founder and CTO of Octoblu, walk you through how Oct...
Major trends and emerging technologies – from virtual reality and IoT, to Big Data and algorithms – are helping organizations innovate in the digital era. However, to create real business value, IT must think beyond the ‘what’ of digital transformation to the ‘how’ to harness emerging trends, innovation and disruption. Architecture is the key that underpins and ties all these efforts together. In the digital age, it’s important to invest in architecture, extend the enterprise footprint to the cl...
Almost everyone sees the potential of Internet of Things but how can businesses truly unlock that potential. The key will be in the ability to discover business insight in the midst of an ocean of Big Data generated from billions of embedded devices via Systems of Discover. Businesses will also need to ensure that they can sustain that insight by leveraging the cloud for global reach, scale and elasticity.
A critical component of any IoT project is what to do with all the data being generated. This data needs to be captured, processed, structured, and stored in a way to facilitate different kinds of queries. Traditional data warehouse and analytical systems are mature technologies that can be used to handle certain kinds of queries, but they are not always well suited to many problems, particularly when there is a need for real-time insights.
One of biggest questions about Big Data is “How do we harness all that information for business use quickly and effectively?” Geographic Information Systems (GIS) or spatial technology is about more than making maps, but adding critical context and meaning to data of all types, coming from all different channels – even sensors. In his session at @ThingsExpo, William (Bill) Meehan, director of utility solutions for Esri, will take a closer look at the current state of spatial technology and ar...
Explosive growth in connected devices. Enormous amounts of data for collection and analysis. Critical use of data for split-second decision making and actionable information. All three are factors in making the Internet of Things a reality. Yet, any one factor would have an IT organization pondering its infrastructure strategy. How should your organization enhance its IT framework to enable an Internet of Things implementation? In his session at @ThingsExpo, James Kirkland, Red Hat's Chief Arch...
The IoT industry is now at a crossroads, between the fast-paced innovation of technologies and the pending mass adoption by global enterprises. The complexity of combining rapidly evolving technologies and the need to establish practices for market acceleration pose a strong challenge to global enterprises as well as IoT vendors. In his session at @ThingsExpo, Clark Smith, senior product manager for Numerex, will discuss how Numerex, as an experienced, established IoT provider, has embraced a ...
Everyone knows that truly innovative companies learn as they go along, pushing boundaries in response to market changes and demands. What's more of a mystery is how to balance innovation on a fresh platform built from scratch with the legacy tech stack, product suite and customers that continue to serve as the business' foundation. In his General Session at 19th Cloud Expo, Michael Chambliss, Head of Engineering at ReadyTalk, will discuss why and how ReadyTalk diverted from healthy revenue an...
SYS-CON Events announced today that Hitrons Solutions will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Hitrons Solutions Inc. is distributor in the North American market for unique products and services of small and medium-size businesses, including cloud services and solutions, SEO marketing platforms, and mobile applications.
SYS-CON Media announced today that @WebRTCSummit Blog, the largest WebRTC resource in the world, has been launched. @WebRTCSummit Blog offers top articles, news stories, and blog posts from the world's well-known experts and guarantees better exposure for its authors than any other publication. @WebRTCSummit Blog can be bookmarked ▸ Here @WebRTCSummit conference site can be bookmarked ▸ Here
SYS-CON Events announced today that Super Micro Computer, Inc., a global leader in Embedded and IoT solutions, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 7-9, 2017, at the Javits Center in New York City, NY. Supermicro (NASDAQ: SMCI), the leading innovator in high-performance, high-efficiency server technology, is a premier provider of advanced server Building Block Solutions® for Data Center, Cloud Computing, Enterprise IT, Hadoop/Big Data, HPC and ...
SYS-CON Events announced today that Streamlyzer will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Streamlyzer is a powerful analytics for video streaming service that enables video streaming providers to monitor and analyze QoE (Quality-of-Experience) from end-user devices in real time.
You have great SaaS business app ideas. You want to turn your idea quickly into a functional and engaging proof of concept. You need to be able to modify it to meet customers' needs, and you need to deliver a complete and secure SaaS application. How could you achieve all the above and yet avoid unforeseen IT requirements that add unnecessary cost and complexity? You also want your app to be responsive in any device at any time. In his session at 19th Cloud Expo, Mark Allen, General Manager of...
SYS-CON Events announced today that LeaseWeb USA, a cloud Infrastructure-as-a-Service (IaaS) provider, will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. LeaseWeb is one of the world's largest hosting brands. The company helps customers define, develop and deploy IT infrastructure tailored to their exact business needs, by combining various kinds cloud solutions.
@ThingsExpo has been named the Top 5 Most Influential M2M Brand by Onalytica in the ‘Machine to Machine: Top 100 Influencers and Brands.' Onalytica analyzed the online debate on M2M by looking at over 85,000 tweets to provide the most influential individuals and brands that drive the discussion. According to Onalytica the "analysis showed a very engaged community with a lot of interactive tweets. The M2M discussion seems to be more fragmented and driven by some of the major brands present in the...
The Open Connectivity Foundation (OCF), sponsor of the IoTivity open source project, and AllSeen Alliance, which provides the AllJoyn® open source IoT framework, today announced that the two organizations’ boards have approved a merger under the OCF name and bylaws. This merger will advance interoperability between connected devices from both groups, enabling the full operating potential of IoT and representing a significant step towards a connected ecosystem.