Welcome!

Open Source Cloud Authors: Jyoti Bansal, Liz McMillan, Elizabeth White, Carmen Gonzalez, Radu Gheorghe

Related Topics: Containers Expo Blog, Java IoT, Linux Containers, Open Source Cloud

Containers Expo Blog: Blog Post

GridFS and MongoDB: Pros and Cons

Should I use GridFS for file storage with MongoDB

GridFS looks like a great idea on paper - a virtual filesystem held within MongoDB which allows for larger than 16MB files to be held, synced and replicated. It's very tempting when architecting your solutions to want to consider using GridFS. It appears to be able to take on the problem of storing many thousands or millions of files without consuming file-system resources where there are often hard limits on the number of file names. It also seems to allow for massive files to be stored without any obvious downsides.

It is important, though, to know what GridFS is under the hood. For any file being stored with GridFS, the file is chopped into 255KB chunks. Those chunks are saved in a bucket, called fs, and a collection in that bucket, fs.chunks. Metadata about the files is stored in another collection in the same bucket, fs.files, though you can have more buckets with different bucket names in the same database. An index makes retrieving the chunks quick. All this chunking and metadata management is not done by the MongoDB database though. It is a task performed by the client's driver which is then wrapped in a GridFS API for that driver.

When you put or retrieve a large file of size nKB, in its entirety, what is happening is the driver is retrieving all the relevant chunks, all nKB/255KB chunks, as documents, assembling them at the client end and writing them out to wherever they are needed. So a 16MB file is retrieved as 65 documents of 255K each. Consider what would happen if you did that regularly on your MongoDB database outside of GridFS; there would be severe competition for the servers RAM between those documents and the rest of the database.

The chunking with GridFS and the fact that it is done by the driver also means that large operations like replacing an entire file within GridFS are not atomic and there's no built in versioning to fall back on. This may, or may not, be a problem for some applications where files are concurrently accessible by many users or their applications. You can, though, work around this by layering on your own versioning scheme over GridFS, only making replacement files the latest version when they have completed writing.

There is an upside of to chunking though - it is remarkably cheap to access particular sections of files because they are broken up into manageable blocks during the chunking so if you need to access particular parts of large binary files, you won't be pushing the working set out of memory in the server. You can also adjust the chunk size so if your application would work better with a smaller or larger chunk, you can tune your GridFS usage by requesting a particular size of chunk. Smaller chunks would push fewer parts of the working set out of memory, larger chunks less so.

You can avoid the entire issue of contention with your working set of data by having another MongoDB server dedicated to GridFS storage and optimized towards your file storage use patterns. This also lets you focus on tuning the best performance out of your core database instance without having to look over your shoulder for the march of the GridFS files through your working set. With MongoHQ, creating a separate database instance is easier than ever and there'll be a plan to suit your needs.

If you are dealing with files less than 16MB and want to handle them as atomic entities, it is also worth considering whether you need to use GridFS at all because you can have MongoDB documents with 16MB fields. You will have to ensure that when reading the database, you only pull the large fields into memory when you need to, but this does give you the atomic replacement writes and architecturally simpler system you may desire. The down side is that access to binary ranges within the file will likely require downloading the file, modifying it and re-writing it, but it is all about balance and matching your file storage and use patterns.

So, as we said at the beginning, "it depends" if GridFS is a good fit for your application's file storage needs. There are some pitfalls which you can avoid at planning time with some estimates of what quantity of file data you want to store and how you want to access it. There are also many benefits to a MongoDB and GridFS solution, especially in terms of replication and synchronization.

More Stories By DJ Walker-Morgan

Content Curator at MongoHQ, Dj has been both a developer and writer since Apples came in ][ flavors and Commodores had Pets.

@ThingsExpo Stories
IoT is at the core or many Digital Transformation initiatives with the goal of re-inventing a company's business model. We all agree that collecting relevant IoT data will result in massive amounts of data needing to be stored. However, with the rapid development of IoT devices and ongoing business model transformation, we are not able to predict the volume and growth of IoT data. And with the lack of IoT history, traditional methods of IT and infrastructure planning based on the past do not app...
"LinearHub provides smart video conferencing, which is the Roundee service, and we archive all the video conferences and we also provide the transcript," stated Sunghyuk Kim, CEO of LinearHub, in this SYS-CON.tv interview at @ThingsExpo, held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA.
With major technology companies and startups seriously embracing IoT strategies, now is the perfect time to attend @ThingsExpo 2016 in New York. Learn what is going on, contribute to the discussions, and ensure that your enterprise is as "IoT-Ready" as it can be! Internet of @ThingsExpo, taking place June 6-8, 2017, at the Javits Center in New York City, New York, is co-located with 20th Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry p...
"There's a growing demand from users for things to be faster. When you think about all the transactions or interactions users will have with your product and everything that is between those transactions and interactions - what drives us at Catchpoint Systems is the idea to measure that and to analyze it," explained Leo Vasiliou, Director of Web Performance Engineering at Catchpoint Systems, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York Ci...
The 20th International Cloud Expo has announced that its Call for Papers is open. Cloud Expo, to be held June 6-8, 2017, at the Javits Center in New York City, brings together Cloud Computing, Big Data, Internet of Things, DevOps, Containers, Microservices and WebRTC to one location. With cloud computing driving a higher percentage of enterprise IT budgets every year, it becomes increasingly important to plant your flag in this fast-expanding business opportunity. Submit your speaking proposal ...
Discover top technologies and tools all under one roof at April 24–28, 2017, at the Westin San Diego in San Diego, CA. Explore the Mobile Dev + Test and IoT Dev + Test Expo and enjoy all of these unique opportunities: The latest solutions, technologies, and tools in mobile or IoT software development and testing. Meet one-on-one with representatives from some of today's most innovative organizations
20th Cloud Expo, taking place June 6-8, 2017, at the Javits Center in New York City, NY, will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud strategy.
SYS-CON Events announced today that Super Micro Computer, Inc., a global leader in Embedded and IoT solutions, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 7-9, 2017, at the Javits Center in New York City, NY. Supermicro (NASDAQ: SMCI), the leading innovator in high-performance, high-efficiency server technology, is a premier provider of advanced server Building Block Solutions® for Data Center, Cloud Computing, Enterprise IT, Hadoop/Big Data, HPC and E...
SYS-CON Events announced today that Linux Academy, the foremost online Linux and cloud training platform and community, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Linux Academy was founded on the belief that providing high-quality, in-depth training should be available at an affordable price. Industry leaders in quality training, provided services, and student certification passes, its goal is to c...
WebRTC is the future of browser-to-browser communications, and continues to make inroads into the traditional, difficult, plug-in web communications world. The 6th WebRTC Summit continues our tradition of delivering the latest and greatest presentations within the world of WebRTC. Topics include voice calling, video chat, P2P file sharing, and use cases that have already leveraged the power and convenience of WebRTC.
"A lot of times people will come to us and have a very diverse set of requirements or very customized need and we'll help them to implement it in a fashion that you can't just buy off of the shelf," explained Nick Rose, CTO of Enzu, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York City, NY.
WebRTC sits at the intersection between VoIP and the Web. As such, it poses some interesting challenges for those developing services on top of it, but also for those who need to test and monitor these services. In his session at WebRTC Summit, Tsahi Levent-Levi, co-founder of testRTC, reviewed the various challenges posed by WebRTC when it comes to testing and monitoring and on ways to overcome them.
Every successful software product evolves from an idea to an enterprise system. Notably, the same way is passed by the product owner's company. In his session at 20th Cloud Expo, Oleg Lola, CEO of MobiDev, will provide a generalized overview of the evolution of a software product, the product owner, the needs that arise at various stages of this process, and the value brought by a software development partner to the product owner as a response to these needs.
WebRTC services have already permeated corporate communications in the form of videoconferencing solutions. However, WebRTC has the potential of going beyond and catalyzing a new class of services providing more than calls with capabilities such as mass-scale real-time media broadcasting, enriched and augmented video, person-to-machine and machine-to-machine communications. In his session at @ThingsExpo, Luis Lopez, CEO of Kurento, introduced the technologies required for implementing these idea...
The WebRTC Summit New York, to be held June 6-8, 2017, at the Javits Center in New York City, NY, announces that its Call for Papers is now open. Topics include all aspects of improving IT delivery by eliminating waste through automated business models leveraging cloud technologies. WebRTC Summit is co-located with 20th International Cloud Expo and @ThingsExpo. WebRTC is the future of browser-to-browser communications, and continues to make inroads into the traditional, difficult, plug-in web co...
The Internet of Things will challenge the status quo of how IT and development organizations operate. Or will it? Certainly the fog layer of IoT requires special insights about data ontology, security and transactional integrity. But the developmental challenges are the same: People, Process and Platform. In his session at @ThingsExpo, Craig Sproule, CEO of Metavine, demonstrated how to move beyond today's coding paradigm and shared the must-have mindsets for removing complexity from the develop...
While not quite mainstream yet, WebRTC is starting to gain ground with Carriers, Enterprises and Independent Software Vendors (ISV’s) alike. WebRTC makes it easy for developers to add audio and video communications into their applications by using Web browsers as their platform. But like any market, every customer engagement has unique requirements, as well as constraints. And of course, one size does not fit all. In her session at WebRTC Summit, Dr. Natasha Tamaskar, Vice President, Head of C...
Who are you? How do you introduce yourself? Do you use a name, or do you greet a friend by the last four digits of his social security number? Assuming you don’t, why are we content to associate our identity with 10 random digits assigned by our phone company? Identity is an issue that affects everyone, but as individuals we don’t spend a lot of time thinking about it. In his session at @ThingsExpo, Ben Klang, Founder & President of Mojo Lingo, discussed the impact of technology on identity. Sho...
DevOps is being widely accepted (if not fully adopted) as essential in enterprise IT. But as Enterprise DevOps gains maturity, expands scope, and increases velocity, the need for data-driven decisions across teams becomes more acute. DevOps teams in any modern business must wrangle the ‘digital exhaust’ from the delivery toolchain, "pervasive" and "cognitive" computing, APIs and services, mobile devices and applications, the Internet of Things, and now even blockchain. In this power panel at @...
With all the incredible momentum behind the Internet of Things (IoT) industry, it is easy to forget that not a single CEO wakes up and wonders if “my IoT is broken.” What they wonder is if they are making the right decisions to do all they can to increase revenue, decrease costs, and improve customer experience – effectively the same challenges they have always had in growing their business. The exciting thing about the IoT industry is now these decisions can be better, faster, and smarter. Now ...