News
15 March 2016 | Dropbox drops Amazon Web Services for in-house system
Dropbox has announced that it will no longer be utilizing Amazon Web Service’s cloud infrastructure, favouring its own in-house solution.
The project, named “Magic Pocket” has been in the works for over two and a half years, and will store and serve over 90% of users’ data on the company’s own custom-built infrastructure. Dropbox was one of Amazon’s first customers to utilize its S3 service to store bulk data eight years ago, but has commented that the relationship will continue in certain areas.
“As the needs of our users and customers kept growing, we decided to invest seriously in building our own in-house storage system,” said Akhil Gupta, Dropbox VP of Engineering. While the company has traditionally stored file content on Amazon, the hosting of metadata and Dropbox web servers has always been in data centres managed by Dropbox itself.
“There were a couple reasons behind this decision. First, one of our key product differentiators is performance. Bringing storage in-house allows us to customize the entire stack end-to-end and improve performance for our particular use case,” said Gupta. “Second, as one of the world’s leading providers of cloud services, our use case for block storage is unique. We can leverage our scale and particular use case to customize both the hardware and software, resulting in better unit economics.”
The company has witnessed healthy growth over recent years, recently passing the milestone of 500 million users and 500 petabytes of user data, prompting the in-house move. Back in 2012, the company only had around 40 petabytes of user data, demonstrating 12-fold growth in the last four years. Dropbox initially began building its own storage infrastructure in 2013, with the company first storing user files in house in February 2015. The team hit its goal of storing 90% of its data in-house on 7 October 2015.
“Magic Pocket became a major initiative in the summer of 2013. We’d built a small prototype as a proof of concept prior to this to get a sense of our workloads and file distributions. Software was a big part of the project, and we iterated on how to build this in production while validating rigorously at every stage,” said Gupta “We knew we’d be building one of only a handful of exabyte-scale storage systems in the world. It was clear to us from the beginning that we’d have to build everything from scratch, since there’s nothing in the open source community that’s proven to work reliably at our scale.”
The move highlights the transition through to private cloud as a business benefit once enterprise reaches a certain level. Zynga is another company who have a similar story, moving between private and public cloud in recent years. Zynga is now in the process of shifting its data back onto in-house infrastructure. Dropbox’s move highlights the potential for overhead reductions when effectively moving onto private cloud, though if the company fails to scale as planned, the move could become a financial burden.
While the move does result in AWS losing a substantial amount of business, it is not the end of the relationship. The team will continue to partner with Amazon for new projects, but will also offer its European customers the opportunity to store data on AWS infrastructure in Germany, should they request it.
Back to all news items
Find us elsewhere: