The Story of Sharding at Box

A session at Velocity Europe 2012

Thursday 4th October, 2012

4:15pm to 4:55pm (GMT)

The database is typically the most difficult web application component to scale horizontally. This is especially true if the application was originally built with a single relational database in mind. In this talk we will present a step-by-step description of how Box’s web application stack was transitioned from one monolithic MySQL database to a fully sharded MySQL architecture. The emphasis will be on the incremental steps and best practices that enabled the successful execution of this fundamental change, all the while continuously serving 2 billion queries per day. The techniques presented will be relevant to web application architects, DevOps and DBAs trying to make the initial transition to a sharded solution.

We will begin with an overview of our web application architecture both before and after sharding. We will then walk through the modifications we made to our ORM layer to support this new architecture, including support for cross-shard queries and online moving of data between shards. We will continue with a detailed description of the technique we developed for migrating live data to shards without downtime, which also supports table by table migration for added flexibility. We will then describe how we use a similar migration technique to move and split shards for rebalancing purposes. Finally, we will discuss some of the challenges of maintaining a sharded databases architecture, and present some tools and techniques we have found to be useful. Throughout the talk we will focus on how to make large-scale changes in an incremental fashion, without adversely affecting functionality, and most importantly without downtime.

About the speaker

This person is speaking at this event.
Tamar Bercovici

Sign in to add slides, notes or videos to this session

Tell your friends!


Time 4:15pm4:55pm GMT

Date Thu 4th October 2012

Short URL


View the schedule


See something wrong?

Report an issue with this session