Massively Sharded MySQL at Tumblr

A session at Velocity Europe 2011

Tuesday 8th November, 2011

1:50pm to 2:35pm (CET)

When correctly architected, horizontal partitioning of data can lead to dramatically better fault tolerance and isolation, while significantly improving the manageability and flexibility of a web site. A well-sharded MySQL architecture can perform comparably to newer NoSQL solutions, while still offering the expressiveness of SQL and the time-tested robustness of InnoDB.

The road to complete sharding of Tumblr’s primary data has been a long one, and I’ll discuss how, when, and why we transitioned from a single database, to several functionally-partitioned datasets, and finally to a massively horizontally-sharded architecture.

Along the way, we’ve developed a set of tools that has made the sharding process significantly easier to execute and monitor. The bulk of this session will focus on the practical aspects of sharding, digging into the nitty-gritty of the tools we use to:

  • Transition from a single MySQL database to many without downtime
  • Automate the split of a large shard into several smaller ones extremely efficiently, without having to lock any rows or fail a single query, through clever use of MySQL replication
  • Determine the location of data in the application
  • Update the application’s database configuration across hundreds of web servers, without hitting problems from inconsistent state while the deploy is in progress
  • I’ll also cover some of the mistakes we’ve made, and the tools and tricks we used to gracefully back out of them, such as choosing the wrong shard sizes, and fixing uneven distribution of queries and data.

The session will broadly appeal to engineers at rapidly-growing, MySQL-backed sites. The information presented is all drawn from first-hand experience, and mostly cannot be found in existing books or web sites. Some level of audience familiarity with MySQL is assumed, but previous experience with sharding, InnoDB internals, or MySQL replication is not required.

About the speaker

This person is speaking at this event.
Evan Elias


Sign in to add slides, notes or videos to this session

Velocity Europe 2011

Germany Germany, Berlin

8th9th November 2011

Tell your friends!


Time 1:50pm2:35pm CET

Date Tue 8th November 2011

Short URL


View the schedule



See something wrong?

Report an issue with this session