The Etsy Shard Architecture: Starts with S and Ends With Hard

A session at Percona Live: MySQL Conference And Expo 2012

Wednesday 11th April, 2012

2:00pm to 2:50pm (PST)

In the beginning, all websites start from a single data store, but the abundance of open source solutions today make it apparent that read and write bottlenecks are quickly reached and some form of scaling is eventually necessary. The two extreme choices are to vertically scale and hope that hardware outpaces your capacity requirements, or to horizontally scale by sharding the data so that additional capacity is simply a matter of adding shards.

At Etsy, we made a decision to horizontally scale our data by sharding it across MySQL multi-master replicated shard pairs. This gives us the benefit of infinite horizontal scalability, redundancy within each shard, excellent fault tolerance, and improved performance.

Scaling is not without its caveats, and some specific decisions must be made about the overall architecture. For example choosing the shard based on a range, a key hash, or a lookup table can be one of the most important decisions about how the shards are initially set up. Other decisions such as how to generate primary keys also come into play very early on.

Etsy uses a lookup based approach where index servers are queried initially for the shard the data resides on. This gives us the flexibility to easily move data around based on capacity requirements. This talk will cover everything from how the multi-master pairs are setup and queried, how we find the data using index servers, and how we generate keys using ticket servers. It will also cover some of the more non-trivial aspects such as schema changes and redistributing data using custom shard migration tools.

Horizontal scaling using a time tested database such as MySQL is a great way to avoid the bottlenecks of a single data store. With a carefully thought out architecture and the proper tooling, it will grow with your traffic and allow you to combat performance and storage limitations by simply adding hardware. This talk will show you exactly how Etsy does this using MySQL.

About the speaker

This person is speaking at this event.
John Goulah

working on infrastructure tools @ Etsy, enjoy food and travel bio from Twitter

Coverage of this session

Sign in to add slides, notes or videos to this session

Tell your friends!


Time 2:00pm2:50pm PST

Date Wed 11th April 2012

Short URL


View the schedule



See something wrong?

Report an issue with this session