The Little Warehouse That Couldn't or: How we learned to stop worrying and move to Spark

A session at Spark Summit 2015

Monday 15th June, 2015

5:45pm to 6:00pm

Shopify's commerce platform now powers over 150K merchants and continues to grow. The huge volume and variety of our data pushed our homegrown reporting and warehousing system to the edge. As the maintenace and performance costs became too much we moved to HDFS and Spark, using both python and scala to transform our vast amounts of operational data into dimensional models to provide better insight. Find out the lessons we've learned moving our entire organization onto fully conformed facts and dimensions, the clusters we've cratered, the walls we've hit, and what we did to overcome them to build our Starscream framework.

About the speaker

This person is speaking at this event.
Yandu Oppacher

Developer at Shopify bio from LinkedIn

Sign in to add slides, notes or videos to this session

Tell your friends!


Time 5:45pm6:00pm PST

Date Mon 15th June 2015

Short URL


Official event site


View the schedule


See something wrong?

Report an issue with this session