The Evolution of Hadoop at Spotify - Through Failures and Pain

A session at GOTO Copenhagen 2015

Tuesday 6th October, 2015

5:00pm to 5:50pm (CET)

The quickest way to learn and evolve infrastructure is by encountering obstacles and being forced to overcome limitations that keep you inches away from project goals. At Spotify, we've encountered many of these obstacles and frustrations as we grew our Hadoop cluster from a few machines in an office closet aggregating played song events for financial reports, to our current 900 node cluster that plays a large role in many features that you see in our application today.

Two members of Spotify's Hadoop 'squad' will weave in war stories, failures, frustrations and lessons learned to describe the Hadoop/Big Data architecture at Spotify and talk about how that architecture has evolved.

We'll talk about how and why we use a number of tools, including Apache Falcon and Apache Bigtop to test changes; Apache Crunch, Scalding and Hive w/ Tez to build features and provide analytics; and Snakebite and Luigi, two in-house tools created to overcome common frustrations.

About the speaker

This person is speaking at this event.
Josh Baer

Hadoop Engineer/Product Owner at Spotify

Sign in to add slides, notes or videos to this session

Tell your friends!

View the schedule


See something wrong?

Report an issue with this session