Cascading through Hadoop: A DSL for Simpler MapReduce

Hadoop is a MapReduce framework that has literally sprung into the vernacular of "big data" developers everywhere. But coding to the raw Hadoop APIs can be a real chore. Data analysts can express what they want in more English-like vocabularies, but it seems the Hadoop APIs require us to be the translator to a less comprehensible functional and data-centric DSL.

The Cascading framework gives developers a convenient higher level abstraction for querying and scheduling complex jobs on a Hadoop cluster. Programmers can think more holistically about the questions being asked of the data and the flow that such data will take without concern for the minutia.

We'll explore how to set up, code to, and leverage the Cascading API on top of a Hadoop sample or production cluster for a more effective way to code MapReduce applications all while being able to think in a more natural (less than fully MapReduce) way.

During this presentation, we'll also explore Cascading's Clojure-based derivative, Cascalog, and how functional programming paradigms and language syntax are emerging as the next important step in big-data thinking and processing.

Prerequisite: A very basic knowledge of MapReduce and Hadoop