Sunday 15th September, 2013
1:00pm to 1:50pm
In this talk I will discuss how to make use of Luigi, which is a tool written by Spotify to manage a pipeline of long running batch processes. Luigi allows you to write long running batch processes in pure Python and provides built-in support for Hadoop mapreduce jobs and abstractions for HDFS and local file systems.
I'll show how to use Luigi to perform a simple set of chained mapreduce jobs and show how Luigi helps with dependency resolution, workflow management, and visualization.
Developer Advocate @Google for @GoogleCloud covering APAC. Vice Chair for @PyConJ. Sometimes tweets in Japanese. #python #golang #kubernetes #atheist #progrock
Sign in to add slides, notes or videos to this session