•  

Luigi, The Friendly Pipeline Plumber

A session at PyCon APAC 2013

Sunday 15th September, 2013

1:00pm to 1:50pm (CJT)

In this talk I will discuss how to make use of Luigi, which is a tool written by Spotify to manage a pipeline of long running batch processes. Luigi allows you to write long running batch processes in pure Python and provides built-in support for Hadoop mapreduce jobs and abstractions for HDFS and local file systems.
I'll show how to use Luigi to perform a simple set of chained mapreduce jobs and show how Luigi helps with dependency resolution, workflow management, and visualization.

About the speaker

This person is speaking at this event.
Ian Lewis

Developer Advocate @Google for @GoogleCloud covering APAC. Vice Chair for @PyConJ. Sometimes tweets in Japanese. #python #golang #kubernetes #atheist #progrock

Coverage of this session

Sign in to add slides, notes or videos to this session

PyCon APAC 2013

Japan Japan, Tokyo

13th16th September 2013

Tell your friends!

When

Time 1:00pm1:50pm CJT

Date Sun 15th September 2013

Short URL

lanyrd.com/sdyypd

View the schedule

Share

See something wrong?

Report an issue with this session