Sessions at PyCon US 2012 about ZeroMQ and Python

Your current filters are…

Clear

Wednesday 7th March 2012

  • The real-time web with co-routines

    by John Anderson

    Learn how to build fast and interactive web applications using a wsgi compliant web framework and co-routines. Utilizing Redis/ZeroMQ, Socket.IO, and GEvent you will learn how to build a responsive and concurrent web app while maintaining good test coverage.

    We will build a collaborative todo list system that will show you how to utilize Python, Redis/ZeroMQ, Socket.io, and GEvent for real-time communication.

    We will focus on the following topics:

    Using a wsgi framework to build a RESTful interface and hooking up socket.io to it.
    Using socketio and redis/zeromq for subscribing to named channels and communicating in real-time over websockets.
    Gevent
    Testing your real-time application
    Deployment and Monitoring
    A rough outline
    GEvent (30 min)

    Intro to GEvent and co-routines, what it is, how it works, the benefits of using it, and how simple it is to wrap your mind around a co-routine vs the callback methodology of threads.

    ZeroMQ / Redis (30 min)

    Intro to pub/sub, the communication model we'll use for our realtime communication and how to utilize ZeroMQ or Redis to achieve this.

    Socket.IO and Backbone.js (30 min)

    Discussion of Polling, Longpolling, Flash sockets, and web sockets and the benefits of each and how Socket.IO lets us not care. I will discuss how to get socket.io client side working with the server side.

    Testing (15 min)

    We will cover how to architect your co-routine based applications to make them easily testable and how to utilize mock to make things easier. We will cover the gotcha's of testing GEvent based code.

    Deployment (15 min)

    The final portion of the tutorial we discuss deployment and how to manage and monitor your WSGI server that is using GEvent patched libraries. This will cover basics on which webservers to use as well as exception handling.

    We leave 30minutes of leeway to further discussion in a certain area that the students are most interested in as well as for questions on things specific that they feel I may not have covered so far.

    I will provide the non-realtime code base for the application we will be working on and we will start from there and slowly add in support for the realtime communication as we discuss the technologies needed.

    Most introductions to the web with gevent just show a basic chat application which leaves a lot of questions about where to go next once you are building a real application and I hope to cover all of those in this tutorial.

    At 1:20pm to 4:40pm, Wednesday 7th March

    In F2, Santa Clara Convention Center

Friday 9th March 2012

  • Build reliable, traceable, distributed systems with ZeroMQ

    by Jérôme Petazzoni

    We will show how to build simple yet powerful RPC code with ZeroMQ, with very few (if any!) modification to existing code. We will build fan-in and fan-out topologies with ZeroMQ special socket types to implement PUB/SUB patterns and scale up job-processing tasks. Thanks to introspection, the resulting services will be self-documented. Finally, we will show how to implement distributed tracing.

    We will show how to leverage ZeroMQ to build a simple yet powerful RPC for Python code. We will focus on simplicity, the goal being to expose almost any Python module or class to network calls – with very few (if any!) modification to existing code.

    We will then explain the purpose and show some use-cases for ZeroMQ special socket types (PUSH/PULL, PUB/SUB, ROUTER/DEALER) to build fan-in and fan-out topologies, as well as asynchronous processing (to avoid blocking when doing long-running requests). A by-product is the ability to scale up job-processing tasks with a message queue, which can even be made broker-less (you don’t have to deploy heavy machinery if you don’t need it).

    We will also demonstrate how introspection can make development and debugging easier, exposing docstrings, and provideing a few command-line helpers to poke, debug, and experiment directly from the shell.

    At the end of the talk (or in a separate talk), we will explain how to implement a tracing framework for distributed RPC. By hooking into the right places, we will show how to get full tracebacks and profiling information; more precisely:

    how each complex call (involving multiple subcalls) can be accurately traced;
    how to handle exceptions, and know easily when and where they happened (without checking dozens of log files);
    which complex calls take too long, and where they spend their time (distributed profiling).
    Those guidelines are the result of an on-going development work at dotCloud, and actively used and implemented at the core of our leading Platform-as-a-Service offering.

    We don’t expect the audience to be familiar with ZeroMQ or RPC. However, it will certainly help to have basic knowledge of serialization (e.g. pickle) and sockets.

    At 2:00pm to 2:40pm, Friday 9th March

    In E4, Santa Clara Convention Center