Sessions at EuroPython 2011 about Scaling

Your current filters are…

Tuesday 21st June 2011

  • Building Scalable Web Apps

    by David Cramer

    Building scalable web applications may seem like a distant problem, but the problem is not thinking about it before it's too late. This talk will focus on designing a scalable web application before it needs to be (and avoiding the premature optimization dilemma). It will cover topics such as efficient usage of your database, appropriate caching (and invalidation!), and what the speaker considers as best practices.

    At 12:15pm to 1:15pm, Tuesday 21st June

Wednesday 22nd June 2011

  • Spotify: Horizontal scalability for great success

    by Nick Barkas

    If you run on CPython, it's not possible to get a single-process, multithreaded Python program to use more than one CPU core at a time because of the Global Interpreter Lock (GIL). A common way of dealing with this is to run one instance of a Python program for each core a machine has and spread the load amongst those processes. This forces developers to write simple, stateless programs that naturally scale out to many many servers when needed, while also not having to think about things like locking and thread scheduling.

    I'll discuss some tools and methods Spotify's backend uses for managing multiple identical server processes as well as load balancing with DNS, proxy servers, and using hashing to send repeated requests to the same process. I will also talk about the difficulties that arise when you really need to share data or state between processes, and how they can be dealt with.

    At 11:15am to 12:15pm, Wednesday 22nd June

    Coverage video

  • Tricks and challenges developing a large Django site

    by Simon Willison

    Django makes it easy to build web sites and applications on tight deadlines - but how does it deal with sites that scale up in both traffic and complexity? Using Lanyrd.com as a case study, this talk will explore tricks and techniques for taking advantage of what Django has to offer while coping with the unexpected challenges raised by a highly dynamic social web application.

    Topics covered will include performance monitoring and optimisation, advanced caching tricks, denormalisation with Solr and Redis and finding the right balance between technical elegance and pragmatism.

    At 2:30pm to 3:30pm, Wednesday 22nd June

Thursday 23rd June 2011

  • A Python Takeover

    by Mark Ramm-Christensen

    Two years ago I started working on SourceForge with a couple of python developers. Back then the site was written in PHP and was in some ways buried in technical debt. Everything is different today, we have a complete set of tools for Ticket tracking, Wiki documents, Git, Mercurial, Subversion, and IRC -- all written in Python, and now released as open source software.

    This is the story of how the amazing libraries, tools, and talent in the Python web community has radically changed an established company, and allowed us to reinvent sourceforge.net.

    This talk will go over our tech stack, talk about the libraries we used, the way we solved scalability, internationalization, and performance issues issues. Ultimately we know that without python, we could not have built a flexible plugin based system, and gotten permission from management to open source it.

    At 12:15pm to 1:15pm, Thursday 23rd June

    Coverage video

  • Data Driven Cache Invalidation

    by Magnus Hagander

    In todays world, nobody (should) deploy a web application facing the
    internet without having a proper caching system in place. There are
    many different solutions to choose from, from manual use of memcached
    through framework integrated caching to external caches like Squid or
    Varnish. Most modern frameworks come with integrated functionality for
    at least one of these methods, and often more than one.

    However, they often relies on all traffic going through the same
    framework to work properly - a caching layer in Rails is hard to share
    with one in Django. This talk will show a way to break the design
    rules of these frameworks just a little, and have the database help
    solve this problem.

    This talk will use a small application written in Python using Django
    to illustrate the examples, but the method is language independent.
    Unsurprisingly, the database used is PostgreSQL.

    At 3:30pm to 4:30pm, Thursday 23rd June

    Coverage video

Friday 24th June 2011

  • Leveraging an instant messaging protocol to build a scalable cloud architecture

    by Achiel van der Mandele

    XMPP is an open technology for real-time communication. Although primarily known for its application in the IM service "Jabber" (and, subsequently, gchat/gtalk), it has numerous applications and creates new opportunities for inter-system communication.

    While researching for an alternative to webservices at Mendix, we came across XMPP and realized it solved many of the shortcomings of webservices. It is currently being used to handle all inter-system and inter-process communication in the Mendix cloud hosting environment.

    The talk will give a general overview of what XMPP is, an introduction to the sleekxmpp python library and how the core features of xmpp can help an architecture scale well horizontally.

    Required knowledge: basic understanding of hosting environments.

    At 11:15am to 12:15pm, Friday 24th June

    Coverage video