Sessions at EuroPython 2011 about Scaling

Tuesday 21st June 2011

• Building Scalable Web Apps

  by David Cramer

  Building scalable web applications may seem like a distant problem, but the problem is not thinking about it before it's too late. This talk will focus on designing a scalable web application before it needs to be (and avoiding the premature optimization dilemma). It will cover topics such as efficient usage of your database, appropriate caching (and invalidation!), and what the speaker considers as best practices.

  • Disqus
  • Django
  • Scaling

  At 12:15pm to 1:15pm, Tuesday 21st June

  Coverage slide deck handout video

Wednesday 22nd June 2011

• Spotify: Horizontal scalability for great success

  by Nick Barkas

  If you run on CPython, it's not possible to get a single-process, multithreaded Python program to use more than one CPU core at a time because of the Global Interpreter Lock (GIL). A common way of dealing with this is to run one instance of a Python program for each core a machine has and spread the load amongst those processes. This forces developers to write simple, stateless programs that naturally scale out to many many servers when needed, while also not having to think about things like locking and thread scheduling.

  I'll discuss some tools and methods Spotify's backend uses for managing multiple identical server processes as well as load balancing with DNS, proxy servers, and using hashing to send repeated requests to the same process. I will also talk about the difficulties that arise when you really need to share data or state between processes, and how they can be dealt with.

  • Python
  • Scaling
  • Spotify

  At 11:15am to 12:15pm, Wednesday 22nd June

  Coverage video

• Tricks and challenges developing a large Django site

  by Simon Willison

  Django makes it easy to build web sites and applications on tight deadlines - but how does it deal with sites that scale up in both traffic and complexity? Using Lanyrd.com as a case study, this talk will explore tricks and techniques for taking advantage of what Django has to offer while coping with the unexpected challenges raised by a highly dynamic social web application.

  Topics covered will include performance monitoring and optimisation, advanced caching tricks, denormalisation with Solr and Redis and finding the right balance between technical elegance and pragmatism.

  • Django
  • Lanyrd
  • Redis
  • Scaling
  • Solr

  At 2:30pm to 3:30pm, Wednesday 22nd June

  Coverage video video slide deck

Thursday 23rd June 2011

• A Python Takeover

  by Mark Ramm-Christensen

  Two years ago I started working on SourceForge with a couple of python developers. Back then the site was written in PHP and was in some ways buried in technical debt. Everything is different today, we have a complete set of tools for Ticket tracking, Wiki documents, Git, Mercurial, Subversion, and IRC -- all written in Python, and now released as open source software.

  This is the story of how the amazing libraries, tools, and talent in the Python web community has radically changed an established company, and allowed us to reinvent sourceforge.net.

  This talk will go over our tech stack, talk about the libraries we used, the way we solved scalability, internationalization, and performance issues issues. Ultimately we know that without python, we could not have built a flexible plugin based system, and gotten permission from management to open source it.

  • Python
  • Scaling

  At 12:15pm to 1:15pm, Thursday 23rd June

  Coverage video

• Data Driven Cache Invalidation

  by Magnus Hagander

  In todays world, nobody (should) deploy a web application facing the
  internet without having a proper caching system in place. There are
  many different solutions to choose from, from manual use of memcached
  through framework integrated caching to external caches like Squid or
  Varnish. Most modern frameworks come with integrated functionality for
  at least one of these methods, and often more than one.

  However, they often relies on all traffic going through the same
  framework to work properly - a caching layer in Rails is hard to share
  with one in Django. This talk will show a way to break the design
  rules of these frameworks just a little, and have the database help
  solve this problem.

  This talk will use a small application written in Python using Django
  to illustrate the examples, but the method is language independent.
  Unsurprisingly, the database used is PostgreSQL.

  • Caching
  • Django
  • Memcached
  • PostgreSQL
  • Python
  • Scaling

  At 3:30pm to 4:30pm, Thursday 23rd June

  Coverage video

Friday 24th June 2011

• Leveraging an instant messaging protocol to build a scalable cloud architecture

  by Achiel van der Mandele

  XMPP is an open technology for real-time communication. Although primarily known for its application in the IM service "Jabber" (and, subsequently, gchat/gtalk), it has numerous applications and creates new opportunities for inter-system communication.

  While researching for an alternative to webservices at Mendix, we came across XMPP and realized it solved many of the shortcomings of webservices. It is currently being used to handle all inter-system and inter-process communication in the Mendix cloud hosting environment.

  The talk will give a general overview of what XMPP is, an introduction to the sleekxmpp python library and how the core features of xmpp can help an architecture scale well horizontally.

  Required knowledge: basic understanding of hosting environments.

  • Scaling
  • XMPP

  At 11:15am to 12:15pm, Friday 24th June

  Coverage video