Sessions at Surge 2010 about Varnish and Scaling

Thursday 30th September 2010

• Scaling and Loadbalancing Wikia Across The World

  by Artur Bergman

  Wikia hosts around a 100 000 wikis using the open source Mediawiki software. In this talk I'll take a tour through the process of taking a legacy source code and turning it into a globally distributed system. Wikia runs across 6 datacenters in US and Europe, with half of them being CDN nodes and half being full datacenters. Traffic is directed to closest node depending on traffic situation. In a case of degradation the system turns into a read-only mode. The multiple level of redundancy and distribution contributed to a 99.995% availability to end users.

  Specific issues involve:

  • Varnish - caching and loadbalancing
  • Memcache - implementing cache coherency across distributed datacenters
  • Session management -- using Riak to transparently failing over
  • Mysql replication
  • Filesystem
  • Monitoring
  • Small footprint -- high throughput using SSD based machines
  • Mediawiki
  • Dealing with loadspikes like Lost Season finale.
  • HTTP
  • Memcached
  • MySQL
  • Riak
  • Scaling
  • Varnish

  At 2:30pm to 3:30pm, Thursday 30th September

  Coverage video