Scaling and Loadbalancing Wikia Across The World

Wikia hosts around a 100 000 wikis using the open source Mediawiki software. In this talk I'll take a tour through the process of taking a legacy source code and turning it into a globally distributed system. Wikia runs across 6 datacenters in US and Europe, with half of them being CDN nodes and half being full datacenters. Traffic is directed to closest node depending on traffic situation. In a case of degradation the system turns into a read-only mode. The multiple level of redundancy and distribution contributed to a 99.995% availability to end users.

Specific issues involve:

• Varnish - caching and loadbalancing
• Memcache - implementing cache coherency across distributed datacenters
• Session management -- using Riak to transparently failing over
• Mysql replication
• Filesystem
• Monitoring
• Small footprint -- high throughput using SSD based machines
• Mediawiki
• Dealing with loadspikes like Lost Season finale.