Wal-E is an open source tool designed to continuously archive your business's Postgres data to Amazon's S3 cloud storage service. It can help with recovery from failures even when the individual database servers are not individually very reliable, which is the case in many common cloud database deployments.
Individual systems in any environment can't be relied on to save critical data perfectly. No matter how much reundancy your database server includes, you still need to save backups elsewhere. The fact that no single machine can ever be reliable enough is particularly obvious when deploying servers onto a cloud deployment such as Amazon's EC2 service. Individual nodes are never expected to be as reliable as a traditional dedicated database server.
The architecture at cloud service provider Heroku solves this problem by reversing the normal approach to mission critical database deployment: they make the backups the most reliable part of the system. PostgreSQL's write-ahead log (WAL) mechanism can be used to make streaming, real-time backups of your database. The WAL-E tool developed there archives this data into Amazon's S3 storage, which so far boasts a perfect track record for storing all data ever committed to it. This approach is currently in use protecting over 100,000 Postgres databases, and has allowed Heroku to survive major interruptions in Amazon's cloud infrastructure with minimal data loss. In this session, learn how and why WAL-E was implemented, and how you can use it to keep your own data safe and secure.
by Adam Lowry
Adventures in data stores at a growing startup.
by Gavin M. Roy
There are many strategies in scaling PostgreSQL infrastructure. Using Skytools, a set of technologies developed by Skype, Postgres can scale easily both horizontally and vertically. Learn in a hands on tutorial on how pgBouncer can be a significant boost to PostgreSQL performance and how to horizontally scale your PostgreSQL data with plProxy.
Skytools includes a variety of tools useful for scaling large PostgreSQL sites. The pgBouncer connection pooler allows incoming database load to be carefully controlled, as well as adding features for easier management of 24x7 availability. And the Skytools pl/Proxy language allows horizontal partitioning of data by ID across multiple servers, allowing a sharding style approach to scaling PostgreSQL. Learn how these tools have been used to scale upwards a PostgreSQL backed web site supporting billions of page views per month.
14th–16th September 2011