Sessions at Cassandra SF 2011 about Cassandra on Monday 11th July

Twitter currently runs a couple hundred Cassandra nodes across a half dozen clusters. These span a variety of workloads– from time series to data, to low latency, high throughput key/value. Each workload has led the team to new techniques for operating Cassandra at scale. Chris Goffinet, an engineer at Twitter and Cassandra committer, will be sharing some of the most interesting ones. For those of you interested in Cassandra and operations, this is a must-attend talk.

Cassandra provides a wide set of mechanisms for indexing and searching data. Cassandra’s built-in secondary indexes make it easy to get started, but it also provides unique and powerful capabilities for building custom indexes that can be used for sophisticated queries of object and document data. Cassandra 0.8 introduces Composite columns, which are a key building block of custom indexes. Find out which indexing techniques are best suited to your application, and see what’s possible using some of the advanced indexing techniques.

For years SQL has provided a stable and (nearly) compatible interface to relational databases, regardless of platform or development environment. Contrast this to the NoSQL ecosystem where each project has implemented it’s own query interface, with specialized tool chain, and a unique set of idiosyncrasies.

With Cassandra 0.8, we have the release of the Cassandra Query Language (CQL). CQL functions similarly to SQL and contains most of the SQL core keywords – CREATE, DROP, INSERT, UPDATE, SELECT, USE are all there and function as one would expect.

Cassandra is mostly known for its I/O scalability, but its shared-nothing, highly available foundation is equally useful for applications requiring multi-datacenter distribution and constant uptime. It’s actually quite easy to build and manage HA services by punting the real HA problems to Cassandra’s battle-tested replication and sharding implementation. In this session, we’ll explore Pantheon’s edge routing layer, including:

• How application servers register their presence in Cassandra with a Twisted Python-based REST API. We’ll test updating routes using using cURL.

• Using Twisted Python to expose the Cassandra route mappings via DNS. This stack will provide us with a distributed, highly available UDP DNS cluster in about 30 lines of Python code back-ending to Cassandra. We’ll then read routes using the “dig” shell client.

• Using node.js as an HTTP reverse proxy to look up, cache (in local Redis), and use (with load balancing) routes discovered by reading the “A” records found in DNS. This part tops off the stack and allows us to distribute HTTP requests among application servers registered in Cassandra.

Since this is a case study, we’ll also consider how Pantheon manages the DNS/Cassandra layer to maintain full availability (read and write) during deployments and data center interlink failures.

In our initial move of the Netflix streaming service to the cloud, we made copies of data and synchronized changes back to Oracle in the datacenter. This year Netflix is making the cloud the master copy, and phasing out Oracle as a data store. At the same time we are moving from our initial memcached / SimpleDB based back-end to Apache Cassandra with data replicated to AWS availability zones, and supporting a global business with asynchronous cross region replication. We have implemented full and incremental backup and restore to S3, and integration with our existing Business Intelligence back-end. We already use Hadoop, and are currently investigating Brisk. Netflix chose Apache Cassandra because it’s flexible operational model fits our need for highly available and globally distributed data sources. We have also engaged in the Apache development process, contributing fixes and new features such as off-heap row cache and incremental backup hooks, and find that it integrates well with our primarily Java based development environment.

5m lighting talk on things *not* to do with Cassandra