Sessions at Strange Loop 2011 about Distributed Systems on Tuesday 20th September

We at Greplin have had an exciting time learning how to scale our service. A year ago, we were storing a few million documents on just two servers – while we’re now storing billions of documents on hundreds of servers (and adding up to 5000 documents a second).

When we started designing our architecture, we gave a lot of thought to distributed data stores (Riak and HBase) and message buses (RabbitMQ and Redis). But we drastically underestimated the importance of the tools and support systems that actually make managing a large and growing cluster manageable, such as:

• Monitoring
• Analytics
• Scripts (deployment/debugging/etc)
• Logging
• Security

Some of these topics aren’t as sexy as the CAP theorem – but they are just as important to running distributed systems in the real world. I’d like to talk about some of the best practices and tools we’ve discovered or created (and open-sourced!) at Greplin to make this possible. I’ll also be sharing some examples of problems that our tools have allowed us to discover and fix quickly.

Erlang and Scala have a very complementary set of strengths and weaknesses for building large scale distributed systems. Cliff will show how to integrate Erlang and Scala using scalang, a library that allows scala actors to transparently participate in Erlang/OTP based distributed systems.

TorqueBox is a Ruby application server built on top of JBoss AS7 that extends the footprint of Ruby applications to include built-in support for services such as messaging, scheduling, and daemons. Infinispan is an extremely scalable, highly available data grid platform, designed to make the most of modern multi-processor/multi-core architectures while at the same time providing distributed cache capabilities. DataMapper is an Object Relational Mapper for Ruby. TorqueBox brings the two together to provide a fast, scalable, distributed NoSQL data store for your Ruby applications. In this presentation, I will take a brief look at each of these technologies and combine them with Sinatra to create a super awesome beer catalogue.