Storm: Twitter's scalable realtime computation system

A session at Strange Loop 2011

Monday 19th September, 2011

10:30am to 11:20am (CST)

Storm makes it easy to write and scale complex realtime computations on a cluster of computers, doing for realtime processing what Hadoop did for batch processing. Storm guarantees that every message will be processed. And it’s fast — you can process millions of messages per second with a small cluster. Best of all, you can write Storm topologies using any programming language.

Storm has a wide range of use cases. The basic use case is “stream processing”: processing a stream of new data and updating databases in realtime. Unlike the standard approach of doing stream processing with queues and workers, Storm is fault-tolerant and scalable.

Another use case is “continuous computation”: streaming the results of a query to clients to visualize in realtime. An example is streaming trending topics on Twitter into browsers.

A third use case is “distributed RPC”: computing an intense query on the fly in parallel. With distributed RPC, a Storm topology is a distributed function that you can invoke like a normal function.

In this talk, I’ll release Storm as open-source. I’ll show how Storm’s simple programming model makes realtime computation easy, robust, and even fun.

About the speaker

This person is speaking at this event.
Nathan Marz

Twitter engineer. Author of Storm and Cascalog. Writing the upcoming book Big Data http://manning.com/marz/ bio from Twitter

Next session in Grand Ballroom

1pm A Tale of Three Trees by Scott Chacon

Coverage of this session

Sign in to add slides, notes or videos to this session

Tell your friends!

When

Time 10:30am11:20am CST

Date Mon 19th September 2011

Where

Grand Ballroom, Hilton St. Louis at the Ballpark

Short URL

lanyrd.com/sfypx

View the schedule

Share

Topics

See something wrong?

Report an issue with this session