Runaway complexity in Big Data systems... and a plan to stop it

A session at GOTO Aarhus 2012

Tuesday 2nd October, 2012

1:20pm to 2:10pm (CET)

Big Data has dramatically increased the complexity of building data systems. Big Data forces you to leave the comfortable world of ACID, transactions, and relations, and thrusts you into a challenging world of distributed systems, CAP, and restrictive data models.

You cannot battle complexity with ever more complex systems. This leads to to restrictive systems that are difficult to operate and have poor performance. The only way to reasonably address the complexity of Big Data systems is to fundamentally rethink your approach to avoid that complexity in the first place. A key insight is that the ability to store and process very large amounts of data opens up entirely new ways of building systems that were not possible pre-"Big Data".

NoSQL is not a panacea. Nor is Hadoop, Storm, or any of the other tools out there for Big Data. Yet there is a way to use these tools in conjunction with one another to build complete and robust realtime data systems with a minimum of complexity. These techniques are possible today and can be implemented and operated by small teams.

In this talk you'll learn:

  • How a huge amount of complexity stems from the CRUD paradigm, and why you only need (and want) CR
  • Why embracing immutability is the key to simplifying data systems
  • Where NoSQL fits into the big picture
  • The "Lambda Architecture": a generic approach to building data systems using a combination of batch processing and realtime processing

About the speaker

This person is speaking at this event.
Nathan Marz

Engineer at Twitter

Sign in to add slides, notes or videos to this session

GOTO Aarhus 2012

Denmark Denmark, Arhus

1st3rd October 2012

Tell your friends!


Time 1:20pm2:10pm CET

Date Tue 2nd October 2012

Short URL


View the schedule


See something wrong?

Report an issue with this session