Logs as Data

A session at Clojure/conj 2011

Friday 11th November, 2011

1:50pm to 2:30pm (EST)

A unifying principle of Clojure's design is expressing programs in terms of a few universal abstractions such as functions, maps, and sequences. Such unification allows us to apply Clojure's powerful library to diverse problem domains and to compose programs that cross these domains. We?ve previously seen this principle applied to HTTP servers and runtime exceptions. In this talk we examine the traditionally-overlooked area of logging in light of the unifying abstractions of functions, maps, and sequences. Treating logs as a sequence of maps and log analysis as the application of functions over map sequences allows logs to become an enormously powerful and general tool for metrics, analytics, alerting, and debugging in production systems. In additional to a general discussion of logs-as-data, we examine the paradigm in detail for Clojure applications on both the producing and consuming sides. On the producing side, we show how a Clojure application can be instrumented to capture log data and to emit it in a way that is orthogonal both to the application's execution environment and to the one or more possible consumers of the log sequence.

On the consuming side, we discuss the Pulse service. Pulse is a distributed log consumer, analyzer, and aggregator, written in Clojure, that processes the log firehouse from the Heroku cloud platform and uses it to power a real-time Heroku operations dashboard.

About the speaker

This person is speaking at this event.
Mark McGranaghan

Engineering @ Heroku bio from Twitter

Coverage of this session

Sign in to add slides, notes or videos to this session

Tell your friends!


Time 1:50pm2:30pm EST

Date Fri 11th November 2011

Short URL


Official event site


View the schedule



See something wrong?

Report an issue with this session