by Antony Falco
Since Basho Technologies was founded back in mid-2007, we have been the recipients of a fair amount of (often unsolicited) advice regarding our business. No single facet of our business has prompted more commentary than the early decision to build our products using predominantly Erlang. From hiring to funding to adoption, otherwise smart people have predicted that Erlang would prove to be an Achilles heel for Basho. Not a single person outside the Erlang community -- with one notable exception -- saw Erlang as a strategic advantage.
In this talk, I will objectively consider the challenges and advantages we have faced over the past four years. I will share lessons learned seeking funding, recruiting, bui lding a Riak-oriented community, and driving adoption inside some of the world's largest enterprises. I will also talk about overcoming challenges our advocates have faced getting Erlang-based initiatives adopted in the enterprise.
Finally, I will discuss the role the Erlang community has played in our success and what I think the future can be for commercial and open-source projects that happen to be built on "that Swedish boutique language."
Given how popular writing your own Erlang web server seems to be these days, perhaps legendary Erlang programmer Claes "Klacke" Wikström was ahead of his time when nearly a decade ago he chose to call his then-new project "Yet Another Web Server." Famous for the "Apache vs. Yaws" graphs that boasted its scalability, Yaws has managed to maintain a relatively simple core while acquiring a rich set of features over the years. These features include the support of Erlang code embedded within HTML pages, mapping URI paths to specific application modules for request processing, streamed responses, web sockets, virtual servers, URL rewriting, integration with existing Erlang applications, sendfile support, and embedding within larger Erlang applications. When combined with the advanced concurrency, distribution, and reliability features of its underlying Erlang/OTP platform, Yaws provides a rich environment for production-quality web applications.
In this talk, Steve, a committer on the Yaws project, will provide an overview of a number of Yaws features and how applications use them, and will also explain some of its internal architecture.
Let's take the following for-granted: Doing security, Privacy and Scalability last is a bad idea.
But you do not engineer a system that is supposed to have at the most a thousand users the same way you engineer a system that might have millions of them. That said, when you are constructing a Framework the assumptions you make on the use cases and number of users are at best a wild guess. Developers are like that, crazy. Even if you get a lot of really smart people in a room there is a very good chance you'll get it wrong. Enough to look at the IPV4 address space... but if IPV6 addresses were implemented at the beginning of the Internet the network overhead might have been unacceptable. Maybe the whole thing would not have worked. So one of the major issues when designing a new system that does not have a very strict, limited and stable use case, is writing only the code of the scalability you need while deferring that of future needs while having a very strong notion of why there are no strict impediments for it to scale later.
We will look at some of the design decisions at the heart of U.C. Engine our Realtime Collaboration Framework, and see why choosing a documented oriented store as a persistence layer is a good idea when there are some assumptions that are too early to be made. Why and how it allows us to defer some of the scalability issues. We'll look specifically at MongoDB that we chose it side by side with Mnesia. We shall examine the issue of impedance mismatch between records and documents. And to get really down to the nitty gritty stuff we will also take a hard look at the current state of MongoDB libraries in Erlang. Why we chose emongo, what we learnt from this about the state of libraries in Erlang and how this effects doing a larger web oriented open source project.
by Kostis Sagonas
PropEr is an open-source, Quickcheck-inspired tool for property-based testing in Erlang. Its salient feature, which is also the primary reason for PropEr's existence, is that it offers a proper integration of the language of types and specs of Erlang with properties. Any type can be used as a generator (PropEr knows how to create a shrinker for it) and any function spec can be directly used as simple property of a function.
In addition, PropEr offers support that significantly simplifies the task of writing generators for recursive and opaque data types. Last but not least, PropEr includes efficient support for testing stateful applications.
The talk will overview the features of PropEr and will include a PropEr demo.
Everyone wants to talk about "conversational web services" nowadays, but the mental model of a conversation tends to be constrained by what our current web services frameworks are capable of delivering.
Erlang was designed for controlling real-time services of "conversational quality", meaning that the service should allow people to meet and converse with a "real-life" feeling, without disruptive delays and with a life-like presentation of information. Web frameworks in Erlang have largely copied the traditional MVC frameworks, but for the back-end logic, they have drawn on the full power of Erlang's distributed, real- time processing environment. In effect, Erlang frameworks have been constrained by the long-polling RESTful nature of today's web services, but can now start breaking out of that mold, with the advent of web sockets and other real-time delivery technologies for the web. This talk will illustrate how Erlang provides all the tools necessary for building the next generation of truly conversational web services.
At Extreme Forge, we believe that the most efficient and enjoyable way to successful projects is to focus on having clean code that works. In the wake of test-driven development, software craftsmanship and coding dojos, we are working on an contest engine that allows people from around the world to gather online and compete for fun and learning on an interesting coding problem, in real-time.
A beta session of what we called the Agile Cup was held in April, where the problem was to find the way out of a maze of increasing complexity. In addition to solving mazes, the competitor's program must communicate with the engine through an HTTP/JSON interface.
I will code, live, an Erlang solution to this problem, in the hope of showing some clean Erlang code that works to the audience, and in order to provide a live, commented demonstration of the test-driven development approach to programming.
by Jonas Bonér
We believe that one should never have to choose between productivity and scalability, which has been the case with traditional approaches to concurrency and distribution. The cause of that has been the wrong tools and the wrong layer of abstraction and Akka is here to change that. Akka is using the Actors together with Software Transactional Memory (STM) to create a unified runtime and programming model for scaling both UP (utilizing multi-core processors) and OUT (utilizing the grid/cloud). Akka provides location and network transparency by abstracting away both these tangents of scalability by turning them into an operations and configuration task. This gives the Akka runtime freedom to do adaptive automatic load-balancing, cluster rebalancing, replication and partitioning. In this talk you will learn what Akka is and how it can be used to solve hard scalability problems.
Akka is available at http://akka.io (under Apache 2 license).
by Pero Subasic
We present a framework in Erlang for distributed event processing and data analytics in the domain of computational advertising and share practical use cases and performance test results on ad serving events on one of the largest online advertising networks. We also discuss related technologies such as Hadoop, NoSQL, MPI/CUDA and Erlang's place in such large-scale data analysis ecosystem.
The consistency model used by Riak is designed to continue operating well even when nodes in your Riak cluster are down or unreachable, a property which also makes it a good model for mobile data. To me, "down or unreachable" sounds a lot like "off or without-cell-phone-connectivity" - common states for a mobile device.
In this talk, I'll present Riak Mobile, a component that can be embedded into your iOS or Android application to make it a full Riak replication-aware peer. Typical usages for Riak Mobile is as a mobile content distribution platform using one-way sync; or with two-way sync to also push updated data back to your Riak cluster when the network is "eventually" available.
In the talk, I'll walk through how Riak Mobile works. The client-part of Riak Mobile does not require an Erlang VM. Rather, it comes as either a Java or an Objective-C component using SQLite for on-device storage, and thus integrates nicely into the native development environments. The server-part is an OTP application running with Riak. Riak Mobile uses a flow-based algorithm using Merkle trees and vector clocks optimized specifically for providing efficient incremental synchronization for slow, high-latency network conditions.
Couchbeam is an OpenSource project that provide a simple CouchDB framework in Erlang.
This talk will explore how to use couchbeam in your application from building a web application to use it to rewrite Couchapp a command line tool to manage couchapps (embedded CouchDB applications) from Python to Erlang. Everything you should know about Couchbeam and more generally how to use the possibilities provided by Erlang to provide generic ways to use CouchDB features in your application.
by James Aimonetti
Whistle is a powerful, open-source platform designed to give developers the most flexibility in creating VoIP-enabled applications and services.
While APIs exist for interacting with Whistle via REST and AMQP in your language of choice, we will show you how to create your own Erlang-based whApps inside the Whistle application container. The container provides functionality for easily interacting with CouchDB, RabbitMQ, mochijson2-decoded JSON strings and more, all designed to be easy to use without you having to delve into the details.
We will show how, once you have the idea for your whApp in mind, bringing it to life as a prototype is straightforward and fast. In this session we will show the steps for building a simple auto-attendant module to answer a call, play a menu, record digits pressed and transfer the call..
It comes without saying that web is not static, in many ways. It evolves at a rapid pace and poses new challenges to the web developers community regularly. So how do we deal with this? By relentless experimenting and constantly being at the bleeding edge.
This talk will cover bleeding edge techniques for building web apps in Erlang. "Thick" clients, Socket.IO magic, session-oriented development, redundant/fault-tolerant session management & other new developments in the Erlang ecosphere.
by Darach Ennis
Complex Event Processing is a technology that enables the observation, orientation of, decision against and issuance of actions in low latency near real time. In a nutshell CEP (or Event Stream Processing, ESP) engines invert the traditional relationship between processing engines and data stores. Rather than persisting data on disk and querying in an ad hoc fashion. We take the query, optimize the hell out of it, compile it ultimately to native code and flood it with a high frequency of real time events. StreamBase is a commercial compiler oriented flow oriented (neither functional nor object oriented) CEP engine with a graphical development experience. Like Erjang & Erlang, it's event driven, tuple oriented, supports behaviour driven development.
This talk will introduce CEP, demonstrate integrations with Erjang and Erlang and look at how two tuple oriented technologies and their ecosystems can mutually benefit for the greater good of the developers and organizations that leverage them in mission critical environments.
At wooga we build backends for games that have millions of daily users.
In the gaming business we have a write heavy environment, with a high frequency of requests, traffic bursts and distribution across many nodes. These are all problems we need to solve and keep in mind in order to write a system that stands the chance of supporting the required load.
How do you meet the challenge of writing a brand new system with performance in mind? Where should the line between necessary efficiency and premature optimization be drawn? How do you measure performances? How do we generate synthetic load that reflects real usage patterns? How do you know you have enough capacity? How do you combine all the above with safely introducing changes and new features working in a two people team?
We had to answer to all the above questions and we want to share the solutions we found and the problems that we consider still open.
by Huiqing Li and Simon Thompson
Wrangler is an Erlang refactoring tool which provides a range of refactorings - including renaming, function extraction and generalisation - as well as facilities for clone detection and removal as well as improvement of the module structure of projects. Wrangler is integrated into emacs and Eclipse (via ErlIDE).
Up to now, you could only use the refactorings built into Wrangler, or dive deep into the internals to extend it. The latest version of Wrangler now provides a much easier way of defining and applying refactorings for yourself. All you need to do is implement them as a refactoring behaviour and they are accessible in emacs, and you can write descriptions of what the refactorings do using a combination of Erlang macros and templates that describe the particular changes to be made. You can also use the same facilities write new forms of code inspection equally easily.
After describing the DIY facilities and giving an overview demo of Wrangler we'll show how to use this DIY refactoring API in practice.
Having spent time working in a number of functional languages in addition to Erlang such as Haskell, Scheme, Lisp, OCaml and others, there are many nice features of those languages that would increase expressivity of Erlang. In my case, having spent a couple of years working in Haskell before returning to the RabbitMQ fold, all sorts of features are “missing”, such as laziness, type classes, additional infix operators, the ability to specify precedence of functions, fewer parenthesis, partial application, more consistent standard libraries and do-notation. That’s a fair list, and it’ll take me a while to get around to implementing them all in Erlang, but here are two for starters.
'Cut' is a lightweight syntax for abstracting over expressions with holes in them. Similar to partial application, it permits implicit lambda abstraction without the line-noise of `funs'.
'Do' introduces Haskell's do-notation to Erlang together with an extensible implementation of monads.
Together, these extensions can be used to write code that is clearer of intent, more legible and maintainable than would otherwise be the case.
by Christian Colombo
Ascertaining program correctness through testing is limited because it is inherently hard to cover all possible execution scenarios through finite test runs. An alternative, albeit complementing technique to testing is that of Runtime Verification, whereby a monitor program runs concurrently with the main program and performs runtime checks with respect to the main program's current execution.
Erlang is particularly suited to such a technique. Due, in part, to its lack of static checks, the language has traditionally advocated for programs that incorporate precautionary runtime checks; these defensive checks make programs resilient to unforseen circumstances not covered by the normal program logic. Moreover, Erlang's native support for concurrent processes, together with the recent advent of multicore architectures, should in theory enable the deployment of monitor programs running concurrently with the main program at a negligible runtime cost.
In this talk we advocate for a Runtime Verification approach to Erlang program development where runtime checks are teased apart from the actualy program logic, and delegated to a separate monitor program. In some sense, this is an extension of the existing link/monitor mechanism in Erlang; at present this only comes into force once a monitored process dies; through native mechanisms such as tracing and messaging, this can however be made aware of the intermediate execution of programs so as to be able to check whether this execution adheres to some specified property. This approach carries with it a number of advantages:
i) it yeilds better separation of concerns, where the main code is not cluttered with runtime checks obscuring the main program logic;
ii) it allows for the monitored properties to be specified using independent property-based logic tools. Properties written in this logic can then be synthesised automatically into an Erlang monitor process and deployed.
iii) it allows for a more modular approach to property monitoring, whereby properties monitored can be changed without affecting the main code of the program.
We evince this approach by adapting Larva, an existing tool for monitoring Java programs, to monitor Erlang program executions with respect to automata-based properties. We discuss our achievements so far and outline exciting directions for future work.
by Malcom Dowse
Demonware is one of the world's largest online game service providers. We handle over 2.5 million concurrent online users, and host the online component for over 40 titles including Call of Duty Black Ops, the world's best selling game. Games studios come up with feature requests, and we develop the server code, figure our how to scale it and then keep it running for years.
We use Erlang mostly for what it was designed - highly concurrent loads. In our case we need to manage 100,000s of concurrent TCP connections across many physical machines each with many cores, sending messages between those users. We need a presence system to prevent duplicate logins and re-use of licenses, and a queuing system to schedule requests onwards to components built in a more sequential style. And we want this to have a webservice and SNMP interface, so we can control and monitor everything.
It's rare that our business logic itself is written in Erlang, but we do sometimes use it for game-related tasks. We use it as a server for caching in-game leaderboards, for allowing a user to test their bandwidth, and for holding some transient in-game user data.
This talk will be a brief history of the company, from scrappy startup, and why we made the switch to Erlang over four years ago. We will talk about how we use Erlang, and the various mistakes we have made and lessons we have learned over that time.
Since the release of Erlang/OTP on GitHub the possibilities for the Erlang community to contribute to the development of Erlang has increased tremendously. We should leverage this possibility and the increased interest in Erlang to keep Erlang as a living and growing language by letting the community help the language evolve. In this slot, Erik will present the EPP process and committee.
The committee members present at the Erlang Factory will participate in a panel discussion together with the audience discussing the most important EEP proposals.
by Eric Cestari
ejabberd is the most popular XMPP server. It will turn 9 years old this year and connects millions of users everyday. One of the main reasons? It's written in erlang!
This presentation will focus on what edge erlang gives to ejabberd. We will talk about mnesia, pattern matching, clustering and all the other nice features we get for free with erlang, and how we use them in ejabberd.
While this is not a technical talk, the attendance may be exposed to some short erlang snippets.
6th–10th June 2011