Constructing a Next Generation Open Source Web Querying System with Node.js and ql.io

In most modern web and mobile applications, much of the raw data source backbone comes from the mashing up of HTTP based APIs from multiple sites and services, both internally and externally. These sources are used for everything from localization to personalization and product querying. As more services are added to provide this functionality, the number of network roundtrips increases along with bandwidth consumption, and due to interdependency problems and inconsistent APIs, product reliability suffers.

These were the main developer pains that lead to the development of ql.io, an open source querying system built on top of Node.js. Using a SQL and JSON based language to declare API calls, interdependencies, forks and joins, filtering and projections, app developers are able to cut down the amount of effort required to integrate multiple APIs from hundreds of lines of code down to a few. Besides the language itself, ql.io can be deployed as a stand-alone server, allowing you to deploy it as an HTTP gateway between client apps and API servers to have ql.io process and condense the data to just the fields that the client needs.

In this talk we will explore the underpinnings of ql.io and how Node.js powers this next generation system. We will look into how to query multiple HTTP API data sources simply and efficiently, and explore how the service can be deployed on your own systems to handle raw mashups of your own and external data, all while decreasing overall bandwidth usage, network roundtrips, and increasing product reliability.