Data Science with Hadoop at Opower

A session at Strata New York 2012

  • Erik Shilts

Wednesday 24th October, 2012

10:50am to 11:30am (EST)

Opower works with utility companies to provide engaging, relevant, and personalized content about home energy use to millions of households. We have found that simply providing data is not enough to change behavior; data are meaningful only to the extent that people can relate to it. Opower makes energy use data relatable and meaningful by using normative comparisons and personalized insights to help reduce energy use. This simple framework has enabled us to reduce consumer’s energy use by over one terawatt hour, or about 25% of the entire output of the US solar industry in 2011. Currently, Opower works with over 60 utilities domestically and internationally and houses energy data on over 30 million households. And our big data is becoming bigger—many households measure energy use monthly, but more and more homes have smart meters which keep energy usage reads at hourly or half hourly intervals.

This talk will discuss how we interact with the Hadoop ecosystem to transform raw data into contextualized information that drives behavior change. This starts with our Hadoop setup and how it has changed how we store data and think about problems. We then interface Hadoop with tools like R and Python to extract features, visualize, and validate data. On top of this setup, we have extended our set of addressable problems by utilizing curated crowdsourcing to classify patterns that humans can easily recognize. Lastly, we use statistical models and machine learning algorithms including nearest neighbors, self organizing maps, and regularized regressions to create models of characteristics that drive energy use.

The inputs and outputs of each step are stored in Hadoop for ease of use, transparency, and to facilitate collaboration. Throughout the talk I’ll give examples of data science and efficiency problems this infrastructure has solved and where we expect it to take us in the future.

About the speaker

This person is speaking at this event.
Erik Shilts

Senior Quantitative Analyst, Opower

Sign in to add slides, notes or videos to this session

Tell your friends!


Time 10:50am11:30am EST

Date Wed 24th October 2012

Short URL


View the schedule


See something wrong?

Report an issue with this session