Measuring Hard-to-Measure Things

A session at UCD2015 Humanity in Digital Landscapes: Mind, Cognition & Psychology

Saturday 24th October, 2015

4:00pm to 4:20pm (GMT)

GitHub has an abundance of quantitative data about what people are doing. Over the past two years we've built a practice of qualitative research dedicated to uncovering the why. Qualitative research surfaces blind spots with our product and our customers and has changed the way we ship features. We'll cover three GitHub stories, sharing how we now roll features out as controlled experiments.

Early product success is exciting and terrifying, because if the growth path is up-and-to-the-right your audience will broaden as will your challenges. Newcomers will arrive with experiences less like your earliest users and less like your own, and may begin using your product in ways you didn’t intend. This may leave you puzzled, scratching your head.

In GitHub’s case, there was a distinct shift in growth and audience. By 2013 developers experienced with version control and git signing up for accounts were far outnumbered by newcomers with limited experience, hungry and expecting to learn both. The caché of a GitHub profile and membership to its community brought many types of people through the door and into a product that is frankly pretty hard to figure out. More people were failing than those who were succeeding.

All products have blind spots, and no amount of analytics and monitoring can account for the hidden variables. Even the best systems can gather data on unusual behaviors for a long time without signaling that something is amiss. The sheer volume of data we have now can make it hard for talented analysts to utilize information, and powerful tools and visualizations are rendered less useful.

With regards to engineering challenges, data collection, and analysis we tend to think in terms of technical instruments. If you want to surface blind spots, you will have to get out from behind the numbers and in front of people. Think of yourself as a human instrument.

This talk will cover three examples of how GitHub uses research to deliver product and connect with customers:

1. How we measuring customer happiness with longitudinal survey analysis with our least happy customers (enterprise system administrators).
2. How and why we roll out new features on GitHub.com as controlled experiments.
3. How we design feature prioritization sneak attacks (learning through asking the right question) to learn more about customer culture.


About the speaker

This person is speaking at this event.
Chrissie Brodigan

Leading @github's UX research team. Ice skater once. Human interface for Paisley pug.

Next session in Maynard Theatre 2

4:20pm Alternate Reality Games. Non-linear Storytelling Across Multiple Platforms by Susan Martinez

4 attendees

  • Hayley Marsden
  • Melinda Seckington
  • Montgomery Webster
  • Chrissie Brodigan

Sign in to add slides, notes or videos to this session

Sign in to track this session

Tell your friends!


Time 4:00pm4:20pm GMT

Date Sat 24th October 2015


Maynard Theatre 2, The Kings Fund

Session Hash Tag


Short URL


Official event site


View the schedule


See something wrong?

Report an issue with this session