Speedily Practical Large-Scale Tests

A session at PyCon US 2012

Friday 9th March, 2012

12:10pm to 12:55pm (PST)

Mozilla's projects have thousands of tests, so we've had to venture beyond vanilla test runners to keep things manageable. Our secret sauce can be used with your project as well. Reach beyond the test facilities that came with your project, harnessing pluggable test frameworks, dynamically reordering tests for speed, exploring various mocking libraries, and profiling your way to testing nirvana.

A partial outline:

Motivation: a test not run is no test at all.
For most web apps, the easiest test speed win is a conquest of I/O.
The nose testrunner
Test discovery lets you organize tests well.
Gluing to projects with custom testrunners: django-nose and test-utils
Compare to nose. Nose forked from it. Explain history.
Very cool assertion re-evaluation
Plugin compatibility between py.test and nose
Start here. Premature optimization sucks.
time on the commandline to divide CPU from I/O
Killing I/O for speedy justice: case study of support.mozilla.com
Fixture speed hacks (a 5x improvement!)
Once-per-class setup
How to use DB transactions to avoid repetitive I/O
Dynamic test reordering and fixture sharing
DB reuse and other startup optimizations
37,583 queries to 4,116. Watch them fly by!
What to do instead of fixtures: the model-maker pattern
Lexical proximity
Lower coupling
Using mocking to kill the fixtures altogether
mock, the canonical lib
fudge, new declarative hotness
Syntax, capabilities
Example: oedipus, a better API for the Sphinx search engine. I used fudge to unit-test oedipus without requiring devs to set up and populate Sphinx.
Dangers of mocking
Don't mock out your caching unless your invalidation is perfect.
Some of our mistakes in oedipus
The nose-progressive display engine
Test results that are a pain to read don't get read.
Progress indication
Elision of junk frames
Easier round-tripping from test failure to source code
Continuous integration
IRC bots
Next steps: what to do once you're CPU-bound
More parallelization.
Multithreading really buys you no speed bump for CPU-bound (or I/O bound?) tasks in Python due to the GIL. (Ref: PyCodeConf talk by David Beazley.)
State of multiprocess plugins in various testrunners.
Mozilla's Jenkins test farm
QA's big stacks of Mac Minis
What global warming? ;-)

About the speaker

This person is speaking at this event.
Erik Rose

Python wrangler, language hacker, elegance chaser

Next session in D5

2pm IPython: Python at your fingertips by Min Ragan-Kelley, Brian E. Granger, Thomas Kluyver and Fernando Pérez

Coverage of this session

Sign in to add slides, notes or videos to this session

Tell your friends!


Time 12:10pm12:55pm PST

Date Fri 9th March 2012

Short URL


Official session page


View the schedule



Books by speaker

  • Plone 3 for Education

See something wrong?

Report an issue with this session