Sessions at PyCon US 2012 about Mozilla and Testing on Friday 9th March

Your current filters are…

  • Speedily Practical Large-Scale Tests

    by Erik Rose

    Mozilla's projects have thousands of tests, so we've had to venture beyond vanilla test runners to keep things manageable. Our secret sauce can be used with your project as well. Reach beyond the test facilities that came with your project, harnessing pluggable test frameworks, dynamically reordering tests for speed, exploring various mocking libraries, and profiling your way to testing nirvana.

    A partial outline:

    Motivation: a test not run is no test at all.
    For most web apps, the easiest test speed win is a conquest of I/O.
    The nose testrunner
    Test discovery lets you organize tests well.
    Gluing to projects with custom testrunners: django-nose and test-utils
    Compare to nose. Nose forked from it. Explain history.
    Very cool assertion re-evaluation
    Plugin compatibility between py.test and nose
    Start here. Premature optimization sucks.
    time on the commandline to divide CPU from I/O
    Killing I/O for speedy justice: case study of support.mozilla.com
    Fixture speed hacks (a 5x improvement!)
    Once-per-class setup
    How to use DB transactions to avoid repetitive I/O
    Dynamic test reordering and fixture sharing
    DB reuse and other startup optimizations
    37,583 queries to 4,116. Watch them fly by!
    What to do instead of fixtures: the model-maker pattern
    Lexical proximity
    Lower coupling
    Using mocking to kill the fixtures altogether
    mock, the canonical lib
    fudge, new declarative hotness
    Syntax, capabilities
    Example: oedipus, a better API for the Sphinx search engine. I used fudge to unit-test oedipus without requiring devs to set up and populate Sphinx.
    Dangers of mocking
    Don't mock out your caching unless your invalidation is perfect.
    Some of our mistakes in oedipus
    The nose-progressive display engine
    Test results that are a pain to read don't get read.
    Progress indication
    Elision of junk frames
    Easier round-tripping from test failure to source code
    Continuous integration
    IRC bots
    Next steps: what to do once you're CPU-bound
    More parallelization.
    Multithreading really buys you no speed bump for CPU-bound (or I/O bound?) tasks in Python due to the GIL. (Ref: PyCodeConf talk by David Beazley.)
    State of multiprocess plugins in various testrunners.
    Mozilla's Jenkins test farm
    QA's big stacks of Mac Minis
    What global warming? ;-)

    At 12:10pm to 12:55pm, Friday 9th March

    In D5, Santa Clara Convention Center