Add an eventSessions at PyCon US 2012 about Performance and Python
Wednesday 7th March 2012
-
Faster Python Programs through Optimization
by Mike Müller
This tutorial provides an overview of techniques to improve the performance of Python programs. The focus is on concepts such as profiling, difference of data structures and algorithms as well as a selection of tools and libraries that help to speed up Python.
Objective
This tutorial provides an overview of techniques to improve the performance of Python programs. The focus is on concepts such as profiling, diffrence of data structures and algorithms as well as a selection of tools an libraries that help to speed up Python.
Intended Audience
Python programmers who would like concepts to improve performance.
Audience Level
Programmers with good Python knowledge.
Prerequisites
Please bring your laptop with the operating system of your choice (Linux, Mac OS X, Windows). In addition to Python 2.6 or 2.7, we need:
- RunSnakeRun (http://www.vrplumber.com/program...)
- the Guppy_PE framework (http://guppy-pe.sourceforge.net)
- psyco (http://psyco.sourceforge.net, Python 2.6 only, version 1.5.2 or higher)
- pypy (http://pypy.org) and
- NumPy (http://numpy.scipy.org, version 1.2 or higher).
Method
This is a hands-on course. Students are strongly encouraged to work along with the trainer at the interactive prompt. There will be exercises the students need to do on their own. Experience shows that this active involvement is essential for an effective learning.
Outline
- How fast is fast enough? (10 min)
- Optimization guidelines (10 min)
- Premature optimization
- Optimization rules
- Seven steps for incremental optimization
- Optimization strategy (30 min)
- Measuring in stones
- Profiling CPU usage
- Profiling memory usage
- Algorithms and Anti-patterns (40 min)
- String concatenation
- List and generator comprehensions
- The right data structure
- Caching
- The example (5 min)
- Testing speed (5 min)
- Pure Python (15 min)
- Meet Psyco, the JIT (5 min)
- Using PyPy (15 min)
- NumPy for numeric arrays (10 min)
- Using multiple CPUs with multiprocessing (20 min)
- Combination of optimization strategies (10 min)
- Results of different example implementations (5 min)
Thursday 8th March 2012
-
High Performance Python I
by Ian Ozsvald
At EuroPython 2011 I ran a very hands-on tutorial for High Performance Python techniques. This updated tutorial will cover profiling, PyPy, Cython, numpy, NumExpr, ShedSkin, multiprocessing, ParallelPython and pyCUDA. Here's a 55 page PDF write-up of the EuroPython material: http://ianozsvald.com/2011/07/25...
At EuroPython 2011 I ran a very hands-on tutorial for High Performance Python techniques. This updated tutorial will cover:
- profiling with cProfile, run_snake and line_profiler
- PyPy
- Cython
- numpy with and without vectors
- NumExpr
- ShedSkin Py->C++ compiler
- multiprocessing for multi-core
- ParallelPython for multi-machine
- pyCUDA demos
I plan to expand the original material and to maybe also cover other tools like execnet and PyPy-numpy.
-
Optimize Performance and Scalability with Parallelism and Concurrency
From how the operating system handles your requests through design principles on how to use concurrency and parallelism to optimize your program's performance and scalability. We will cover processes, threads, generators, coroutines, non-blocking IO, and the gevent library.
How processes, threads, coroutines, and non-blocking IO work from the operating system through code implementation and design principles to optimize Python programs. The difference between parallelism and concurrency and when to use each.
The premise is that to make an informed decision you need to know what is happening under the hood. Once you understand the low level functionality, you can make the correct decision in the design phase.
The emphasis is on practical application to solve real world problems.
Outline
- How the operating system handles traps and interrupts
- Scheduling
- Processes
- Threads
- The GIL
- Generators
- What is a coroutine?
- What is a Python coroutine?
- Blocking/Non-blocking I/O.
- Parallelism versus Concurrency
- How do these work with CPython, Pypy, and Stackless
- Greenlets and libevent (gevent)
- Design principles
- Example networked application
- Performance results
- What are other the other options?
-
High Performance Python II
In this tutorial, I will cover how to write very fast Python code for data analysis. I will briefly introduce NumPy and illustrate how fast code for Python is written in SciPy using tools like Fwrap / F2py and Cython. I will also describe interesting new approaches to creating fast code that is leading changes to NumPy on a fundamental level.
In this tutorial, I will cover how to write very fast Python code for data analysis including making use of NumPy and using GPUs. I will largely focus on writing extensions to Python using hand-wrapping and Cython but will touch also on using tools like weave, Instant, ShedSkin and compare them to PyPy. I will also spend the last part of the tutorial on using GPUs with Python and discuss the performance trade-offs of the technology. This will be a high-level overview of the space with deep dives in Cython and GPUs
Outline:
- Brief Introduction to NumPy, SciPy and array-oriented computing with Python including exercises (1 hour)
- Introduction to hand-wrapping and extending Python (1 hour)
- Detailed description of Cython and how to use it to connect to machine-compiled code (1 hour)
- Detailed description of GPUs and how to use them best with NumPy (45 minutes)
- Summary and overview of using Python to write super fast code (15 minutes)
Friday 9th March 2012
-
Fast Test, Slow Test
Most unit tests aren't and their authors suffer for it. What is a unit test, really? How can writing them prevent classic testing problems? If you do write them, what trade-offs are you implicitly making?
Your unit test suite takes three minutes to run 500 tests. On a modern CPU, that's just shy of a billion instructions per test. Can something that takes a billion CPU instructions really be called a "unit"? What is that suite really testing?
Many (and probably most) unit testing failures are related to size. Most common are the suite that takes half an hour to run (so no one runs it), the suite whose runtime scales like lines_of_code^2 (so, again, no one runs it), and the suite that requires huge maintenance for small changes (leading to the "testing is slow" myth).
This talk is about the "unit" in "unit test": what does it really mean, why do we care, and how does it prevent the three crippling problems above? And, of course, if we do shift our focus toward unit tests, what trade-offs are we really making?
Saturday 10th March 2012
-
Web Server Bottlenecks And Performance Tuning
New Python web developers seem to love running benchmarks on WSGI servers. Reality is that they often have no idea what they are doing or what to look at. This talk will look at a range of factors which can influence the performance of your Python web application. This includes the impact of using threads vs processes, number of processors, memory available, the GIL and slow HTTP clients.
A benchmark of a hello world application is often what developers use to make the all important decision of what web hosting infrastructure they use. Worse is that in many cases this is the only sort of performance testing or monitoring they will ever do. When it comes to their production applications they are usually flying blind and have no idea of how it is performing and what they need to do to tune their web application stack.
This talk will discuss different limiting factors or bottlenecks within your WSGI server stack and system that can affect the performance of your Python web application. It will illustrate the impacts of these by looking at typical configurations for the more popular WSGI hosting mechanisms of Apache/mod_wsgi, gunicorn and uWSGI, seeing how they perform under various types of traffic and request loads and then tweaking the configurations to see whether they perform better or worse.
Such factors that will be discussed will include:
Use of threads vs processes.
Number of processors available.
Python global interpreter lock (GIL)
Amount of memory available.
Slow HTTP browsers/clients.
Browser keep alive connections.
Need to handle static assets.
From this an attempt will be made to provide some general guidelines of what is a good configuration/architecture to use for different types of Python web applications. The importance of continuous production monitoring will also be covered to ensure that you know when the performance of your system is dropping off due to changing traffic patterns as well as code changes you have made in your actual web application.
Schedule incomplete?
Filter by Day
- Wednesday 7th 1
- Thursday 8th 3
- Friday 9th 1
- Saturday 10th 1
- Sunday 11th 0
- Monday 12th 0
- Tuesday 13th 0
- Wednesday 14th 0
- Thursday 15th 0