Sessions at PyCon US 2012 on Saturday 10th March in E1

One of the goals of PyPy is to make existing Python code faster, however an even broader goal was to make it possible to write things in Python that previous would needed to be written in C or other low-level language. This talk will show examples of this, and describe how they represent the tremendous progress PyPy has made, and what it means for people looking to use PyPy.

In this talk we'll talk about PyPy's present status and for what kinds of applications it might be useful. We'll also show examples of things that are possible with PyPy that were impossible to do with Python before, like real time video processing done in pure python. Our objective with each of the examples will be to highlight the type of work that is sped up and why this represents both a boon for existing Python programmers, as well as an opportunity for Python to expand to new audiences.

We'll also explain a bit PyPy's accomplishements in the passing year that make all of this possible, current status, its goals and the near future.

The Python community is abuzz about the major speed gains PyPy can offer pure Python code. But how does PyPy JIT actually work? This talk will discuss how the PyPy JIT is implemented. It will include descriptions of the tracing, optimization, and assembly generation phases. I will demonstrate each step with a example loop.

Strictly speaking, PyPy doesn't have a JIT; it has a JIT generator. I will describe how the JIT is generated during the translation of the interpreter as well as the advantages of the meta-JIT method.

Next, we'll explore tracing, in which the JIT records operations in a hot loop. I'll introduce the meta interpreter and the JIT IR (intermediate representation).

Optimizations are the next step of JITing. The focus will be on the most important optimizations for dynamic languages: virtuals and virtualizables. However, PyPy also includes the standard set of compiler peephole optimizations, like strength reduction, and well as some more complicated loop optimizations, like constant hoisting.

The final topic will be assembly generation. We'll see register allocation and how specific high-level Python operations are compiled down to tight x86 instructions.

Time permitting, additional topics may include how the garbage collector is integrated with the JIT and how the JIT bails back to the interpreter when a guard fails.

Along the way, I will demonstrate how each phase of JITing acts on an example Python loop. This will also allow me to introduce the jitviewer, a program to view how PyPy is compiling your loops.

As a dynamic language, Python is difficult to optimize. In addition, these dynamic features make using Python code from Java currently too complex. However, Java 7 adds the invokedynamic bytecode and corresponding library support, making it possible to finally address these problems in Jython. This talk will describe work in progress to make Jython faster and better (improving Java integration).

Jython demonstrates that it is quite possible to fit Python’s dynamic features on the Java Virtual Machine (JVM) to provide for seamless integration, while also taking advantage of the breadth of the Java platform. My favorite aspect of this blending is certainly is the amazing java.util.concurrent package.

However, from a performance perspective, it’s an awkward fit. In certain cases, the JVM is able to aggressively inline Python codepaths through its JIT. But generally it cannot for a variety of technical reasons. In addition, while Jython can conveniently call Java code, and support callbacks, it’s not at all convenient right now to go the opposite way. This mismatch is perhaps best seen now in Jython’s lack of support for Java annotations.

Consider how you, as a human translator, might attempt to optimize Python code for the JVM, or to fix the Java integration issues. Your plan of attack is simple: translate idiomatic Python to similarly idiomatic (and highly JIT-able) Java. Python developers signal their intent through a variety of mechanisms. They use builtin names like True/False or range/xrange, which through common convention, no one would seriously expect to see changed. They rarely monkey patch namespaces (action at a distance), although import time can get quite interesting. Importing packages from the java.* namespace means something when integrating from Jython. The challenge would be in supporting the dynamic functionality, however. Rewriting truly dynamic code into statically-typed code is just not the right way. It is non-trivial, error prone, and certainly not fun.

Enter the invokedynamic bytecode and the java.lang.invoke package, introduced with Java 7. It enables a wide range of optimizations, while allowing for the correctness of Python code, with its full range of dynamic features, no matter how crazy, to be maintained. There are some obvious wins, such as ensuring that a callsite (the point in the code where a call to a given function is invoked), a MethodHandle to a function in that namespace is linked, with all parameters properly permuted so that it’s a straight call through the Java calling convention. If there’s a namespace change, simply relink.

But there are also opportunities to use static analysis. For example, iterating over an xrange looks like a Java for loop, and can be optimistically compiled as such. If the builtin is rebound, the controlling SwitchPoint in invalidated and a continuation is setup to re-execute into an interpreter using unoptimized code (actually running Python bytecode). Other static analysis opportunities include being able to control the construction of frames for functions, use of decorators and function annotations to describe gradual typing (especially useful for Java integration), and so forth. This talk will cover a variety of these translations, demonstrate how we support both the fast and slow paths, and also describe some of the current performance benchmarks. I will also describe the pitfalls: obvious optimizations frequently result in bad performance due to the number of moving parts.

In addition, this talk with cover the state of Jython 2.6+. In particular, I will describe forthcoming changes to the Jython API (for embedding into Java). These include limited backwards breaking changes in ThreadState and PySystemState, to support increased performance, cleanup APIs, and remove issues in the garbage collection of ClassLoader objects.

This talk traces lightweight concurrency from Python 2.2's generators, which enabled semi-coroutines as a mechanism for scheduling "weightless" threads; to PEP 342, which created true coroutines, and hence made event-driven programming easier; to 3rd party libraries built around coroutines, from older GTasklet and peak.events to the current Greenlet/gevent and Twisted Reactor.

This talk aims to provide both a practical guide and theoretical underpinnings to the use of generator-based lightweight concurrency in Python.

Lightning tour of generator constructs. Why generator-based scheduling is particularly useful for event-based programming.
Simple example of a "trampoline" or scheduler.
Slightly fleshed out example of scheduler with discussion of data-passing issues.
Examples using GTasklet to make coroutine code look more like familiar sequential code (the framework is based on greenlets rather than generators, but accomplishes similar purpose).
Brief examples of Twisted Reactors and Deferreds.
Limits of generator-based concurrency (i.e. doesn't help with multiple cores and multiple servers). "Throw at the wall" list of ways to generalize to larger scales than single cores.

To really take advantage of Python you must understand how imports work and how to use them effectively. In this talk we'll discuss both of these. After a short introduction to imports, we'll dive right in and look at how customizing import behavior can make all your wildest dreams come true.

Python's import statement has been a powerful feature since the first release, and only gotten better with age. Understanding how imports work under the hood will let you take advantage of that power.

A big key to customizing Python's imports is the importers introduced by PEP 302. That's a tool that you want in your belt! We'll be covering such import hooks as well as a couple other customization methods.

Due to its robust namespacing, Python uniquely equips developers to write and distribute reusable code. The Python community has a tool for this: the Python Package Index. PyPI is a massive repository of code, and in this talk you'll learn how to take code that you've written it and register and distribute it for use by others.

Sharing is Caring: Posting to the Python Package Index
Introduction (2.5m)
Why?
I had been writing Python professionally for over a year before I learned how to do this. If I’d known how easy it was, perhaps I would make more code available.
What is the Python Package Index? (5m)
A repository of Python code written by thousands of different contributors
Brief history (“the Cheese Shop”), and bare-bones explanation of pip
Ultimately, it’s a place to help you re-invent as few wheels as possible
Reminders: Reusable Code (5m)
Everyone has a different use case.
Use options.
Support subclassing.
Some best practices for this.
The bottom line: Think about general needs, not just your specific needs.
Python internal documentation
...and write your own, too!
Walkthrough of Posting a Package (12.5m)
An explanation of setup.py
The file
The setup function
Things you need: name, author, version, packages, scripts
Bonus material
Registering and uploading your project.
Questions? (5m)

I won't just demonstrate how to use projects that bridge programming languages, I'll walk through the lower-level code that allows inter-language communication to happen.

This talk is written to engage developers interested in exploring the different forms that polyglot programs can take. The examples all demonstrate techniques for integrating Python with Scala, but the concepts are applicable beyond the specific use case of Python/Scala interop, and the talk does not assume prior knowledge of Scala.

Topics include:

Brief intro to Scala (enough to be able to follow the examples)
Basic script hosting APIs
Python/Scala integration via Jython
JEPP (CPython/JVM bridge)
Leveraging Python’s support for metaprogramming to make foreign function calls virtually indistinguishable from host language function calls
Passing a Python function to a Scala function that takes a Scala function as an argument
Interface Definition Language (IDL) and IDL-based strategies
Using TCP for inter-language communication
Attendees should be familiar Python's metaprogramming capabilities because I won’t be providing background information about adding or modifying attributes dynamically or interrogating a Python object for attribute details.

Java is in some ways a bogeyman to the Python community -- the language that parents scare their children with, the Cobol of the 21st century. But if we look past the cesspool of JEE it turns out that Java has quietly become an excellent systems environment, one that is still in many ways ahead of its time.

• Introduction
• The difference between systems and web (and scientific) computing
• Why Pythonistas should care about systems programming
• Concurrency
• Why "just use multiple processes" is inadequate
• Why event loops are inadequate
• java.util.concurrent: low level (collections, synch primitives)
• j.u.c.: high level (executors, futures, fork/join)
• The VM
• State of the art GC
• Built-in (and extensible) telemetry
• More functional than Python?!
• immutability
• Guava: com.google.*