Mrs: MapReduce for Scientific Computing in Python

A session at Python for High Performance and Scientific Computing (PyHPC 2012)

Friday 16th November, 2012

11:45am to 12:30pm (MST)

The MapReduce parallel programming model is designed for large-scale data processing, but its benefits, such as fault tolerance and automatic message routing, are also helpful for computationally-intensive algorithms. However, popular MapReduce frameworks such as Hadoop are slow for many scientific applications and are inconvenient on supercomputers and clusters which are common in research institutions.
Mrs is a Python-based MapReduce framework that is well suited for scientific computing. We present comparisons of programs and run scripts to argue that Mrs is more convenient than Hadoop, the most popular MapReduce implementation. We also demonstrate that Mrs outperforms Hadoop for several types of problems that are relevant to scientific computing. In particular, Mrs demonstrates per-iteration overhead of about 0.3 seconds for Particle Swarm Optimization, while Hadoop takes at least 30 seconds for each MapReduce operation, a difference of two orders of magnitude.

Coverage of this session

Sign in to add slides, notes or videos to this session

Tell your friends!


Time 11:45am12:30pm MST

Date Fri 16th November 2012

Short URL


Official event site


View the schedule



See something wrong?

Report an issue with this session