•  

ClusterShell, Python library and tools for scalable cluster administration

A session at Python for High Performance and Scientific Computing (PyHPC 2013)

  • Stéphane Thiell
  • Henri Doreau

Monday 18th November, 2013

11:10am to 11:50am (MST)

Cluster-wide administrative tasks and other distributed jobs are often executed by administrators using locally developed tools and do not rely on a solid, common and efficient execution framework. This document covers this subject by giving an overview of ClusterShell, an open source Python middleware framework developed to improve the administration of HPC Linux clusters or server farms.

ClusterShell provides an event-driven library interface that eases the management of parallel system tasks, such as copying files, executing shell commands and gathering results. By default, remote shell commands rely on SSH, a standard and secure network protocol. Based on a scalable, distributed execution model using asynchronous and non-blocking I/O, the library has shown very good performance on petaflop systems. Furthermore, by providing efficient support for node sets and more particularly node groups bindings, the library and its associated tools can ease cluster installations and daily tasks performed by administrators. In addition to the library interface, this document addresses resiliency and topology changes in homogenous or heterogenous environments. It also focuses on scalability challenges encountered during software development and on the lessons learned to achieve maximum performance from a Python software engineering point of view.

About the speakers

This person is speaking at this event.
Stéphane Thiell

CEA

This person is speaking at this event.
Aurélien Degrémont

HPC System Engineer chez CEA bio from LinkedIn

This person is speaking at this event.
Henri Doreau

CEA

Next session in 505

12:50pm Bohrium: Unmodified NumPy Code on CPU, GPU, and Cluster by Brian Vinter, Kenneth Skovhede, Troels Blum, Simon A. F. Lund and Mads Ruben Burgdorff Kristensen

Coverage of this session

Sign in to add slides, notes or videos to this session

Tell your friends!

When

Time 11:10am11:50am MST

Date Mon 18th November 2013

Session Hash Tag

#SC13

Short URL

lanyrd.com/sctbgr

Official event site

www.dlr.de/sc/pyhpc2013

View the schedule

Share

See something wrong?

Report an issue with this session