The Apache Hadoop MapReduce framework has hit a scalability limit around 4,000 machines. We are developing the next generation of Hadoop MapReduce that factors the framework into a generic resource scheduler and a per-job, user-defined component that manages the application's execution. High availability, security, and improved multi-tenancy are fundamental to the new architecture. The new architecture also increases innovation, agility and hardware utilization.
23rd March 2011