Dynamic MapReduce

Google Summer of Code 2010 ideas

Primary Mentor: Shantenu Jha
Secondary Mentor: Andre Merzky
Project: OMII SAGA.

Background

Most MapReduce implementations utilize a static set of resources. Those that can utilize a dynamic set of resources, are bound to a specific type of infrastructure. A SAGA-based MapReduce was developed as part of earlier GSoC, which removes the latter restriction. SAGA-based MapReduce has been shown to utilize a broad range of grids and clouds concurrently.

Project Goals

This project aims to develop an extended and enhanced SAGA-based MapReduce which has the capability to dynamically adapt to a changing set of resources based upon data-transfer requirements and work-load requirements.

Project Description

These Capabilities will be tested and ultimately provided on the US-TeraGrid and UK-NGS, as well as a range of cloud offerings.

Project Requirements

Intermediate C/C++ skills along experience working in a Unix environment.