Re-implementation of the OGSA-DAI Distributed Query Processor in Clojure

Google Summer of Code 2010 ideas

Primary Mentor: Bartosz Dobrzelecki
Secondary Mentor: Amy Krause
Project: OGSA-DAI.

Background

Java is rather verbose and requires special care when you want to ensure immutability in your collections of objects. Also, the object oriented programming paradigm is not the best fit for rapid prototyping and evolutionary design. Modern IDEs can help with re-factoring but still, once you set a type structure it gets harder and harder to change it as your project progresses specially, when you are developing your understanding of the problem domain while you code. It is this inflexibility that has started to creep into the OGSA-DAI Distributed Query Processor (DQP). The code is full of design patterns: factories, strategies, visitors but these do not make the finding subtle bugs any easier. Our code is highly extensible, but we are starting to discover previously unanticipated extensibility points - the addition of these strains our type structure to the limits. We have also discovered that most of the data structures in our code are lists or trees (nested lists) and many of the algorithms transform lists. Thus now is the time to revisit this problem. Forget about using Java for this project though. Since lists are everywhere, the plan is to use a list processing language - Clojure.

Project Goals

The goals for this project are thus:

1. Re-implement the OGSA-DAI DQP query compiler in Clojure.

2. Asses how well the new code responds to new requirements.

3. Re-implement query optimisers in Clojure.

Project Description

The DQP compiler code that this project aims to replace takes an abstract syntax tree of an SQL query and translates it into an executable logical query plan (a tree of relational operators). Such a query plan is then rewritten by a set of query optimisers to produce a more efficient query plan.

The DQP compiler/optimiser code is part of a larger Java framework. We will need to ensure flawless interoperability with this. The query plans generated by the Clojure code will need to be mapped into an equivalent data structure in Java, so they can be translated by the existing Java code into the runtime representations that are used by OGSA-DAI.

Project Requirements

• previous experience with LISP
• Java
• relational databases (theory of, optional)