An Open Standards-based Scalable Heavy Lifting Data Transfer Service for e-Research

Attendees

David Meredith (faciliator), Steve Crouch (scribe), Peter Turner, Alex Arana, Gerson Galang, David Wallom, Phil Kershaw, Weijian Fang, Ally Hume, Mario Antonioletti, Steve McGough

Overview

Data management across Grid and related infrastructures is one of the next big challenges in Cloud and Grid computing. How should data be curated/transferred/used between heterogeneous data sources, from generation by scientific instrument to the e-researcher's user desktop? In addition, initiating data transfers onto/off Grids from a range of clients, across different storage services (and importantly across different protocols) is a real issue. The DataMINX project aims to develop a service that addresses these issues in a scalable way.

The aims of this topic, covered over 4 sessions, were to address the following questions:

• As a service, architecturally, how could this capability be provided in an efficient and scalable manner to address simultaneous use by many researchers with possibly very large sets of data?
• The HPC File Staging Profile and OGSA Data Management Interface standards from the Open Grid Forum provide are two candidate specifications for providing an open interface to such a service. Where required, which modifications to these standards should be proposed, and how should these modifications be taken forward with the relevant standards groups?

Discussions were held on a variety of areas, including:

• Iterating use cases
• High level requirements and properties of such a service
• The proposed architecture
• How such an architecture could be rendered in different implementations, including GridSAM and GrisU
• Possible additions to using Commons-VFS to moving data
• Necessary OGF interactions.

Conclusions

• The proposed architecture was discussed and enhanced, and it was agreed that it meets the general requirements at this stage.
• In addition to Commons-VFS to move data and provide metadata, SAGA and Byte-IO are two candidates that could be used in the future to include support for additional protocols.
• A list of issues with the HPC File Staging Profile and OGSA-DMI specifications and ways to address these issues was created.
• There are some technical issues that will need to be addressed, such as how to route data movement requests to specific workers in different domains.
• The work has reached a suitable point to begin prototyping.

Further Work

• Specification issues:
  • S1: Schedule discussion within OGSA-DMI WG (Mario to organise)
  • S2: HPC File Staging Profile/JSDL WG’s (David M/Steve C to organise)
  • S3: DW: attend the OGF PGI sessions (Steve C)
• Design considerations
  • D4: Messaging routing for DTS worker nodes with domain constraints’ issue
• Implementation - ready to prototype:
  • I5: 1. JMS worker nodes
  • I6: 2. JSDL/extensions messaging format key for parallelising work
  • I7: 3. Self-contained WS + client (i.e. GridSAM, GrisU + standalone webservice, Hermes, NGS Portal)
• Resourcing:
  • DataMINX – 2 developers + Alex/Gerson over 2 years already committed
  • STFC/NGS – David Meredith
  • OMII-UK – Steve Crouch