Skip to content
Site map |
Annotation and Assessment
    Contents and Rights
      Discovery and Access
        Data Collection, Monitoring and Quality AssuranceStorage and Interoperability
          		  

          DMQ5: Improve the intelligence of the storage framework

          DMQ5 Report [PDF 86Kb]. 

          DMQ5, ‘Improve the intelligence of the storage framework.’ is extremely closely coupled to the DMQ2 Work Package, ‘Ensuring effective and reliable connection of selected instruments and sensors to storage repositories (SRB) via CIMA middleware, and efficient use via changed work practices.’

          The milestones for this work package have been achieved whilst also creating a significant component, the Data Manager, which is vital for integration with the Distributed Integrated Multi-Sensor and Instrument Middleware (DIMSIM), planned for the ARCHER project. 

          Triggers built into Storage Resource Broker (SRB) source stream

          The initial focus for solving this problem was by using the workflow system Kepler. This solution triggered events when changes were made in SRB using a Kepler actor. This method was flawed as files often appeared in the SRB collection before the process writing that file to the collection had completed. Since SRB uses a standard database, it was decided that the standard database triggers could be used to perform the trigger functions. Some Proof of Concept triggers were implemented in Postgresql.

          Production Implementation

          Production triggers have been implemented in Postgressql as part of the implementation of the X-ray Crystallography demonstrators, JCU And Indiana Instrument Service (JAINIS). Triggers will be implemented once live data becomes available from the ReefGrid project, DMQ1 and DMQ2. A third milestone was achieved that integrated DMQ2 DART components with SRB utilising Kepler and was a result of the investigations performed in the initial milestones.

          Kepler Workflow Production

          The DMQ2 work package required a set of triggers and workflow to ensure data was move into SRB. It was determined that a Kepler workflow would be an ideal method to create a Data Manager. A significant amount of effort was spent designing workflows for data management and building new actors to suit the requirements of the task. A significant investment was made in scoping the instruments and assessing work cases.

          The project achieved its milestones and exceeded requirements by implementing production quality workflow solutions in JAINIS. It is recommended that development of Kepler workflow continue for integration with both Instrument Middleware, DIMSIM, and for Grid based processing applications be continued by the Archer project to bring value to the scientific community. Scientific workflow provides great value to any collaborative environments by automating previously labour intensive and repetitive tasks  while dealing with institutional repositories and applications.