APS Data Management Software
Introduction
The APS Data Management system will streamline processing files created during data collection, and will ease the process of electronic access to data for users. The main design goal for the system is to alleviate tedious data management tasks for beamline staff, while ensuring the integrity and security of data. The system provides an infrastructure for organizing the files that comprise these data sets. By simplifying remote electronic access to data for users, it allows them to transfer or analyze their data shortly after their beam time.
The Data Management system provides easy-to-use automatic data transfers from beamlines to the central APS storage system (currently 250 TB) for short-term curation. From there, the system can also be used to transfer data to outside facilities, such as to a user’s home institution or to a longer-term storage system such as that maintained by Argonne’s CELS Directorate (currently 1.7 PB) allowing data to be staged for analysis with Argonne’s Leadership Computing Facility resources, such as the Mira supercomputer. A service layer within the Data Management system automates data transfer from acquisition systems to the storage system and policy enforcement, such as automatic deletion/archival. The GlobusOnline service is used to transfer data to remote locations
A web portal, integrated with the APS User database and experiment scheduling and safety systems, is used to control data access settings and policies. The GUI is largely complete and the service layer is under development. Prototype deployment targeted for mid-2015. The status of the project can be found on the Project Status page.
Details about the software and hardware design and infrastructure can be found of the project's Infrastructure page.
This project was started using Argonne LDRD funds and is now being made production ready using operational funding from the APS, contract DE-AC02-06CH11357.
Installation
Work is underway to separate general features of the Data Management System to make this useable at facilities outside of the APS. As these efforts continue, this page will be updated to provide instructions for developers and for production use at the APS and then also to provide instructions for use outside of the APS.