Skip to content
Snippets Groups Projects
RELEASE_NOTES.txt 11 KiB
Newer Older
Release 1.9 (01/XX/2018)
=============================
  
- DM Station GUI fixes/enhancements:
  - Added support for DM_DATA_DIRECTORY_MAP environment variable; this allows
    browsing data locally, while movind data using GridFTP third
    party transfers

sveseli's avatar
sveseli committed
Release 1.8 (12/15/2017)
sveseli's avatar
sveseli committed
=============================
  
sveseli's avatar
sveseli committed
- Enhanced algorithm for processing of existing files; this resolved issues
  with multiple simultaneous DAQs involving large number of pre-existing
  files that need to be processed
- Added support for retrieving experiment dataset statistics:
  - New command:
    * get-file-collection-stats
sveseli's avatar
sveseli committed
- Added support for output variables in workflow definitions
- Added generic HDF5 file metadata reader
- Developed DAQ/PROC service integration via new DAQ plugin; this enables
  development and deployment of fully automated data acquisition/processing 
  pipelines based on user-defined workflows
  - Modified commands (all get --workflow-name, --workflow-owner, 
    --workflow-job-owner and --workflow-args options):
    * start-daq
    * upload
    * ${DM_STATION_NAME}-daq
    * ${DM_STATION_NAME}-upload
sveseli's avatar
sveseli committed
- Added HTTPS support for accessing file metadata catalog via Mongo Express
- DM Station GUI fixes/enhancements:
  - Added support for viewing experiment file metadata and file collection
    statistics
  - Added support for DM_BEAMLINE_MANAGERS environment variable
  - Improved dialogs for DAQ/upload configuration options
sveseli's avatar
sveseli committed

sveseli's avatar
sveseli committed
Release 1.7 (11/01/2017)
=============================

- DM Station GUI fixes/enhancements:
  - Added integration with ESAF DB
sveseli's avatar
sveseli committed
- Added new output formating option "--display-format=key-per-line" that is
  common to all commands 
sveseli's avatar
sveseli committed
- Introduced service monitoring infrastructure, which enabled deployment of
  Nagios-based system monitoring
sveseli's avatar
sveseli committed
  - New commands:
    * get-service-status 
    * clear-service-status 
sveseli's avatar
sveseli committed
Release 1.6 (10/07/2017)
sveseli's avatar
sveseli committed
=============================

sveseli's avatar
sveseli committed
- Added integration with APS ESAF DB via new APIs and CLIs
  - New commands:
    * list-esafs: list sector ESAFs by year
    * get-esafs: retrieve ESAF by id
  - Modified commands (new ESAF id option for setting experiment users)
    * ${DM_STATION_NAME}-daq 
    * ${DM_STATION_NAME}-upload
- Enhanced DAQ processing framework with introduction of meta
  classes for plugins and supporting utilities; this enables using
  different processing chain depending on data location, or other 
  distinguishing criteria
sveseli's avatar
sveseli committed
- DM Station GUI fixes/enhancements:
  - Added timer for automatic refresh of DAQ/upload status screens
  - Split configuration options for DAQs/uploads
  - Fixed issue with duplicate proposal users
  - Used current date as default for experiment start/end dates

sveseli's avatar
sveseli committed
Release 1.5 (08/04/2017)
sveseli's avatar
sveseli committed
=============================

- Added DS interface and Java API for downloading files 
sveseli's avatar
sveseli committed
- Added APIs and CLIs to clear (force service to forget) DAQs and uploads
  - New commands:
    * clear-daq
    * clear-upload
sveseli's avatar
sveseli committed
- Introduced APSU-related enhancements:
  - Processing for SDDS metadata in MongoDB cataloging plugin
  - Component Database processing plugin 
sveseli's avatar
sveseli committed
- DM Station GUI fixes/enhancements:
sveseli's avatar
sveseli committed
  - Fixed issue with DM_BEAMLINE_NAME not set
sveseli's avatar
sveseli committed
  - Fixed issues with buttons that require previous item selection
  - Added start date to list of DAQs and uploads
sveseli's avatar
sveseli committed
  - Added ability to remove completed DAQs or uploads from the top-level list
sveseli's avatar
sveseli committed

sveseli's avatar
sveseli committed
Release 1.4 (07/20/2017)
sveseli's avatar
sveseli committed
=============================

- Introduced DM Station GUI (accessed via dm-station-gui command)
- Introduced automated system test framework for DM stations
sveseli's avatar
sveseli committed
- Added --process-existing option to start-daq and <station>-daq
  commands; this will cause upload of existing files when starting DAQs
sveseli's avatar
sveseli committed

sveseli's avatar
sveseli committed
Release 1.3 (05/10/2017)
sveseli's avatar
sveseli committed
=============================

- Added APIs and CLIs to update experiment attributes and metadata
  catalog, as well as to delete experiments
sveseli's avatar
sveseli committed
  - New commands:
    * update-experiment
    * update-experiment-files
    * get-async-update-status
    * delete-experiment
sveseli's avatar
sveseli committed

Release 1.2 (04/18/2017)
=============================

- Modified scheduling algorithm for DAQs/uploads to simplify status monitoring

sveseli's avatar
sveseli committed
Release 1.1 (03/10/2017)
=============================

sveseli's avatar
sveseli committed
- Introduced integration with Beamline Scheduling System:
  - New commands: 
sveseli's avatar
sveseli committed
    * list-runs
    * list-proposals
    * get-proposal
  - Added the following options to the add-experiment command:
    --proposal-id: automatically add to experiment all 
      users associated with a given beamline proposal 
    --run: look for beamline proposal in a given run (current run is the 
      default)
    --users: comma specified list of usernames to be added to experiment
      as users
- Added the following options for managing DAQs:
  --duration: DAQ will be stopped automatically after given
    number of days or hours
  --dest-directory: files will be uploaded into
    a specific directory relative to experiment root path
sveseli's avatar
sveseli committed
  --upload-data-directory-on-exit: when DAQ finishes, upload of the given 
    data directory will be executed automatically
  --upload-dest-directory-on-exit: specifies destination directory for
    upload after DAQ completes
- Added the following options for managing uploads:
  --dest-directory: files will be uploaded into
    a specific directory relative to experiment root path
sveseli's avatar
sveseli committed
- Introduced framework for beamline-specific tools; added 
  beamline-specific commands that combine adding new experiment with running 
  DAQs or uploads: dm-${DM_STATION_NAME}-daq and dm-${DM_STATION_NAME}-upload 
- Introduced sphinx as python API documentation framework
sveseli's avatar
sveseli committed
- Resolved possible timeout issue when starting DAQ or directory upload 
  with a directory containing large number of files
- Simplified data directory command line option for beamlines that use 
  gridftp (via DM_DATA_DIRECTORY_MAP environment variable)
sveseli's avatar
sveseli committed
Release 1.0 (01/31/2017)
sveseli's avatar
sveseli committed
=============================

- Introduced concept of experiment station and redesigned authorization 
  mechanisms to allow beamline managers to manage their stations; all
  APIs and CLIs now conform to the new authorization scheme
- Modified get-experiments utility to allow retrieving list of experiments
  for a given station
sveseli's avatar
sveseli committed
- Cleaned up web portal by removing unused views, and enabled station 
  management functionality
sveseli's avatar
sveseli committed
- GPFS DDN (extrepid) has replaced xstor as the main APS storage
sveseli's avatar
sveseli committed
- CLI changes:
  - add-experiment command requires station name (can be set from env.
    variable); experiment type can be specified using type name
  - get-experiments command requires station name for beamline managers (can 
    be set from env. variable)
  - start-experiment command is now optional
sveseli's avatar
sveseli committed
Release 0.15 (11/01/2016)
sveseli's avatar
sveseli committed
=============================

- Resolved issue with incorrect accounting of processing errors for DAQs
sveseli's avatar
sveseli committed
- Improved DAQ processing algorithm to avoid resource starvation between
sveseli's avatar
sveseli committed
- Enhanced monitoring status information for both DAQs and uploads
sveseli's avatar
sveseli committed

Release 0.14 (10/14/2016)
sveseli's avatar
sveseli committed
=============================

- Introduced new framework and utilities for synchronizing users with 
  APS DB
- Resolved several issues with special characters in file names for
  gridftp transfer plugin  

sveseli's avatar
sveseli committed
Release 0.13 (05/27/2016)
sveseli's avatar
sveseli committed
=============================

- Added SFTP file system observer agent
sveseli's avatar
sveseli committed
- Enhanced MongoDB plugin with file md5 sum calculation
sveseli's avatar
sveseli committed

sveseli's avatar
sveseli committed
Release 0.12 (05/06/2016)
=============================

sveseli's avatar
sveseli committed
- Developed processing for HDF5 metadata in MongoDB cataloging plugin
sveseli's avatar
sveseli committed
- Modified catalog API and service interfaces to use file collections on 
  a per-experiment basis

Release 0.11 (04/29/2016)
=============================

- Resolved issue with upload command for directories containing large
  number of files
- Implemented enhanced upload processing algorithm to avoid resource 
  starvation between simultaneous DAQs and uploads
- Added new polling file system observer agent as option for monitoring 
  directories
sveseli's avatar
sveseli committed
- Reworked catalog API and corresponding MongoDB interfaces to use unique 
  experiment file paths, rather than file names
sveseli's avatar
sveseli committed
Release 0.10 (03/11/2016)
=============================

- Added dm-list-daqs and dm-list-uploads commands
- Resolved issue with newly created directories treated as files for 
  real-time data acquisitions
Release 0.9 (02/25/2016)
=============================

- Developed directory processing mode for uploads; in this mode file transfer 
  plugins transfer entire directories as opposed to individual files
- Added dm-get-processing-plugins command
- Resolved working directory issue that may occur with simutaneous uploads

sveseli's avatar
sveseli committed
Release 0.8 (01/26/2016)
=============================
sveseli's avatar
sveseli committed

sveseli's avatar
sveseli committed
- Enhanced upload/daq performance and functionality (hidden files are not
sveseli's avatar
sveseli committed
  processed; for uploads system can detect files that had been processed 
  already; improved handling and reporting of processing errors)
sveseli's avatar
sveseli committed
- Source file checksum is calculated for rsync/gridftp plugins by default
- Resolved globus online user authorization delay issue 
sveseli's avatar
sveseli committed

sveseli's avatar
sveseli committed
Release 0.7 (12/08/2015)
=============================

sveseli's avatar
sveseli committed
- Introduced framework and user interfaces for tracking progress of file 
  uploads and data acquisitions in DAQ service
- Added ability to monitor multiple directories for the same experiment 
  simultaneously (required changes to DAQ service REST interfaces)
sveseli's avatar
sveseli committed
- Enhanced start/stop DAQ and upload commands to use DM_FILE_SERVER_URL
  environment variable
- Added user interfaces and utilities that enable experiment data download
  from machines that have SSH access to the storage host
sveseli's avatar
sveseli committed
Release 0.6 (11/06/2015)
=============================

- Added file system observer agent interface for DAQ service
sveseli's avatar
sveseli committed
- Implemented FTP file system observer for DAQ service
- Added interfaces for deleting user experiment role in DS service
- Introduced java REST API framework, and specific experiment DS service API 
- Web Portal notifies DS service about experiment user modifications 
sveseli's avatar
sveseli committed
Release 0.5 (10/08/2015)
sveseli's avatar
sveseli committed
=============================

sveseli's avatar
sveseli committed
- Implemented Single Sign-On solution for backend services
- Enabled user authentication via login file
- Added file stat (with checksum) interface in DS web service
- After adding user role to experiment via command line, user is also
  added to experiment group (if one exists)
sveseli's avatar
sveseli committed
- Added rsync file transfer plugin with checksum and delete
sveseli's avatar
sveseli committed

Release 0.4 (09/21/2015) 
=============================

sveseli's avatar
sveseli committed
- Number of minor modifications made in preparation for test deployment at
sveseli's avatar
sveseli committed
  beamlines

Release 0.3 (07/22/2015) 
=============================

sveseli's avatar
sveseli committed
- Developed initial version of Catalogging Web Service based on MongoDB
- Developed sample processing plugins: file metadata catalog, SDDS processing, 
  SGE job submission
sveseli's avatar
sveseli committed

Release 0.2 (06/30/2015) 
=============================

- Implemented storage permission management and user group management
- Developed common file processing service plugin framework

Release 0.1 (04/21/2015) 
=============================

- Functional web portal (user, experiment, and policy pages)
- Developed web service and its API/CLI frameworks
- Developed initial version of Data Storage Web Service
- Developed initial version of Data Acquisition Web Service;
- DAQ service can monitor file system on a detector node and subsequently
  transfer data to storage
sveseli's avatar
sveseli committed