Newer
Older
Release 1.9 (01/XX/2018)
=============================
- DM Station GUI fixes/enhancements:
- Added support for DM_DATA_DIRECTORY_MAP environment variable; this allows
browsing data locally, while movind data using GridFTP third
party transfers
- Enhanced algorithm for processing of existing files; this resolved issues
with multiple simultaneous DAQs involving large number of pre-existing
files that need to be processed
- Added support for retrieving experiment dataset statistics:
- New command:
* get-file-collection-stats
- Added support for output variables in workflow definitions
- Added generic HDF5 file metadata reader
- Developed DAQ/PROC service integration via new DAQ plugin; this enables
development and deployment of fully automated data acquisition/processing
pipelines based on user-defined workflows
- Modified commands (all get --workflow-name, --workflow-owner,
--workflow-job-owner and --workflow-args options):
* start-daq
* upload
* ${DM_STATION_NAME}-daq
* ${DM_STATION_NAME}-upload
- Added HTTPS support for accessing file metadata catalog via Mongo Express
- DM Station GUI fixes/enhancements:
- Added support for viewing experiment file metadata and file collection
statistics
- Added support for DM_BEAMLINE_MANAGERS environment variable
- Improved dialogs for DAQ/upload configuration options
=============================
- DM Station GUI fixes/enhancements:
- Added integration with ESAF DB
- Added new output formating option "--display-format=key-per-line" that is
common to all commands
- Introduced service monitoring infrastructure, which enabled deployment of
Nagios-based system monitoring
- New commands:
* get-service-status
* clear-service-status
- Added integration with APS ESAF DB via new APIs and CLIs
- New commands:
* list-esafs: list sector ESAFs by year
* get-esafs: retrieve ESAF by id
- Modified commands (new ESAF id option for setting experiment users)
* ${DM_STATION_NAME}-daq
* ${DM_STATION_NAME}-upload
- Enhanced DAQ processing framework with introduction of meta
classes for plugins and supporting utilities; this enables using
different processing chain depending on data location, or other
distinguishing criteria
- DM Station GUI fixes/enhancements:
- Added timer for automatic refresh of DAQ/upload status screens
- Split configuration options for DAQs/uploads
- Fixed issue with duplicate proposal users
- Used current date as default for experiment start/end dates
=============================
- Added DS interface and Java API for downloading files
- Added APIs and CLIs to clear (force service to forget) DAQs and uploads
- New commands:
* clear-daq
* clear-upload
- Introduced APSU-related enhancements:
- Processing for SDDS metadata in MongoDB cataloging plugin
- Component Database processing plugin
- Fixed issues with buttons that require previous item selection
- Added start date to list of DAQs and uploads
- Added ability to remove completed DAQs or uploads from the top-level list
=============================
- Introduced DM Station GUI (accessed via dm-station-gui command)
- Introduced automated system test framework for DM stations
- Added --process-existing option to start-daq and <station>-daq
commands; this will cause upload of existing files when starting DAQs
=============================
- Added APIs and CLIs to update experiment attributes and metadata
catalog, as well as to delete experiments
- New commands:
* update-experiment
* update-experiment-files
* get-async-update-status
=============================
- Modified scheduling algorithm for DAQs/uploads to simplify status monitoring
- Introduced integration with Beamline Scheduling System:
- New commands:
* list-runs
* list-proposals
* get-proposal
- Added the following options to the add-experiment command:
--proposal-id: automatically add to experiment all
users associated with a given beamline proposal
--run: look for beamline proposal in a given run (current run is the
default)
--users: comma specified list of usernames to be added to experiment
as users
- Added the following options for managing DAQs:
--duration: DAQ will be stopped automatically after given
number of days or hours
--dest-directory: files will be uploaded into
a specific directory relative to experiment root path
--upload-data-directory-on-exit: when DAQ finishes, upload of the given
data directory will be executed automatically
--upload-dest-directory-on-exit: specifies destination directory for
upload after DAQ completes
- Added the following options for managing uploads:
--dest-directory: files will be uploaded into
a specific directory relative to experiment root path
- Introduced framework for beamline-specific tools; added
beamline-specific commands that combine adding new experiment with running
DAQs or uploads: dm-${DM_STATION_NAME}-daq and dm-${DM_STATION_NAME}-upload
- Introduced sphinx as python API documentation framework
- Resolved possible timeout issue when starting DAQ or directory upload
with a directory containing large number of files
- Simplified data directory command line option for beamlines that use
gridftp (via DM_DATA_DIRECTORY_MAP environment variable)
=============================
- Introduced concept of experiment station and redesigned authorization
mechanisms to allow beamline managers to manage their stations; all
APIs and CLIs now conform to the new authorization scheme
- Modified get-experiments utility to allow retrieving list of experiments
for a given station
- Cleaned up web portal by removing unused views, and enabled station
management functionality
- GPFS DDN (extrepid) has replaced xstor as the main APS storage
- CLI changes:
- add-experiment command requires station name (can be set from env.
variable); experiment type can be specified using type name
- get-experiments command requires station name for beamline managers (can
be set from env. variable)
- start-experiment command is now optional
=============================
- Resolved issue with incorrect accounting of processing errors for DAQs
- Improved DAQ processing algorithm to avoid resource starvation between
simultaneous DAQs and uploads
- Enhanced monitoring status information for both DAQs and uploads
=============================
- Introduced new framework and utilities for synchronizing users with
APS DB
- Resolved several issues with special characters in file names for
gridftp transfer plugin
=============================
- Added SFTP file system observer agent
Release 0.12 (05/06/2016)
=============================
- Developed processing for HDF5 metadata in MongoDB cataloging plugin
- Modified catalog API and service interfaces to use file collections on
a per-experiment basis
Release 0.11 (04/29/2016)
=============================
- Resolved issue with upload command for directories containing large
number of files
- Implemented enhanced upload processing algorithm to avoid resource
starvation between simultaneous DAQs and uploads
- Added new polling file system observer agent as option for monitoring
directories
- Reworked catalog API and corresponding MongoDB interfaces to use unique
experiment file paths, rather than file names
=============================
- Added dm-list-daqs and dm-list-uploads commands
- Resolved issue with newly created directories treated as files for
real-time data acquisitions
sveseli
committed
Release 0.9 (02/25/2016)
=============================
- Developed directory processing mode for uploads; in this mode file transfer
plugins transfer entire directories as opposed to individual files
- Added dm-get-processing-plugins command
- Resolved working directory issue that may occur with simutaneous uploads
Release 0.8 (01/26/2016)
=============================
- Enhanced upload/daq performance and functionality (hidden files are not
processed; for uploads system can detect files that had been processed
already; improved handling and reporting of processing errors)
- Source file checksum is calculated for rsync/gridftp plugins by default
- Added dm-stop-upload command
- Resolved globus online user authorization delay issue
- Introduced framework and user interfaces for tracking progress of file
uploads and data acquisitions in DAQ service
- Added ability to monitor multiple directories for the same experiment
simultaneously (required changes to DAQ service REST interfaces)
- Enhanced start/stop DAQ and upload commands to use DM_FILE_SERVER_URL
environment variable
- Added user interfaces and utilities that enable experiment data download
from machines that have SSH access to the storage host
=============================
- Added file system observer agent interface for DAQ service
- Implemented FTP file system observer for DAQ service
- Added interfaces for deleting user experiment role in DS service
- Introduced java REST API framework, and specific experiment DS service API
- Web Portal notifies DS service about experiment user modifications
- Implemented Single Sign-On solution for backend services
- Enabled user authentication via login file
- Added file stat (with checksum) interface in DS web service
- After adding user role to experiment via command line, user is also
added to experiment group (if one exists)
- Added rsync file transfer plugin with checksum and delete
Release 0.4 (09/21/2015)
=============================
- Number of minor modifications made in preparation for test deployment at
beamlines
Release 0.3 (07/22/2015)
=============================
- Developed initial version of Catalogging Web Service based on MongoDB
- Developed sample processing plugins: file metadata catalog, SDDS processing,
SGE job submission
Release 0.2 (06/30/2015)
=============================
- Implemented storage permission management and user group management
- Developed common file processing service plugin framework
Release 0.1 (04/21/2015)
=============================
- Functional web portal (user, experiment, and policy pages)
- Developed web service and its API/CLI frameworks
- Developed initial version of Data Storage Web Service
- Developed initial version of Data Acquisition Web Service;
- DAQ service can monitor file system on a detector node and subsequently
transfer data to storage