RELEASE_NOTES.txt

Release 3.3.3 (XX/YY/2020)
=============================
- Fixed issue with non-ascii characters for daq/upload CLI tools

Release 3.3.2 (07/09/2020)
=============================
- Added ability to upload files from remote machines via rsync plugin and using
  ssh
- Added start/end time into GUP proposal object

Release 3.3.1 (05/06/2020)
=============================
- Maintenance release

Release 3.3.0 (05/04/2020)
=============================
- Change log4j version from 1.2.7 to 2.13.2. Code changes to adapt were
  necessary. Changed config file from log4j.properties to log4j2.xml
- Change jar file for postgresql from 9.3-1102.jdbc41 to 42.2.8.
  The underlying postgresql in support changed from 9.3 to 9.6 a while ago.

Release 3.2.2 (02/24/2020)
=============================
- Added user documentation (Getting Started guide)

Release 3.2.1 (01/15/2020)
=============================
- DM GUI Changes:
  - Reworked configuration options for DAQs and uploads; those now support
    including/excluding file extensions  
  - Fixed bug that was preventing using configuration options and additional
    keywords at the same time

Release 3.2.0 (11/05/2019)
=============================
- Implemented ability to include or exclude file extensions for uploads and DAQs
  - New API Options for starting DAQ/Upload
    * includeFileExtensions
    * excludeFileExtensions
  - New CLI Options for starting DAQ/Upload
    * include-extensions
    * exclude-extensions

Release 3.1.0 (10/25/2019)
=============================
- Added new ESAF interfaces for support of extended data retrieval
- Enabled URL scheme for transfer plugin configuration; this allows
  multiple gridftp hosts per deployment
- Updated support software to allow offsite installation
- DM GUI changes:
  - Add settings page
    - Allow user to specify refresh rates per module. 
    - Fixed protocol parsing bug

Release 3.0.1 (08/09/2019)
=============================
- Fixed issue with unchecked additions to metadata catalog
- Updated support software:
  - OpenJDK (replaces Oracle JDK)
  - Payara (replaces Glassfish)
- Added 12IDB utilities

Release 3.0.0 (07/26/2019)
=============================
- Added python web service API for downloading files 
- Updated DB schema for experiment data archival support
- Web Portal changes:
  - Added storage views
  - Added connection between station and experiment types
  - Updated experiment views with storage and root path fields

Release 2.6.0 (07/15/2019)
=============================
- Moved code repo to gitlab
- Updated release/upgrade utilities

Release 2.5.0 (06/23/2019)
=============================
- Added pprint (pretty-print) option to display format for all commands
- Added support for repeated commands via repeatPeriod/repeatUntil/maxRepeats
  keys in workflow definitions
- Added skip/limit options to list-processing-jobs 
- Implemented ability to stop processing jobs
  - New command:
    * stop-processing-job
- Enhanced support for processing multiple files with a given workflow
  - New command:
    * process-files
- Updated workflow API documentation
- DM Station GUI fixes/enhancements:
  - Fixed issue with modifying user list before saving experiment

Release 2.4.0 (05/20/2019)
=============================
- Added system LDAP utilities for verifying and creating groups
- Added support for retrieving single processing job stage via the --stage
  option for the get-processing-job command
- Prevented core files from being uploaded
- Prevented creation of experiment names with spaces
- Fixed file limit problem in compression utilities
- DM Station GUI fixes/enhancements:
  - Fixed issue with experiment user list modification
  - Added tab for monitoring processing jobs
  - Restored workflow tab 

Release 2.3.0 (04/15/2019)
=============================
- Updated CDB plugin for the latest CDB API changes
- Expanded support for HTML-formatted output
- Optimized amount of data transferred with file metadata listing
- Implemented limit on number of files that can be retrieved using
  a single list-experiment-files command or API call
- Added more diagnostic output to compression/decompression utilities
- DM Station GUI fixes/enhancements:
  - Added logging and ability to use DM_CONSOLE_LOG_LEVEL environment 
    variable to specify logging level
  - Resolved issue with undefined self key in exception hook
  - Added support for single file upload
  - Added support for experiment root path
  - Added warning dialog for experiment updates
  - Added number of tooltips 
  - Optimized handling of file metadata for experiments with large
    number of files
  - Added support for pagination while fetching full list of files
  - Implemented progress dialog when fetching a very large set of files
  - Resolved issue with filter boxes layout not matching data table 
    default column sizes

Release 2.2.2 (02/11/2019)
=============================
- Updated deployment scripts
- Fixed several Python 2/3 compatibility issues

Release 2.2.1 (02/04/2019)
=============================
- Updated API documentation
- DM Station GUI fixes/enhancements:
  - Added file compression capability

Release 2.2.0 (01/15/2019)
=============================
- Added ESAF interfaces to APS DB service 
- Moved all ESAF/BSS command line utilities to APS DB service 
  - New command:
    * list-beamlines 
- Converted software to Python 3, but retained Python 2 compatibility
- Introduced support for API pip and conda packages
- Added support for a single file upload via the --file-path option to
  upload commands
- Added support for arbitrary experiment path under the ${DM_STATION_NAME}
  directory via the --root-path option to the following commands:
    * add-experiment
    * update-experiment
    * ${DM_STATION_NAME}-daq
    * ${DM_STATION_NAME}-upload
  Note that if this option is not used, experiment will be located under
  the experiment type folder, as before. 

Release 2.1.0 (12/31/2018)
=============================
- Added estimated completion time to upload monitoring
- Introduced additional enhancements for cataloging plugin that shorten
  uploads of large datasets
- Developed initial image receiver service and utilities that can accept
  Area Detector NTNDArray frames and save them into files; currently
  supported formats are SDDS, JSON, HDF5, and JPEG.
  - New commands:
    * start-image-receiver
    * stop-image-receiver
- Added new APS DB service for accessing beamline scheduling system 

Release 2.0.0 (10/15/2018)
=============================
- Modified DM DB to support arbitrary experiment root paths

Release 1.9.3 (09/15/2018)
=============================
- Added generic script processing plugin for DAQ service
- Parallelized cataloging plugin; this significantly increased performance 
  for uploads using directory mode

Release 1.9.2 (05/15/2018)
=============================
- Added support for compressing and decompressing files
  - New commands:
    * compress-files
    * decompress-files
- Added support for deleting files
  - New commands:
    * delete-file
    * delete-files
- Modified stat-file utility, which now requires explicit --md5sum flag to
  retrieve checksum
- Added new --path-pattern option to the list-files command
- DM Station GUI fixes/enhancements:
  - Added initial GUI test suite
  - Added right-click copying functionality
  - Fixed issue with trailing comma for DAQ/upload parameters
  - Fixed CPU usage bug 
  - Fixed incorrect handling of user selections
  - Fixed focus policy for several buttons

Release 1.9.1 (03/30/2018)
=============================
- Added support for command output in html format
- Added support for requesting only subset of keys for list of processing jobs
- Resolved issues with symbolic links for uploading directories and files 
  via rsync 
- Fixed DAQ mode issue with double counting of observed files under certain
  circumstances
- Fixed problem with cataloging of HDF5 files that contain non-ascii metadata
- DM Station GUI fixes/enhancements:
  - Fixed issue with user permissions when creating new experiments
  - Fixed upload configuration issue 
  - Fixed caching issues for experiment file metadata and users
  - Added live metadata browsing while files load in background 
  - Added sorting for user table
  - Added ability to copy table information to clipboard
  - Improved selection behavior on tables
  
Release 1.9.0 (01/31/2018)
=============================
  
- Added support for retrieving file metadata for a given experiment in 
  batches; list-experiment-files get --skip and --limit options
- Simplified workflow and processing job management by allowing DM session user
  to be used as default workflow and/or processing job owner
  - Modified commands:
    * list-workflows
    * get-workflow
    * delete-workflow
    * start-processing-job
    * list-processing-jobs
- DM Station GUI fixes/enhancements:
  - Added support for DM_DATA_DIRECTORY_MAP environment variable; this allows
    browsing data locally, while moving data using GridFTP third
    party transfers
  - Enhanced support for viewing large file metadata collections

Release 1.8.0 (12/15/2017)
=============================
  
- Enhanced algorithm for processing of existing files; this resolved issues
  with multiple simultaneous DAQs involving large number of pre-existing
  files that need to be processed
- Added support for retrieving experiment dataset statistics:
  - New command:
    * get-file-collection-stats
- Added support for output variables in workflow definitions
- Added generic HDF5 file metadata reader
- Developed DAQ/PROC service integration via new DAQ plugin; this enables
  development and deployment of fully automated data acquisition/processing 
  pipelines based on user-defined workflows
  - Modified commands (all get --workflow-name, --workflow-owner, 
    --workflow-job-owner and --workflow-args options):
    * start-daq
    * upload
    * ${DM_STATION_NAME}-daq
    * ${DM_STATION_NAME}-upload
- Added HTTPS support for accessing file metadata catalog via Mongo Express
- DM Station GUI fixes/enhancements:
  - Added support for viewing experiment file metadata and file collection
    statistics
  - Added support for DM_BEAMLINE_MANAGERS environment variable
  - Improved dialogs for DAQ/upload configuration options

Release 1.7.0 (11/01/2017)
=============================

- DM Station GUI fixes/enhancements:
  - Added integration with ESAF DB
- Added new output formating option "--display-format=key-per-line" that is
  common to all commands 
- Introduced service monitoring infrastructure, which enabled deployment of
  Nagios-based system monitoring
  - New commands:
    * get-service-status 
    * clear-service-status 

Release 1.6.0 (10/07/2017)
=============================

- Added integration with APS ESAF DB via new APIs and CLIs
  - New commands:
    * list-esafs: list sector ESAFs by year
    * get-esafs: retrieve ESAF by id
  - Modified commands (new ESAF id option for setting experiment users)
    * ${DM_STATION_NAME}-daq 
    * ${DM_STATION_NAME}-upload
- Enhanced DAQ processing framework with introduction of meta
  classes for plugins and supporting utilities; this enables using
  different processing chain depending on data location, or other 
  distinguishing criteria
- DM Station GUI fixes/enhancements:
  - Added timer for automatic refresh of DAQ/upload status screens
  - Split configuration options for DAQs/uploads
  - Fixed issue with duplicate proposal users
  - Used current date as default for experiment start/end dates

Release 1.5.0 (08/04/2017)
=============================

- Added DS interface and Java API for downloading files 
- Added APIs and CLIs to clear (force service to forget) DAQs and uploads
  - New commands:
    * clear-daq
    * clear-upload
- Introduced APSU-related enhancements:
  - Processing for SDDS metadata in MongoDB cataloging plugin
  - Component Database processing plugin 
- DM Station GUI fixes/enhancements:
  - Fixed issue with DM_BEAMLINE_NAME not set
  - Fixed issues with buttons that require previous item selection
  - Added start date to list of DAQs and uploads
  - Added ability to remove completed DAQs or uploads from the top-level list

Release 1.4.0 (07/20/2017)
=============================

- Introduced DM Station GUI (accessed via dm-station-gui command)
- Introduced automated system test framework for DM stations
- Added --process-existing option to start-daq and <station>-daq
  commands; this will cause upload of existing files when starting DAQs

Release 1.3.0 (05/10/2017)
=============================

- Added APIs and CLIs to update experiment attributes and metadata
  catalog, as well as to delete experiments
  - New commands:
    * update-experiment
    * update-experiment-files
    * get-async-update-status
    * delete-experiment

Release 1.2.0 (04/18/2017)
=============================

- Modified scheduling algorithm for DAQs/uploads to simplify status monitoring

Release 1.1.0 (03/10/2017)
=============================

- Introduced integration with Beamline Scheduling System:
  - New commands: 
    * list-runs
    * list-proposals
    * get-proposal
  - Added the following options to the add-experiment command:
    --proposal-id: automatically add to experiment all 
      users associated with a given beamline proposal 
    --run: look for beamline proposal in a given run (current run is the 
      default)
    --users: comma specified list of usernames to be added to experiment
      as users
- Added the following options for managing DAQs:
  --duration: DAQ will be stopped automatically after given
    number of days or hours
  --dest-directory: files will be uploaded into
    a specific directory relative to experiment root path
  --upload-data-directory-on-exit: when DAQ finishes, upload of the given 
    data directory will be executed automatically
  --upload-dest-directory-on-exit: specifies destination directory for
    upload after DAQ completes
- Added the following options for managing uploads:
  --dest-directory: files will be uploaded into
    a specific directory relative to experiment root path
- Introduced framework for beamline-specific tools; added 
  beamline-specific commands that combine adding new experiment with running 
  DAQs or uploads: dm-${DM_STATION_NAME}-daq and dm-${DM_STATION_NAME}-upload 
- Introduced sphinx as python API documentation framework
- Resolved possible timeout issue when starting DAQ or directory upload 
  with a directory containing large number of files
- Simplified data directory command line option for beamlines that use 
  gridftp (via DM_DATA_DIRECTORY_MAP environment variable)

Release 1.0.0 (01/31/2017)
=============================

- Introduced concept of experiment station and redesigned authorization 
  mechanisms to allow beamline managers to manage their stations; all
  APIs and CLIs now conform to the new authorization scheme
- Modified get-experiments utility to allow retrieving list of experiments
  for a given station
- Cleaned up web portal by removing unused views, and enabled station 
  management functionality
- GPFS DDN (extrepid) has replaced xstor as the main APS storage
- CLI changes:
  - add-experiment command requires station name (can be set from env.
    variable); experiment type can be specified using type name
  - get-experiments command requires station name for beamline managers (can 
    be set from env. variable)
  - start-experiment command is now optional

Release 0.9.6 (11/01/2016)
=============================

- Resolved issue with incorrect accounting of processing errors for DAQs
- Improved DAQ processing algorithm to avoid resource starvation between
  simultaneous DAQs and uploads
- Enhanced monitoring status information for both DAQs and uploads

Release 0.9.5 (10/14/2016)
=============================

- Introduced new framework and utilities for synchronizing users with 
  APS DB
- Resolved several issues with special characters in file names for
  gridftp transfer plugin  

Release 0.9.4 (05/27/2016)
=============================

- Added SFTP file system observer agent
- Enhanced MongoDB plugin with file md5 sum calculation

Release 0.9.3 (05/06/2016)
=============================

- Developed processing for HDF5 metadata in MongoDB cataloging plugin
- Modified catalog API and service interfaces to use file collections on 
  a per-experiment basis

Release 0.9.2 (04/29/2016)
=============================

- Resolved issue with upload command for directories containing large
  number of files
- Implemented enhanced upload processing algorithm to avoid resource 
  starvation between simultaneous DAQs and uploads
- Added new polling file system observer agent as option for monitoring 
  directories
- Reworked catalog API and corresponding MongoDB interfaces to use unique 
  experiment file paths, rather than file names

Release 0.9.1 (03/11/2016)
=============================

- Added dm-list-daqs and dm-list-uploads commands
- Resolved issue with newly created directories treated as files for 
  real-time data acquisitions

Release 0.9.0 (02/25/2016)
=============================

- Developed directory processing mode for uploads; in this mode file transfer 
  plugins transfer entire directories as opposed to individual files
- Added dm-get-processing-plugins command
- Resolved working directory issue that may occur with simutaneous uploads

Release 0.8.0 (01/26/2016)
=============================

- Enhanced upload/daq performance and functionality (hidden files are not
  processed; for uploads system can detect files that had been processed 
  already; improved handling and reporting of processing errors)
- Source file checksum is calculated for rsync/gridftp plugins by default
- Added dm-stop-upload command
- Resolved globus online user authorization delay issue 

Release 0.7.0 (12/08/2015)
=============================

- Introduced framework and user interfaces for tracking progress of file 
  uploads and data acquisitions in DAQ service
- Added ability to monitor multiple directories for the same experiment 
  simultaneously (required changes to DAQ service REST interfaces)
- Enhanced start/stop DAQ and upload commands to use DM_FILE_SERVER_URL
  environment variable
- Added user interfaces and utilities that enable experiment data download
  from machines that have SSH access to the storage host

Release 0.6.0 (11/06/2015)
=============================

- Added file system observer agent interface for DAQ service
- Implemented FTP file system observer for DAQ service
- Added interfaces for deleting user experiment role in DS service
- Introduced java REST API framework, and specific experiment DS service API 
- Web Portal notifies DS service about experiment user modifications 

Release 0.5.0 (10/08/2015)
=============================

- Implemented Single Sign-On solution for backend services
- Enabled user authentication via login file
- Added file stat (with checksum) interface in DS web service
- After adding user role to experiment via command line, user is also
  added to experiment group (if one exists)
- Added rsync file transfer plugin with checksum and delete

Release 0.4.0 (09/21/2015) 
=============================

- Number of minor modifications made in preparation for test deployment at
  beamlines

Release 0.3.0 (07/22/2015) 
=============================

- Developed initial version of Catalogging Web Service based on MongoDB
- Developed sample processing plugins: file metadata catalog, SDDS processing, 
  SGE job submission

Release 0.2.0 (06/30/2015) 
=============================

- Implemented storage permission management and user group management
- Developed common file processing service plugin framework

Release 0.1.0 (04/21/2015) 
=============================

- Functional web portal (user, experiment, and policy pages)
- Developed web service and its API/CLI frameworks
- Developed initial version of Data Storage Web Service
- Developed initial version of Data Acquisition Web Service;
- DAQ service can monitor file system on a detector node and subsequently
  transfer data to storage