RELEASE_NOTES.txt 17.9 KB
Newer Older
sveseli's avatar
sveseli committed
1
2
3
4
5
6
7
8
Release 3.0.1 (08/09/2019)
=============================
- Fixed issue with unchecked additions to metadata catalog
- Updated support software:
  - OpenJDK (replaces Oracle JDK)
  - Payara (replaces Glassfish)
- Added 12IDB utilities

sveseli's avatar
sveseli committed
9
10
11
12
13
14
15
16
17
Release 3.0.0 (07/26/2019)
=============================
- Added python web service API for downloading files 
- Updated DB schema for experiment data archival support
- Web Portal changes:
  - Added storage views
  - Added connection between station and experiment types
  - Updated experiment views with storage and root path fields

sveseli's avatar
sveseli committed
18
Release 2.6.0 (07/15/2019)
sveseli's avatar
sveseli committed
19
20
21
22
=============================
- Moved code repo to gitlab
- Updated release/upgrade utilities

sveseli's avatar
sveseli committed
23
24
25
26
27
Release 2.5.0 (06/23/2019)
=============================
- Added pprint (pretty-print) option to display format for all commands
- Added support for repeated commands via repeatPeriod/repeatUntil/maxRepeats
  keys in workflow definitions
sveseli's avatar
sveseli committed
28
- Added skip/limit options to list-processing-jobs 
sveseli's avatar
sveseli committed
29
30
31
32
33
34
35
36
37
38
- Implemented ability to stop processing jobs
  - New command:
    * stop-processing-job
- Enhanced support for processing multiple files with a given workflow
  - New command:
    * process-files
- Updated workflow API documentation
- DM Station GUI fixes/enhancements:
  - Fixed issue with modifying user list before saving experiment

sveseli's avatar
sveseli committed
39
Release 2.4.0 (05/20/2019)
sveseli's avatar
sveseli committed
40
41
42
43
=============================
- Added system LDAP utilities for verifying and creating groups
- Added support for retrieving single processing job stage via the --stage
  option for the get-processing-job command
sveseli's avatar
sveseli committed
44
45
- Prevented core files from being uploaded
- Prevented creation of experiment names with spaces
sveseli's avatar
sveseli committed
46
47
48
49
50
- Fixed file limit problem in compression utilities
- DM Station GUI fixes/enhancements:
  - Fixed issue with experiment user list modification
  - Added tab for monitoring processing jobs
  - Restored workflow tab 
sveseli's avatar
sveseli committed
51

sveseli's avatar
sveseli committed
52
Release 2.3.0 (04/15/2019)
sveseli's avatar
sveseli committed
53
=============================
sveseli's avatar
sveseli committed
54
- Updated CDB plugin for the latest CDB API changes
sveseli's avatar
sveseli committed
55
- Expanded support for HTML-formatted output
56
- Optimized amount of data transferred with file metadata listing
57
58
- Implemented limit on number of files that can be retrieved using
  a single list-experiment-files command or API call
sveseli's avatar
sveseli committed
59
- Added more diagnostic output to compression/decompression utilities
sveseli's avatar
sveseli committed
60
- DM Station GUI fixes/enhancements:
sveseli's avatar
sveseli committed
61
62
63
  - Added logging and ability to use DM_CONSOLE_LOG_LEVEL environment 
    variable to specify logging level
  - Resolved issue with undefined self key in exception hook
sveseli's avatar
sveseli committed
64
65
66
67
  - Added support for single file upload
  - Added support for experiment root path
  - Added warning dialog for experiment updates
  - Added number of tooltips 
sveseli's avatar
sveseli committed
68
69
70
71
72
73
  - Optimized handling of file metadata for experiments with large
    number of files
  - Added support for pagination while fetching full list of files
  - Implemented progress dialog when fetching a very large set of files
  - Resolved issue with filter boxes layout not matching data table 
    default column sizes
sveseli's avatar
sveseli committed
74

sveseli's avatar
sveseli committed
75
76
77
78
79
Release 2.2.2 (02/11/2019)
=============================
- Updated deployment scripts
- Fixed several Python 2/3 compatibility issues

sveseli's avatar
sveseli committed
80
81
82
83
84
85
Release 2.2.1 (02/04/2019)
=============================
- Updated API documentation
- DM Station GUI fixes/enhancements:
  - Added file compression capability

sveseli's avatar
sveseli committed
86
Release 2.2.0 (01/15/2019)
sveseli's avatar
sveseli committed
87
88
=============================
- Added ESAF interfaces to APS DB service 
sveseli's avatar
sveseli committed
89
90
91
- Moved all ESAF/BSS command line utilities to APS DB service 
  - New command:
    * list-beamlines 
sveseli's avatar
sveseli committed
92
93
- Converted software to Python 3, but retained Python 2 compatibility
- Introduced support for API pip and conda packages
sveseli's avatar
sveseli committed
94
95
- Added support for a single file upload via the --file-path option to
  upload commands
sveseli's avatar
sveseli committed
96
97
98
99
100
101
102
103
- Added support for arbitrary experiment path under the ${DM_STATION_NAME}
  directory via the --root-path option to the following commands:
    * add-experiment
    * update-experiment
    * ${DM_STATION_NAME}-daq
    * ${DM_STATION_NAME}-upload
  Note that if this option is not used, experiment will be located under
  the experiment type folder, as before. 
sveseli's avatar
sveseli committed
104

sveseli's avatar
sveseli committed
105
Release 2.1.0 (12/31/2018)
sveseli's avatar
sveseli committed
106
107
=============================
- Added estimated completion time to upload monitoring
sveseli's avatar
sveseli committed
108
109
110
111
112
- Introduced additional enhancements for cataloging plugin that shorten
  uploads of large datasets
- Developed initial image receiver service and utilities that can accept
  Area Detector NTNDArray frames and save them into files; currently
  supported formats are SDDS, JSON, HDF5, and JPEG.
sveseli's avatar
sveseli committed
113
114
115
  - New commands:
    * start-image-receiver
    * stop-image-receiver
sveseli's avatar
sveseli committed
116
- Added new APS DB service for accessing beamline scheduling system 
sveseli's avatar
sveseli committed
117

sveseli's avatar
sveseli committed
118
119
120
121
Release 2.0.0 (10/15/2018)
=============================
- Modified DM DB to support arbitrary experiment root paths

sveseli's avatar
sveseli committed
122
Release 1.9.3 (09/15/2018)
123
124
125
126
127
=============================
- Added generic script processing plugin for DAQ service
- Parallelized cataloging plugin; this significantly increased performance 
  for uploads using directory mode

sveseli's avatar
sveseli committed
128
Release 1.9.2 (05/15/2018)
sveseli's avatar
sveseli committed
129
=============================
sveseli's avatar
sveseli committed
130
- Added support for compressing and decompressing files
sveseli's avatar
sveseli committed
131
132
133
134
135
136
137
138
  - New commands:
    * compress-files
    * decompress-files
- Added support for deleting files
  - New commands:
    * delete-file
    * delete-files
- Modified stat-file utility, which now requires explicit --md5sum flag to
sveseli's avatar
sveseli committed
139
140
  retrieve checksum
- Added new --path-pattern option to the list-files command
sveseli's avatar
sveseli committed
141
142
143
- DM Station GUI fixes/enhancements:
  - Added initial GUI test suite
  - Added right-click copying functionality
sveseli's avatar
sveseli committed
144
  - Fixed issue with trailing comma for DAQ/upload parameters
sveseli's avatar
sveseli committed
145
146
147
148
  - Fixed CPU usage bug 
  - Fixed incorrect handling of user selections
  - Fixed focus policy for several buttons

sveseli's avatar
sveseli committed
149
Release 1.9.1 (03/30/2018)
sveseli's avatar
sveseli committed
150
151
=============================
- Added support for command output in html format
152
- Added support for requesting only subset of keys for list of processing jobs
sveseli's avatar
sveseli committed
153
154
- Resolved issues with symbolic links for uploading directories and files 
  via rsync 
155
156
- Fixed DAQ mode issue with double counting of observed files under certain
  circumstances
sveseli's avatar
sveseli committed
157
- Fixed problem with cataloging of HDF5 files that contain non-ascii metadata
sveseli's avatar
sveseli committed
158
159
160
- DM Station GUI fixes/enhancements:
  - Fixed issue with user permissions when creating new experiments
  - Fixed upload configuration issue 
161
  - Fixed caching issues for experiment file metadata and users
sveseli's avatar
sveseli committed
162
163
164
165
  - Added live metadata browsing while files load in background 
  - Added sorting for user table
  - Added ability to copy table information to clipboard
  - Improved selection behavior on tables
sveseli's avatar
sveseli committed
166
  
sveseli's avatar
sveseli committed
167
Release 1.9.0 (01/31/2018)
sveseli's avatar
sveseli committed
168
169
=============================
  
sveseli's avatar
sveseli committed
170
171
- Added support for retrieving file metadata for a given experiment in 
  batches; list-experiment-files get --skip and --limit options
sveseli's avatar
sveseli committed
172
173
174
175
176
177
178
179
- Simplified workflow and processing job management by allowing DM session user
  to be used as default workflow and/or processing job owner
  - Modified commands:
    * list-workflows
    * get-workflow
    * delete-workflow
    * start-processing-job
    * list-processing-jobs
sveseli's avatar
sveseli committed
180
181
- DM Station GUI fixes/enhancements:
  - Added support for DM_DATA_DIRECTORY_MAP environment variable; this allows
sveseli's avatar
sveseli committed
182
    browsing data locally, while moving data using GridFTP third
sveseli's avatar
sveseli committed
183
    party transfers
sveseli's avatar
sveseli committed
184
  - Enhanced support for viewing large file metadata collections
sveseli's avatar
sveseli committed
185

sveseli's avatar
sveseli committed
186
Release 1.8.0 (12/15/2017)
sveseli's avatar
sveseli committed
187
188
=============================
  
sveseli's avatar
sveseli committed
189
190
191
- Enhanced algorithm for processing of existing files; this resolved issues
  with multiple simultaneous DAQs involving large number of pre-existing
  files that need to be processed
192
193
194
- Added support for retrieving experiment dataset statistics:
  - New command:
    * get-file-collection-stats
sveseli's avatar
sveseli committed
195
196
197
198
199
- Added support for output variables in workflow definitions
- Added generic HDF5 file metadata reader
- Developed DAQ/PROC service integration via new DAQ plugin; this enables
  development and deployment of fully automated data acquisition/processing 
  pipelines based on user-defined workflows
200
201
202
203
204
205
  - Modified commands (all get --workflow-name, --workflow-owner, 
    --workflow-job-owner and --workflow-args options):
    * start-daq
    * upload
    * ${DM_STATION_NAME}-daq
    * ${DM_STATION_NAME}-upload
sveseli's avatar
sveseli committed
206
207
208
209
210
211
- Added HTTPS support for accessing file metadata catalog via Mongo Express
- DM Station GUI fixes/enhancements:
  - Added support for viewing experiment file metadata and file collection
    statistics
  - Added support for DM_BEAMLINE_MANAGERS environment variable
  - Improved dialogs for DAQ/upload configuration options
sveseli's avatar
sveseli committed
212

sveseli's avatar
sveseli committed
213
Release 1.7.0 (11/01/2017)
214
215
216
217
=============================

- DM Station GUI fixes/enhancements:
  - Added integration with ESAF DB
sveseli's avatar
sveseli committed
218
219
- Added new output formating option "--display-format=key-per-line" that is
  common to all commands 
sveseli's avatar
sveseli committed
220
221
- Introduced service monitoring infrastructure, which enabled deployment of
  Nagios-based system monitoring
sveseli's avatar
sveseli committed
222
223
224
  - New commands:
    * get-service-status 
    * clear-service-status 
225

sveseli's avatar
sveseli committed
226
Release 1.6.0 (10/07/2017)
sveseli's avatar
sveseli committed
227
228
=============================

sveseli's avatar
sveseli committed
229
230
231
232
233
234
235
236
237
238
239
- Added integration with APS ESAF DB via new APIs and CLIs
  - New commands:
    * list-esafs: list sector ESAFs by year
    * get-esafs: retrieve ESAF by id
  - Modified commands (new ESAF id option for setting experiment users)
    * ${DM_STATION_NAME}-daq 
    * ${DM_STATION_NAME}-upload
- Enhanced DAQ processing framework with introduction of meta
  classes for plugins and supporting utilities; this enables using
  different processing chain depending on data location, or other 
  distinguishing criteria
sveseli's avatar
sveseli committed
240
241
242
243
244
245
- DM Station GUI fixes/enhancements:
  - Added timer for automatic refresh of DAQ/upload status screens
  - Split configuration options for DAQs/uploads
  - Fixed issue with duplicate proposal users
  - Used current date as default for experiment start/end dates

sveseli's avatar
sveseli committed
246
Release 1.5.0 (08/04/2017)
sveseli's avatar
sveseli committed
247
248
249
=============================

- Added DS interface and Java API for downloading files 
sveseli's avatar
sveseli committed
250
251
252
253
- Added APIs and CLIs to clear (force service to forget) DAQs and uploads
  - New commands:
    * clear-daq
    * clear-upload
sveseli's avatar
sveseli committed
254
255
256
- Introduced APSU-related enhancements:
  - Processing for SDDS metadata in MongoDB cataloging plugin
  - Component Database processing plugin 
sveseli's avatar
sveseli committed
257
- DM Station GUI fixes/enhancements:
sveseli's avatar
sveseli committed
258
  - Fixed issue with DM_BEAMLINE_NAME not set
sveseli's avatar
sveseli committed
259
260
  - Fixed issues with buttons that require previous item selection
  - Added start date to list of DAQs and uploads
sveseli's avatar
sveseli committed
261
  - Added ability to remove completed DAQs or uploads from the top-level list
sveseli's avatar
sveseli committed
262

sveseli's avatar
sveseli committed
263
Release 1.4.0 (07/20/2017)
sveseli's avatar
sveseli committed
264
265
266
267
=============================

- Introduced DM Station GUI (accessed via dm-station-gui command)
- Introduced automated system test framework for DM stations
sveseli's avatar
sveseli committed
268
269
- Added --process-existing option to start-daq and <station>-daq
  commands; this will cause upload of existing files when starting DAQs
sveseli's avatar
sveseli committed
270

sveseli's avatar
sveseli committed
271
Release 1.3.0 (05/10/2017)
sveseli's avatar
sveseli committed
272
273
274
=============================

- Added APIs and CLIs to update experiment attributes and metadata
275
  catalog, as well as to delete experiments
sveseli's avatar
sveseli committed
276
277
278
279
  - New commands:
    * update-experiment
    * update-experiment-files
    * get-async-update-status
280
    * delete-experiment
sveseli's avatar
sveseli committed
281

sveseli's avatar
sveseli committed
282
Release 1.2.0 (04/18/2017)
283
284
285
286
=============================

- Modified scheduling algorithm for DAQs/uploads to simplify status monitoring

sveseli's avatar
sveseli committed
287
Release 1.1.0 (03/10/2017)
288
289
=============================

sveseli's avatar
sveseli committed
290
291
- Introduced integration with Beamline Scheduling System:
  - New commands: 
sveseli's avatar
sveseli committed
292
293
294
295
296
297
298
299
300
301
302
303
304
305
    * list-runs
    * list-proposals
    * get-proposal
  - Added the following options to the add-experiment command:
    --proposal-id: automatically add to experiment all 
      users associated with a given beamline proposal 
    --run: look for beamline proposal in a given run (current run is the 
      default)
    --users: comma specified list of usernames to be added to experiment
      as users
- Added the following options for managing DAQs:
  --duration: DAQ will be stopped automatically after given
    number of days or hours
  --dest-directory: files will be uploaded into
306
    a specific directory relative to experiment root path
sveseli's avatar
sveseli committed
307
308
309
310
311
312
  --upload-data-directory-on-exit: when DAQ finishes, upload of the given 
    data directory will be executed automatically
  --upload-dest-directory-on-exit: specifies destination directory for
    upload after DAQ completes
- Added the following options for managing uploads:
  --dest-directory: files will be uploaded into
313
    a specific directory relative to experiment root path
sveseli's avatar
sveseli committed
314
315
316
- Introduced framework for beamline-specific tools; added 
  beamline-specific commands that combine adding new experiment with running 
  DAQs or uploads: dm-${DM_STATION_NAME}-daq and dm-${DM_STATION_NAME}-upload 
317
- Introduced sphinx as python API documentation framework
sveseli's avatar
sveseli committed
318
319
320
321
- Resolved possible timeout issue when starting DAQ or directory upload 
  with a directory containing large number of files
- Simplified data directory command line option for beamlines that use 
  gridftp (via DM_DATA_DIRECTORY_MAP environment variable)
322

sveseli's avatar
sveseli committed
323
Release 1.0.0 (01/31/2017)
sveseli's avatar
sveseli committed
324
325
326
327
328
329
330
=============================

- Introduced concept of experiment station and redesigned authorization 
  mechanisms to allow beamline managers to manage their stations; all
  APIs and CLIs now conform to the new authorization scheme
- Modified get-experiments utility to allow retrieving list of experiments
  for a given station
sveseli's avatar
sveseli committed
331
332
- Cleaned up web portal by removing unused views, and enabled station 
  management functionality
sveseli's avatar
sveseli committed
333
- GPFS DDN (extrepid) has replaced xstor as the main APS storage
sveseli's avatar
sveseli committed
334
335
336
337
338
339
- CLI changes:
  - add-experiment command requires station name (can be set from env.
    variable); experiment type can be specified using type name
  - get-experiments command requires station name for beamline managers (can 
    be set from env. variable)
  - start-experiment command is now optional
sveseli's avatar
sveseli committed
340

sveseli's avatar
sveseli committed
341
Release 0.9.6 (11/01/2016)
sveseli's avatar
sveseli committed
342
343
344
=============================

- Resolved issue with incorrect accounting of processing errors for DAQs
sveseli's avatar
sveseli committed
345
- Improved DAQ processing algorithm to avoid resource starvation between
346
  simultaneous DAQs and uploads
sveseli's avatar
sveseli committed
347
- Enhanced monitoring status information for both DAQs and uploads
sveseli's avatar
sveseli committed
348

sveseli's avatar
sveseli committed
349
Release 0.9.5 (10/14/2016)
sveseli's avatar
sveseli committed
350
351
352
353
354
355
356
=============================

- Introduced new framework and utilities for synchronizing users with 
  APS DB
- Resolved several issues with special characters in file names for
  gridftp transfer plugin  

sveseli's avatar
sveseli committed
357
Release 0.9.4 (05/27/2016)
sveseli's avatar
sveseli committed
358
359
360
=============================

- Added SFTP file system observer agent
sveseli's avatar
sveseli committed
361
- Enhanced MongoDB plugin with file md5 sum calculation
sveseli's avatar
sveseli committed
362

sveseli's avatar
sveseli committed
363
Release 0.9.3 (05/06/2016)
sveseli's avatar
sveseli committed
364
365
=============================

sveseli's avatar
sveseli committed
366
- Developed processing for HDF5 metadata in MongoDB cataloging plugin
sveseli's avatar
sveseli committed
367
368
369
- Modified catalog API and service interfaces to use file collections on 
  a per-experiment basis

sveseli's avatar
sveseli committed
370
Release 0.9.2 (04/29/2016)
371
372
373
374
375
376
377
378
=============================

- Resolved issue with upload command for directories containing large
  number of files
- Implemented enhanced upload processing algorithm to avoid resource 
  starvation between simultaneous DAQs and uploads
- Added new polling file system observer agent as option for monitoring 
  directories
sveseli's avatar
sveseli committed
379
380
- Reworked catalog API and corresponding MongoDB interfaces to use unique 
  experiment file paths, rather than file names
381

sveseli's avatar
sveseli committed
382
Release 0.9.1 (03/11/2016)
383
384
385
=============================

- Added dm-list-daqs and dm-list-uploads commands
386
387
- Resolved issue with newly created directories treated as files for 
  real-time data acquisitions
388

sveseli's avatar
sveseli committed
389
Release 0.9.0 (02/25/2016)
390
391
392
393
394
395
396
=============================

- Developed directory processing mode for uploads; in this mode file transfer 
  plugins transfer entire directories as opposed to individual files
- Added dm-get-processing-plugins command
- Resolved working directory issue that may occur with simutaneous uploads

sveseli's avatar
sveseli committed
397
Release 0.8.0 (01/26/2016)
sveseli's avatar
sveseli committed
398
=============================
sveseli's avatar
sveseli committed
399

sveseli's avatar
sveseli committed
400
- Enhanced upload/daq performance and functionality (hidden files are not
sveseli's avatar
sveseli committed
401
  processed; for uploads system can detect files that had been processed 
402
  already; improved handling and reporting of processing errors)
sveseli's avatar
sveseli committed
403
- Source file checksum is calculated for rsync/gridftp plugins by default
404
- Added dm-stop-upload command
405
- Resolved globus online user authorization delay issue 
sveseli's avatar
sveseli committed
406

sveseli's avatar
sveseli committed
407
Release 0.7.0 (12/08/2015)
408
409
=============================

sveseli's avatar
sveseli committed
410
411
412
413
- Introduced framework and user interfaces for tracking progress of file 
  uploads and data acquisitions in DAQ service
- Added ability to monitor multiple directories for the same experiment 
  simultaneously (required changes to DAQ service REST interfaces)
sveseli's avatar
sveseli committed
414
415
- Enhanced start/stop DAQ and upload commands to use DM_FILE_SERVER_URL
  environment variable
416
417
- Added user interfaces and utilities that enable experiment data download
  from machines that have SSH access to the storage host
418

sveseli's avatar
sveseli committed
419
Release 0.6.0 (11/06/2015)
420
421
422
=============================

- Added file system observer agent interface for DAQ service
sveseli's avatar
sveseli committed
423
424
425
426
- Implemented FTP file system observer for DAQ service
- Added interfaces for deleting user experiment role in DS service
- Introduced java REST API framework, and specific experiment DS service API 
- Web Portal notifies DS service about experiment user modifications 
427

sveseli's avatar
sveseli committed
428
Release 0.5.0 (10/08/2015)
sveseli's avatar
sveseli committed
429
430
=============================

sveseli's avatar
sveseli committed
431
432
433
434
435
- Implemented Single Sign-On solution for backend services
- Enabled user authentication via login file
- Added file stat (with checksum) interface in DS web service
- After adding user role to experiment via command line, user is also
  added to experiment group (if one exists)
sveseli's avatar
sveseli committed
436
- Added rsync file transfer plugin with checksum and delete
sveseli's avatar
sveseli committed
437

sveseli's avatar
sveseli committed
438
Release 0.4.0 (09/21/2015) 
sveseli's avatar
sveseli committed
439
440
=============================

sveseli's avatar
sveseli committed
441
- Number of minor modifications made in preparation for test deployment at
sveseli's avatar
sveseli committed
442
443
  beamlines

sveseli's avatar
sveseli committed
444
Release 0.3.0 (07/22/2015) 
sveseli's avatar
sveseli committed
445
446
=============================

sveseli's avatar
sveseli committed
447
448
449
- Developed initial version of Catalogging Web Service based on MongoDB
- Developed sample processing plugins: file metadata catalog, SDDS processing, 
  SGE job submission
sveseli's avatar
sveseli committed
450

sveseli's avatar
sveseli committed
451
Release 0.2.0 (06/30/2015) 
sveseli's avatar
sveseli committed
452
453
454
455
456
=============================

- Implemented storage permission management and user group management
- Developed common file processing service plugin framework

sveseli's avatar
sveseli committed
457
Release 0.1.0 (04/21/2015) 
sveseli's avatar
sveseli committed
458
459
460
461
462
463
464
465
=============================

- Functional web portal (user, experiment, and policy pages)
- Developed web service and its API/CLI frameworks
- Developed initial version of Data Storage Web Service
- Developed initial version of Data Acquisition Web Service;
- DAQ service can monitor file system on a detector node and subsequently
  transfer data to storage
sveseli's avatar
sveseli committed
466