Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found

Target

Select target project
  • DM/dm-docs
  • hammonds/dm-docs
  • hparraga/dm-docs
3 results
Show changes
Commits on Source (130)
30 additional commits have been omitted to prevent performance issues.
Hide whitespace changes
Inline Side-by-side
Showing
with 186 additions and 1454 deletions
ApsComputingAndDataManagamentInfrastructure-20200210.pdf 0 → 100644
View file @ ae1437b1
File added
ApsDataManagementSystem-2019.12.11.pdf 0 → 100644
View file @ ae1437b1
File added
Makefile deleted 100644 → 0
View file @ e561021c
TOP = ..
SUBDIRS = sphinx
include $(TOP)/tools/make/RULES_DM
RELEASE_NOTES.txt
View file @ ae1437b1
Release 2.2.0 (01/31/2019)
Release 4.0.0 (XX/YY/2021)
=============================
- Fixed issue with non-ascii characters for daq/upload CLI tools
- Fixed issue with locations that involve host/port for SFTP observer
- Fixed issue with incorrect DAQ_DIRECTORY_MAP replacements
- Added globus_group_id to the experiment table in the DM DB, and
modified CLIs, DB and web service APIs accordingly
- Implemented API and CLI support for Globus group management:
- New commands:
* get-globus-group
* list-globus-groups
* create-globus-group
* add-globus-group-memebers
* delete-globus-group-members
* delete-globus-group
Release 3.3.2 (07/09/2020)
=============================
- Added ability to upload files from remote machines via rsync plugin and using
ssh
- Added start/end time into GUP proposal object
Release 3.3.1 (05/06/2020)
=============================
- Maintenance release
Release 3.3.0 (05/04/2020)
=============================
- Change log4j version from 1.2.7 to 2.13.2. Code changes to adapt were
necessary. Changed config file from log4j.properties to log4j2.xml
- Change jar file for postgresql from 9.3-1102.jdbc41 to 42.2.8.
The underlying postgresql in support changed from 9.3 to 9.6 a while ago.
Release 3.2.2 (02/24/2020)
=============================
- Added user documentation (Getting Started guide)
Release 3.2.1 (01/15/2020)
=============================
- DM GUI Changes:
- Reworked configuration options for DAQs and uploads; those now support
including/excluding file extensions
- Fixed bug that was preventing using configuration options and additional
keywords at the same time
Release 3.2.0 (11/05/2019)
=============================
- Implemented ability to include or exclude file extensions for uploads and DAQs
- New API Options for starting DAQ/Upload
* includeFileExtensions
* excludeFileExtensions
- New CLI Options for starting DAQ/Upload
* include-extensions
* exclude-extensions
Release 3.1.0 (10/25/2019)
=============================
- Added new ESAF interfaces for support of extended data retrieval
- Enabled URL scheme for transfer plugin configuration; this allows
multiple gridftp hosts per deployment
- Updated support software to allow offsite installation
- DM GUI changes:
- Add settings page
- Allow user to specify refresh rates per module.
- Fixed protocol parsing bug
Release 3.0.1 (08/09/2019)
=============================
- Fixed issue with unchecked additions to metadata catalog
- Updated support software:
- OpenJDK (replaces Oracle JDK)
- Payara (replaces Glassfish)
- Added 12IDB utilities
Release 3.0.0 (07/26/2019)
=============================
- Added python web service API for downloading files
- Updated DB schema for experiment data archival support
- Web Portal changes:
- Added storage views
- Added connection between station and experiment types
- Updated experiment views with storage and root path fields
Release 2.6.0 (07/15/2019)
=============================
- Moved code repo to gitlab
- Updated release/upgrade utilities
Release 2.5.0 (06/23/2019)
=============================
- Added pprint (pretty-print) option to display format for all commands
- Added support for repeated commands via repeatPeriod/repeatUntil/maxRepeats
keys in workflow definitions
- Added skip/limit options to list-processing-jobs
- Implemented ability to stop processing jobs
- New command:
* stop-processing-job
- Enhanced support for processing multiple files with a given workflow
- New command:
* process-files
- Updated workflow API documentation
- DM Station GUI fixes/enhancements:
- Fixed issue with modifying user list before saving experiment
Release 2.4.0 (05/20/2019)
=============================
- Added system LDAP utilities for verifying and creating groups
- Added support for retrieving single processing job stage via the --stage
option for the get-processing-job command
- Prevented core files from being uploaded
- Prevented creation of experiment names with spaces
- Fixed file limit problem in compression utilities
- DM Station GUI fixes/enhancements:
- Fixed issue with experiment user list modification
- Added tab for monitoring processing jobs
- Restored workflow tab
Release 2.3.0 (04/15/2019)
=============================
- Updated CDB plugin for the latest CDB API changes
- Expanded support for HTML-formatted output
- Optimized amount of data transferred with file metadata listing
- Implemented limit on number of files that can be retrieved using
a single list-experiment-files command or API call
- Added more diagnostic output to compression/decompression utilities
- DM Station GUI fixes/enhancements:
- Added logging and ability to use DM_CONSOLE_LOG_LEVEL environment
variable to specify logging level
- Resolved issue with undefined self key in exception hook
- Added support for single file upload
- Added support for experiment root path
- Added warning dialog for experiment updates
- Added number of tooltips
- Optimized handling of file metadata for experiments with large
number of files
- Added support for pagination while fetching full list of files
- Implemented progress dialog when fetching a very large set of files
- Resolved issue with filter boxes layout not matching data table
default column sizes
Release 2.2.2 (02/11/2019)
=============================
- Updated deployment scripts
- Fixed several Python 2/3 compatibility issues
Release 2.2.1 (02/04/2019)
=============================
- Updated API documentation
- DM Station GUI fixes/enhancements:
- Added file compression capability
Release 2.2.0 (01/15/2019)
=============================
- Added ESAF interfaces to APS DB service
- Moved all ESAF/BSS command line utilities to APS DB service
......@@ -34,13 +185,13 @@ Release 2.0.0 (10/15/2018)
=============================
- Modified DM DB to support arbitrary experiment root paths
Release 1.12 (09/15/2018)
Release 1.9.3 (09/15/2018)
=============================
- Added generic script processing plugin for DAQ service
- Parallelized cataloging plugin; this significantly increased performance
for uploads using directory mode
Release 1.11 (05/15/2018)
Release 1.9.2 (05/15/2018)
=============================
- Added support for compressing and decompressing files
- New commands:
......@@ -61,7 +212,7 @@ Release 1.11 (05/15/2018)
- Fixed incorrect handling of user selections
- Fixed focus policy for several buttons
Release 1.10 (03/30/2018)
Release 1.9.1 (03/30/2018)
=============================
- Added support for command output in html format
- Added support for requesting only subset of keys for list of processing jobs
......@@ -79,7 +230,7 @@ Release 1.10 (03/30/2018)
- Added ability to copy table information to clipboard
- Improved selection behavior on tables
Release 1.9 (01/31/2018)
Release 1.9.0 (01/31/2018)
=============================
- Added support for retrieving file metadata for a given experiment in
......@@ -98,7 +249,7 @@ Release 1.9 (01/31/2018)
party transfers
- Enhanced support for viewing large file metadata collections
Release 1.8 (12/15/2017)
Release 1.8.0 (12/15/2017)
=============================
- Enhanced algorithm for processing of existing files; this resolved issues
......@@ -125,7 +276,7 @@ Release 1.8 (12/15/2017)
- Added support for DM_BEAMLINE_MANAGERS environment variable
- Improved dialogs for DAQ/upload configuration options
Release 1.7 (11/01/2017)
Release 1.7.0 (11/01/2017)
=============================
- DM Station GUI fixes/enhancements:
......@@ -138,7 +289,7 @@ Release 1.7 (11/01/2017)
* get-service-status
* clear-service-status
Release 1.6 (10/07/2017)
Release 1.6.0 (10/07/2017)
=============================
- Added integration with APS ESAF DB via new APIs and CLIs
......@@ -158,7 +309,7 @@ Release 1.6 (10/07/2017)
- Fixed issue with duplicate proposal users
- Used current date as default for experiment start/end dates
Release 1.5 (08/04/2017)
Release 1.5.0 (08/04/2017)
=============================
- Added DS interface and Java API for downloading files
......@@ -175,7 +326,7 @@ Release 1.5 (08/04/2017)
- Added start date to list of DAQs and uploads
- Added ability to remove completed DAQs or uploads from the top-level list
Release 1.4 (07/20/2017)
Release 1.4.0 (07/20/2017)
=============================
- Introduced DM Station GUI (accessed via dm-station-gui command)
......@@ -183,7 +334,7 @@ Release 1.4 (07/20/2017)
- Added --process-existing option to start-daq and <station>-daq
commands; this will cause upload of existing files when starting DAQs
Release 1.3 (05/10/2017)
Release 1.3.0 (05/10/2017)
=============================
- Added APIs and CLIs to update experiment attributes and metadata
......@@ -194,12 +345,12 @@ Release 1.3 (05/10/2017)
* get-async-update-status
* delete-experiment
Release 1.2 (04/18/2017)
Release 1.2.0 (04/18/2017)
=============================
- Modified scheduling algorithm for DAQs/uploads to simplify status monitoring
Release 1.1 (03/10/2017)
Release 1.1.0 (03/10/2017)
=============================
- Introduced integration with Beamline Scheduling System:
......@@ -235,7 +386,7 @@ Release 1.1 (03/10/2017)
- Simplified data directory command line option for beamlines that use
gridftp (via DM_DATA_DIRECTORY_MAP environment variable)
Release 1.0 (01/31/2017)
Release 1.0.0 (01/31/2017)
=============================
- Introduced concept of experiment station and redesigned authorization
......@@ -253,7 +404,7 @@ Release 1.0 (01/31/2017)
be set from env. variable)
- start-experiment command is now optional
Release 0.15 (11/01/2016)
Release 0.9.6 (11/01/2016)
=============================
- Resolved issue with incorrect accounting of processing errors for DAQs
......@@ -261,7 +412,7 @@ Release 0.15 (11/01/2016)
simultaneous DAQs and uploads
- Enhanced monitoring status information for both DAQs and uploads
Release 0.14 (10/14/2016)
Release 0.9.5 (10/14/2016)
=============================
- Introduced new framework and utilities for synchronizing users with
......@@ -269,20 +420,20 @@ Release 0.14 (10/14/2016)
- Resolved several issues with special characters in file names for
gridftp transfer plugin
Release 0.13 (05/27/2016)
Release 0.9.4 (05/27/2016)
=============================
- Added SFTP file system observer agent
- Enhanced MongoDB plugin with file md5 sum calculation
Release 0.12 (05/06/2016)
Release 0.9.3 (05/06/2016)
=============================
- Developed processing for HDF5 metadata in MongoDB cataloging plugin
- Modified catalog API and service interfaces to use file collections on
a per-experiment basis
Release 0.11 (04/29/2016)
Release 0.9.2 (04/29/2016)
=============================
- Resolved issue with upload command for directories containing large
......@@ -294,14 +445,14 @@ Release 0.11 (04/29/2016)
- Reworked catalog API and corresponding MongoDB interfaces to use unique
experiment file paths, rather than file names
Release 0.10 (03/11/2016)
Release 0.9.1 (03/11/2016)
=============================
- Added dm-list-daqs and dm-list-uploads commands
- Resolved issue with newly created directories treated as files for
real-time data acquisitions
Release 0.9 (02/25/2016)
Release 0.9.0 (02/25/2016)
=============================
- Developed directory processing mode for uploads; in this mode file transfer
......@@ -309,7 +460,7 @@ Release 0.9 (02/25/2016)
- Added dm-get-processing-plugins command
- Resolved working directory issue that may occur with simutaneous uploads
Release 0.8 (01/26/2016)
Release 0.8.0 (01/26/2016)
=============================
- Enhanced upload/daq performance and functionality (hidden files are not
......@@ -319,7 +470,7 @@ Release 0.8 (01/26/2016)
- Added dm-stop-upload command
- Resolved globus online user authorization delay issue
Release 0.7 (12/08/2015)
Release 0.7.0 (12/08/2015)
=============================
- Introduced framework and user interfaces for tracking progress of file
......@@ -331,7 +482,7 @@ Release 0.7 (12/08/2015)
- Added user interfaces and utilities that enable experiment data download
from machines that have SSH access to the storage host
Release 0.6 (11/06/2015)
Release 0.6.0 (11/06/2015)
=============================
- Added file system observer agent interface for DAQ service
......@@ -340,7 +491,7 @@ Release 0.6 (11/06/2015)
- Introduced java REST API framework, and specific experiment DS service API
- Web Portal notifies DS service about experiment user modifications
Release 0.5 (10/08/2015)
Release 0.5.0 (10/08/2015)
=============================
- Implemented Single Sign-On solution for backend services
......@@ -350,26 +501,26 @@ Release 0.5 (10/08/2015)
added to experiment group (if one exists)
- Added rsync file transfer plugin with checksum and delete
Release 0.4 (09/21/2015)
Release 0.4.0 (09/21/2015)
=============================
- Number of minor modifications made in preparation for test deployment at
beamlines
Release 0.3 (07/22/2015)
Release 0.3.0 (07/22/2015)
=============================
- Developed initial version of Catalogging Web Service based on MongoDB
- Developed sample processing plugins: file metadata catalog, SDDS processing,
SGE job submission
Release 0.2 (06/30/2015)
Release 0.2.0 (06/30/2015)
=============================
- Implemented storage permission management and user group management
- Developed common file processing service plugin framework
Release 0.1 (04/21/2015)
Release 0.1.0 (04/21/2015)
=============================
- Functional web portal (user, experiment, and policy pages)
......
beamline_names.txt
View file @ ae1437b1
......@@ -59,6 +59,8 @@
"24-ID-E"
"26-ID-C"
"27-ID-B"
"28-ID-B"
"28-ID-C"
"29-ID-C,D"
"30-ID-B,C"
"31-ID-D"
......
demo/apsu-20150709/demo_notes.sv.txt deleted 100644 → 0
View file @ e561021c
# Need 1 terminal on DAQ node, 1 terminal on HPC node and 2 terminals on DS
# node
#
# SCENARIO 1: BASIC UPLOAD
#
# ssh -X dm@dmstorage: start services
cd /opt/DM/dev
source setup.sh
./etc/init.d/dm-postgresql start
./etc/init.d/dm-glassfish start
./etc/init.d/dm-mongodb start
./etc/init.d/dm-ds-web-service start
./etc/init.d/dm-cat-web-service start
# ssh -X dm@dmdaq: start services
cd /opt/DM/dev
source setup.sh
./etc/init.d/dm-daq-web-service start
# ssh -X dm@dmhpc: start services, check NFS
cd /opt/DM/dev
source setup.sh
./etc/init.d/dm-daq-web-service start
ls -l /net/dmstorage/opt/DM
#####################################################################
# dm@dmstorage: check directory content on the storage node
ls -l /opt/DM/data
# ssh sveseli@dmstorage: add and start experiment e1
source /opt/DM/etc/dm.setup.sh
dm-add-experiment --name e1 --type-id 1
dm-start-experiment --name e1
# dm@dmstorage: check directory content on the storage node
# note that experiment directory permissions are restricted
ls -l /opt/DM/data/ESAF
ls -l /opt/DM/data/ESAF/e1/
# ssh dm@dmdaq: source setup file, show test data
source /opt/DM/etc/dm.setup.sh
ls -lR /opt/DM/experiments/e1
cat /opt/DM/experiments/e1/file1
# dm@dmdaq: upload data for experiment e1
dm-upload --experiment e1 --data-directory /opt/DM/experiments/e1
# dm@dmstorage: check experiment storage directory content
# note permissions, ownership
ls -l /opt/DM/data/ESAF/e1/
ls -l /opt/DM/data/ESAF/e1/2015/07/09/
#
# SCENARIO 2: UPLOAD + METADATA CATALOG
#
# sveseli@dmstorage: get metadata for experiment files from catalogging service
dm-get-experiment-files --experiment e1
dm-get-experiment-file --experiment e1 --file file2 --display-keys=__all__
# dm@dmdaq: upload data for experiment e1, this time specify extra keys
dm-upload --experiment e1 --data-directory /opt/DM/experiments/e1 ownerUser:JohnC ownerGroup:APSU memo1:ApprovedByNDA memo2:DislikedByGD
# sveseli@dmstorage: get metadata for file 2 again
dm-get-experiment-file --experiment e1 --file file2 --display-keys=__all__
# sveseli@dmstorage: show metadata updates
dm-update-experiment-file --experiment e1 --file file3 quality:A --display-keys=id,fileName,quality
# sveseli@dmstorage: show metadata search
dm-get-experiment-files --experiment e1 quality:A
dm-get-experiment-files --experiment e1 storageFilePath:2015
#
# SCENARIO 3: UPLOAD + METADATA CATALOG + SDDS PARAMETERS
#
# sveseli@dmstorage: add and start experiment mm1
dm-add-experiment --name mm1 --type-id 1
dm-start-experiment --name mm1
# dm@dmdaq: upload data for experiment mm1, and request SDDS parameter
# processing
ls -lR /opt/DM/experiments/mm1
dm-upload --experiment mm1 --data-directory /opt/DM/experiments/mm1 ownerUser:JohnC ownerGroup:APSU processSddsParameters:True
# sveseli@dmstorage: get mm1 files, observe SDDS parameters
dm-get-experiment-files --experiment mm1
dm-get-experiment-files --experiment mm1 --display-keys=__all__ --display-format=dict
# dm@dmstorage: compare with sddsprintout (permissions do not allow sveseli
# account to access file)
export PATH=$PATH:/opt/epics/extensions/bin/linux-x86_64/
sddsprintout -parameters /opt/DM/data/ESAF/mm1/hallProbeScan-M1Proto-000000072-0009-000000.edf
#
# SCENARIO 4: UPLOAD + METADATA CATALOG + SDDS PARAMETERS + SCRIPT PROCESSING
#
# dm@dmstorage: show processing script
cat /opt/DM/processing/find_sdds_row_count.sh
/opt/DM/processing/find_sdds_row_count.sh /opt/DM/data/ESAF/mm1/hallProbeScan-M1Proto-000000072-0009-000000.edf
# sveseli@dmstorage: get mm1 files, note no key processingScriptOutput
dm-get-experiment-files --experiment mm1 --display-keys=fileName,processingScriptOutput
# dm@dmdaq: upload data for experiment mm1, request SDDS parameter
# processing, specify processing script
dm-upload --experiment mm1 --data-directory /opt/DM/experiments/mm1 processSddsParameters:True processingScript:/opt/DM/processing/find_sdds_row_count.sh
# sveseli@dmstorage: get mm1 files, note present key processingScriptOutput
dm-get-experiment-files --experiment mm1 --display-keys=fileName,processingScriptOutput
#
# SCENARIO 5: UPLOAD + METADATA CATALOG + SDDS PARAMETERS + HPC PROCESSING
#
# dm@dmstorage: show processing script
more /opt/DM/processing/sge_sdds_analysis.sh
# dm@dmstorage: show no png files in experiment directory
ls -l /opt/DM/data/ESAF/mm1/*.png
# dm@dmhpc: show empty home directory
cd
ls -l
# dm@dmhpc: show qstat
source /opt/sge/default/common/settings.sh
qstat -f
watch -d 'qstat -f'
# sveseli@dmstorage: get mm1 files, note only 1 file
dm-get-experiment-files --experiment mm1
# dm@dmdaq: upload data for experiment mm1, request SDDS parameter
# processing, specify SGE processing script
dm-upload --experiment mm1 --data-directory /opt/DM/experiments/mm1 processSddsParameters:True sgeJobScript:/opt/DM/processing/sge_sdds_analysis.sh
# sveseli@dmstorage: get mm1 files, note 2 files
dm-get-experiment-files --experiment mm1
# sveseli@dmstorage: get mm1 .png files, note parentFile key
dm-get-experiment-files --experiment mm1 fileName:.png --display-keys=__all__
# dm@dmhpc: show SGE output in home directory
ls -l
# dm@dmstorage: open processed file
xdg-open /opt/DM/data/ESAF/mm1/hallProbeScan-M1Proto-000000072-0009-000000.edf.png
#
# SCENARIO 6: DAQ + METADATA CATALOG + SDDS PARAMETERS + HPC PROCESSING
#
# sveseli@dmstorage: add and start experiment mm2
dm-add-experiment --name mm2 --type-id 1
dm-start-experiment --name mm2
# sveseli@dmstorage: get mm2 files, note no files
dm-get-experiment-files --experiment mm2
# dm@dmstorage: show no png files in experiment directory
ls -l /opt/DM/data/ESAF/mm2/*.png
# dm@dmstorage: tail log file to observe processing
tail -f /opt/DM/var/log/dm.ds-web-service.log
# dm@dmdaq: start DAQ for experiment mm2, request SDDS parameter
# processing, specify SGE processing script
rm -rf /tmp/data/mm2
mkdir -p /tmp/data/mm2
dm-start-daq --experiment mm2 --data-directory /tmp/data/mm2 processSddsParameters:True sgeJobScript:/opt/DM/processing/sge_sdds_analysis.sh
# dm@dmhpc: show qstat
watch -d 'qstat -f'
# dm@dmdaq: copy experiment mm2 files into observed directory, watch qstat
ls -l /opt/DM/experiments/mm2/
cp /opt/DM/experiments/mm2/* /tmp/data/mm2/ && sleep 5 && touch /tmp/data/mm2/* &
tail -f /opt/DM/var/log/dm.daq-web-service.log
# sveseli@dmstorage: get mm2 files, note original + processed files
dm-get-experiment-files --experiment mm2
# dm@dmstorage: show png files in experiment directory
ls -l /opt/DM/data/ESAF/mm2/*.png
# dm@dmdaq: stop DAQ for experiment mm2
dm-stop-daq --experiment mm2
#
# SCENARIO 7: DATASET DEFINITION
#
# sveseli@dmstorage: add metadata for couple of experiment e2 files
# with different keys
dm-add-experiment-file --experiment e2 --file x1 status:good
dm-add-experiment-file --experiment e2 --file y1 status:bad
dm-get-experiment-files --experiment e2 --display-keys=fileName,status
# sveseli@dmstorage: add dataset metadata
dm-add-experiment-dataset --experiment e2 --dataset d1 status:g.*
# sveseli@dmstorage: get dataset files, note only one file matches
dm-get-experiment-dataset-files --experiment e2 --dataset d1
# sveseli@dmstorage: add metadata for anothare e2 file that
# should match dataset constraint
dm-add-experiment-file --experiment e2 --file x2 status:great
dm-get-experiment-files --experiment e2 --display-keys=fileName,status
# sveseli@dmstorage: get dataset files, note two files match
dm-get-experiment-dataset-files --experiment e2 --dataset d1
demo/apsu-20150709/machine_prep_notes.sv.txt deleted 100644 → 0
View file @ e561021c
# Demo environment consists of three linux VMs:
# - data acquisition (DAQ), data storage (DS), sge cluster (HPC) nodes
# - CentOS 6.6, 64-bit
# - no shared storage
# - DS node runs PostgreSQL database server, Web Portal, DS Web Service,
# CAT Web Service, MongoDB server
# - DAQ node runs DAQ Web Service
# - HPC node runs SGE cluster
# Machine Preparation
# ===================
# install dependencies (all machines)
yum install -y gcc libgcc expect zlib-devel openssl-devel openldap-devel subversion make sed gawk autoconf automake wget readline-devel
# Download globus RPM repo and install gridftp (both machines)
# http://toolkit.globus.org/ftppub/gt6/installers/repo/globus-toolkit-repo-latest.noarch.rpm
yum install globus-gridftp
# Disable requiredtty in /etc/sudoers
# Prepare gridftp server to use sshd (dmstorage machine)
globus-gridftp-server-enable-sshftp
# create system (dm) account on both machines, configure ssh-keys and
# authorized_keys files
# create several user accounts (dmstorage machine): dmuser1, dmuser2, dmuser3
# build and install epics base and SDDS/SDDSepics extensions under
# /opt/epics (dmstorage machine)
# build SDDS python under /opt/epics/extensions/src/SDDS/python/
# copy sdds.py into /opt/DM/support/python/linux-x86_64/lib/python2.7/
# copy /opt/epics/extensions/src/SDDS/python/O.linux-x86_64/sddsdatamodule.so
# into /opt/DM/support/python/linux-x86_64/lib/python2.7/lib-dynload/
# export /opt/DM to dmhpc node
# yum install nfs-util
# edit /etc/exports and add /opt/DM 192.168.100.8(rw,sync)
# exportfs -a
# restart nfs
# install sge on hpc machine, add dmstorage as submission node,
# copy /opt/sge to dmstorage
# configure /opt/DM area for software installation
mkdir -p /opt/DM
chown -R dm.dm /opt/DM
chmod 755 /opt/DM
# configure (or disable) firewall (both machines)
/etc/init.d/iptables stop
# DM Deployment: DS Machine
# =========================
# Log into dmstorage node and create local DM deployment directory
# in dm user home area
cd /opt/DM
ls -l
# Checkout code as release 0.2
svn co https://subversion.xray.aps.anl.gov/DataManagement/trunk dev
# Build support area
cd dev
make support
# Source setup
source setup.sh
# Create db
make db
# Configure Web Portal
# Note:
# - this needs to be done only during the first portal deployment,
# or after portal has been unconfigured explicitly
# - this step configures DB access
# - adds initial DM system user to the DB
make configure-web-portal
# Add few users
#dm-add-user --username dmuser1 --first-name Test --last-name User1
#dm-add-user --username dmuser2 --first-name Test --last-name User2
#dm-add-user --username dmuser3 --first-name Test --last-name User3
# Deploy Web Portal
# Note:
# - deploys portal war file into glassfish
# - after this step, users can access portal at
# https://dmstorage.svdev.net:8181/dm
make deploy-web-portal
# Deploy DS Web Service
# Note:
# - generates SSL certificates and configuration files
# - after this step, DS web service is accessible at port 22236
# - log files are under DM/var/log
# - configuration files are under DM/etc
# - user setup file is DM/etc/dm.setup.sh
# - service control script is under DM/dm-0.2/etc/init.d
make deploy-ds-web-service
# Check functionality. Open second terminal and log into dmstorage node
# as user sveseli
# Source setup file to get access to DM commands
source /opt/DM/etc/dm.setup.sh
# Get user list as administrator (dm) account
dm-get-users
# DM Deployment: DAQ Machine/HPC Machine
# ======================================
# Log into dmdaq node and create local DM deployment directory
# in dm user home area
cd /opt/DM
ls -l
# Checkout code as release 0.2
svn co https://subversion.xray.aps.anl.gov/DataManagement/trunk dev
# Build support area
# Note the following:
# - since demo machines are identical, we could simply copy support/dm code
# from the storage node; this is not necessarily the case in general
# - support area and DM code distribution can be shared between DAQ and DS
# nodes
# - support area on the daq node is much lighter (i.e., no need
# for glassfish, etc.)
cd dev
make support-daq
# Source setup
source setup.sh
# Deploy DAQ Web Service
# Note:
# - requires storage node to be installed
# - generates SSL certificates and configuration files
# - after this step, DAQ web service is accessible at port 33336
# - log files are under DM/var/log
# - configuration files are under DM/etc
# - user setup file is DM/etc/dm.setup.sh
make deploy-daq-web-service
demo/sdm-20150716/demo_notes.sv.txt deleted 100644 → 0
View file @ e561021c
# Need 1 terminal on DAQ node, 1 terminal on HPC node and 2 terminals on DS
# node
#####################################################################
# Prepare ahead of time
# ssh sveseli@dmstorage: add experiment e1, mm2
source /opt/DM/etc/dm.setup.sh
dm-add-experiment --name e1 --type-id 1
dm-add-experiment --name mm2 --type-id 1
# ssh dm@dmstorage: add few users
source /opt/DM/etc/dm.setup.sh
dm-add-user --username dmuser1 --first-name Test --last-name User1
dm-add-user --username dmuser2 --first-name Test --last-name User2
dm-add-user --username dmuser3 --first-name Test --last-name User3
#####################################################################
# Initialize demo
# ssh -X dm@dmstorage: start services
cd /opt/DM/dev
source setup.sh
./etc/init.d/dm-postgresql start
./etc/init.d/dm-glassfish start
./etc/init.d/dm-mongodb start
./etc/init.d/dm-ds-web-service start
./etc/init.d/dm-cat-web-service start
# ssh -X dm@dmdaq: start services
cd /opt/DM/dev
source setup.sh
./etc/init.d/dm-daq-web-service start
# ssh -X dm@dmhpc: start services, check NFS, check SGE
cd /opt/DM/dev
source setup.sh
./etc/init.d/dm-daq-web-service start
ls -l /net/dmstorage/opt/DM
source /opt/sge/default/common/settings.sh
qstat -f
#
# Check portal: https://dmstorage.svdev.net:8181/dm
#
#####################################################################
#
# Log into portal as dm admin: https://dmstorage.svdev.net:8181/dm
#
# Show users
# Show experiments
# Add dmuser1 to experiment e1
#
# SCENARIO 2: UPLOAD + METADATA CATALOG
#
# dm@dmstorage: check directory content on the storage node, should be empty
ls -l /opt/DM/data
# dm@dmstorage: check dmuser1 user, note list of groups
id dmuser1
# ssh sveseli@dmstorage: start experiment e1
source /opt/DM/etc/dm.setup.sh
dm-start-experiment --name e1
# dm@dmstorage: check directory content on the storage node
# note that experiment directory permissions are restricted
ls -l /opt/DM/data/ESAF
ls -l /opt/DM/data/ESAF/e1/
# dm@dmstorage: check dmuser1 user, note user belongs to new experiment group
id dmuser1
# sveseli@dmstorage: show there are no experiment files in catalogging service
dm-get-experiment-files --experiment e1
# ssh dm@dmdaq: source setup file, show test data
source /opt/DM/etc/dm.setup.sh
ls -lR /opt/DM/experiments/e1
cat /opt/DM/experiments/e1/file1
# dm@dmdaq: upload data for experiment e1, specify few arbitrary keys
dm-upload --experiment e1 --data-directory /opt/DM/experiments/e1 ownerUser:JohnC ownerGroup:APSU memo1:ApprovedByNDA memo2:DislikedByGD
# dm@dmstorage: check experiment storage directory content
# note permissions, ownership
ls -l /opt/DM/data/ESAF/e1/
ls -l /opt/DM/data/ESAF/e1/2015/07/09/
# sveseli@dmstorage: get metadata for experiment files from catalogging service
dm-get-experiment-files --experiment e1
dm-get-experiment-file --experiment e1 --file file2 --display-keys=__all__
# sveseli@dmstorage: show metadata updates
dm-update-experiment-file --experiment e1 --file file3 quality:A --display-keys=id,fileName,quality
# sveseli@dmstorage: show metadata search
dm-get-experiment-files --experiment e1 quality:A
dm-get-experiment-files --experiment e1 storageFilePath:2015
#
# SCENARIO 6: DAQ + METADATA CATALOG + SDDS PARAMETERS + HPC PROCESSING
#
# sveseli@dmstorage: add and start experiment mm2
dm-start-experiment --name mm2
# sveseli@dmstorage: get mm2 files, note no files
dm-get-experiment-files --experiment mm2
# dm@dmstorage: show no files in experiment directory
ls -l /opt/DM/data/ESAF
ls -l /opt/DM/data/ESAF/mm2
# dm@dmstorage: show processing script
more /opt/DM/processing/sge_sdds_analysis.sh
# dm@dmstorage: tail log file to observe processing
tail -f /opt/DM/var/log/dm.ds-web-service.log
# dm@dmhpc: show qstat
watch -d 'qstat -f'
# dm@dmdaq: start DAQ for experiment mm2, request SDDS parameter
# processing, specify SGE processing script
rm -rf /tmp/data/mm2
mkdir -p /tmp/data/mm2
dm-start-daq --experiment mm2 --data-directory /tmp/data/mm2 processSddsParameters:True sgeJobScript:/opt/DM/processing/sge_sdds_analysis.sh
# dm@dmdaq: copy experiment mm2 files into observed directory, watch qstat
ls -l /opt/DM/experiments/mm2/
cp /opt/DM/experiments/mm2/* /tmp/data/mm2/ && sleep 5 && touch /tmp/data/mm2/* &
tail -f /opt/DM/var/log/dm.daq-web-service.log
# sveseli@dmstorage: get mm2 files, note original + processed files
dm-get-experiment-files --experiment mm2
# dm@dmstorage: show png files in experiment directory
ls -l /opt/DM/data/ESAF/mm2/*.png
# sveseli@dmstorage: get one mm2 .edf file, note SDDS parameters in metadata
dm-get-experiment-file --experiment mm2 --file `dm-get-experiment-files --experiment mm2 --display-keys=fileName | grep -v png | head -1 | cut -f2 -d '='` --display-keys=__all__ --display-format=dict
# sveseli@dmstorage: get mm2 .png files, note parentFile key
dm-get-experiment-files --experiment mm2 fileName:.png --display-keys=fileName,parentFile --display-format=dict
# dm@dmstorage: open one processed file
xdg-open `ls -c1 /opt/DM/data/ESAF/mm2/*.png | head -1`
# dm@dmdaq: stop DAQ for experiment mm2
dm-stop-daq --experiment mm2
demo/sdm-20150716/machine_prep_notes.sv.txt deleted 100644 → 0
View file @ e561021c
# Demo environment consists of three linux VMs:
# - data acquisition (DAQ), data storage (DS), sge cluster (HPC) nodes
# - CentOS 6.6, 64-bit
# - no shared storage
# - DS node runs PostgreSQL database server, Web Portal, DS Web Service,
# CAT Web Service, MongoDB server
# - DAQ node runs DAQ Web Service
# - HPC node runs SGE cluster
# Machine Preparation
# ===================
# install dependencies (all machines)
yum install -y gcc libgcc expect zlib-devel openssl-devel openldap-devel subversion make sed gawk autoconf automake wget readline-devel
# Download globus RPM repo and install gridftp (both machines)
# http://toolkit.globus.org/ftppub/gt6/installers/repo/globus-toolkit-repo-latest.noarch.rpm
yum install globus-gridftp
# Disable requiredtty in /etc/sudoers
# Prepare gridftp server to use sshd (dmstorage machine)
globus-gridftp-server-enable-sshftp
# create system (dm) account on both machines, configure ssh-keys and
# authorized_keys files
# create several user accounts (dmstorage machine): dmuser1, dmuser2, dmuser3
# build and install epics base and SDDS/SDDSepics extensions under
# /opt/epics (dmstorage machine)
# build SDDS python under /opt/epics/extensions/src/SDDS/python/
# copy sdds.py into /opt/DM/support/python/linux-x86_64/lib/python2.7/
# copy /opt/epics/extensions/src/SDDS/python/O.linux-x86_64/sddsdatamodule.so
# into /opt/DM/support/python/linux-x86_64/lib/python2.7/lib-dynload/
# export /opt/DM to dmhpc node
# yum install nfs-util
# edit /etc/exports and add /opt/DM 192.168.100.8(rw,sync)
# exportfs -a
# restart nfs
# install sge on hpc machine, add dmstorage as submission node,
# copy /opt/sge to dmstorage
# configure /opt/DM area for software installation
mkdir -p /opt/DM
chown -R dm.dm /opt/DM
chmod 755 /opt/DM
# configure (or disable) firewall (both machines)
/etc/init.d/iptables stop
# DM Deployment: DS Machine
# =========================
# Log into dmstorage node and create local DM deployment directory
# in dm user home area
cd /opt/DM
ls -l
# Checkout code as release 0.2
svn co https://subversion.xray.aps.anl.gov/DataManagement/trunk dev
# Build support area
cd dev
make support
# Source setup
source setup.sh
# Create db
make db
# Configure Web Portal
# Note:
# - this needs to be done only during the first portal deployment,
# or after portal has been unconfigured explicitly
# - this step configures DB access
# - adds initial DM system user to the DB
make configure-web-portal
# Add few users
#dm-add-user --username dmuser1 --first-name Test --last-name User1
#dm-add-user --username dmuser2 --first-name Test --last-name User2
#dm-add-user --username dmuser3 --first-name Test --last-name User3
# Deploy Web Portal
# Note:
# - deploys portal war file into glassfish
# - after this step, users can access portal at
# https://dmstorage.svdev.net:8181/dm
make deploy-web-portal
# Deploy DS Web Service
# Note:
# - generates SSL certificates and configuration files
# - after this step, DS web service is accessible at port 22236
# - log files are under DM/var/log
# - configuration files are under DM/etc
# - user setup file is DM/etc/dm.setup.sh
# - service control script is under DM/dm-0.2/etc/init.d
make deploy-ds-web-service
# Check functionality. Open second terminal and log into dmstorage node
# as user sveseli
# Source setup file to get access to DM commands
source /opt/DM/etc/dm.setup.sh
# Get user list as administrator (dm) account
dm-get-users
# DM Deployment: DAQ Machine/HPC Machine
# ======================================
# Log into dmdaq node and create local DM deployment directory
# in dm user home area
cd /opt/DM
ls -l
# Checkout code as release 0.2
svn co https://subversion.xray.aps.anl.gov/DataManagement/trunk dev
# Build support area
# Note the following:
# - since demo machines are identical, we could simply copy support/dm code
# from the storage node; this is not necessarily the case in general
# - support area and DM code distribution can be shared between DAQ and DS
# nodes
# - support area on the daq node is much lighter (i.e., no need
# for glassfish, etc.)
cd dev
make support-daq
# Source setup
source setup.sh
# Deploy DAQ Web Service
# Note:
# - requires storage node to be installed
# - generates SSL certificates and configuration files
# - after this step, DAQ web service is accessible at port 33336
# - log files are under DM/var/log
# - configuration files are under DM/etc
# - user setup file is DM/etc/dm.setup.sh
make deploy-daq-web-service
demo/sprint-20150421/demo_notes.sv.txt deleted 100644 → 0
View file @ e561021c
# Demo environment consists of two linux VMs:
# - data acquisition (DAQ) and data storage (DS) nodes
# - CentOS 6.6, 64-bit
# - no shared storage
# - DS node runs database server, Web Portal and DS Web Service
# - DAQ node runs DAQ Web Service
# Machine Preparation
# ===================
# install dependencies (both machines)
yum install -y gcc libgcc expect zlib-devel openssl-devel openldap-devel subversion make sed gawk autoconf automake wget readline-devel
# create system (dm) account on both machines, configure ssh-keys and
# authorized_keys files
# configure /opt/DM area for software installation
mkdir -p /opt/DM
chown -R dm.dm /opt/DM
chmod 755 /opt/DM
# configure (or disable) firewall (both machines)
/etc/init.d/iptables stop
# DM Deployment: DS Machine
# =========================
# Log into dmstorage node and create local DM deployment directory
# in dm user home area
cd /opt/DM
ls -l
# Checkout code as release 0.1
svn co https://subversion.xray.aps.anl.gov/DataManagement/tags/20150421 dm-0.1
# Build support area
cd dm-0.1
make support
# Source setup
source setup.sh
# Create db
make db
# Configure Web Portal
# Note:
# - this needs to be done only during the first portal deployment,
# or after portal has been unconfigured explicitly
# - this step configures DB access
make configure-web-portal
# Deploy Web Portal
# Note:
# - deploys portal war file into glassfish
# - after this step, users can access portal at
# https://dmstorage.svdev.net:8181/dm
make deploy-web-portal
# Deploy DS Web Service
# Note:
# - generates SSL certificates and configuration files
# - after this step, DS web service is accessible at port 22236
# - log files are under DM/var/log
# - configuration files are under DM/etc
# - user setup file is DM/etc/dm.setup.sh
# - service control script is under DM/dm-0.1/etc/init.d
make deploy-ds-web-service
# Check functionality. Open second terminal and log into dmstorage node
# as user sveseli
# Source setup file to get access to DM commands
source /opt/DM/etc/dm.setup.sh
# Attempt to get list of users as user sveseli, should result
# in authorization error
# Note:
# - every command comes with common set of options
dm-get-users -h
dm-get-users --version
dm-get-users
echo $?
# Repeat command, this time us administrator (dm) account
dm-get-users
# Repeat command, note that session with DS service has been established, so no
# more password prompts until session expires
cat ~/.dm/.ds.session.cache
dm-get-users
# DM Deployment: DAQ Machine
# ==========================
# Log into dmdaq node and create local DM deployment directory
# in dm user home area
cd /opt/DM
ls -l
# Checkout code as release 0.1
svn co https://subversion.xray.aps.anl.gov/DataManagement/tags/20150421 dm-0.1
# Build support area
# Note the following:
# - since demo machines are identical, we could simply copy support/dm code
# from the storage node; this is not necessarily the case in general
# - support area and DM code distribution can be shared between DAQ and DS
# nodes
# - support area on the daq node is much lighter (i.e., no need
# for glassfish, etc.)
cd dm-0.1
make support-daq
# Source setup
source setup.sh
# Deploy DAQ Web Service
# Note:
# - requires storage node to be installed
# - generates SSL certificates and configuration files
# - after this step, DAQ web service is accessible at port 33336
# - log files are under DM/var/log
# - configuration files are under DM/etc
# - user setup file is DM/etc/dm.setup.sh
make deploy-daq-web-service
# DM Functionality: DAQ
# =====================
# add new experiment (sveseli@dmstorage)
dm-add-experiment -h
dm-add-experiment --name exp1 --type-id 1 --description test
dm-get-experiments
dm-get-experiment --name exp1
dm-get-experiment --name exp1 --display-keys=__all__
# check directory content on the storage node (dm@dmstorage)
ls -l /opt/DM/data
# start experiment (sveseli@dmstorage)
dm-start-experiment --name exp1
# check directory content on the storage node (dm@dmstorage)
ls -l /opt/DM/data
ls -l /opt/DM/data/ESAF
ls -l /opt/DM/data/ESAF/exp1/
# at this point we can log into the portal to see experiment that was created
# observe that start time is entered correctly
# in the first terminal on the daq node, tail log file (dm@dmdaq)
tail -f /opt/DM/var/log/dm.daq-web-service.log
# open second terminal for daq node, login as system (dm) user
# source setup file (dm@dmdaq)
cat /opt/DM/etc/dm.setup.sh
source /opt/DM/etc/dm.setup.sh
# prepare DAQ directory for this experiment (dm@dmdaq)
mkdir -p /tmp/data/exp1
# start DAQ (dm@dmdaq)
dm-start-daq -h
dm-start-daq --experiment exp1 --data-directory /tmp/data/exp1
# create test file in the DAQ directory (daq node)
# observe log file entries, point out file transfer
touch /tmp/data/exp1/file1
echo "Hello there, data management is here" > /tmp/data/exp1/file1
# check directory content on the storage node (dm@dmstorage)
# file1 should be transferred
ls -l /opt/DM/data/ESAF/exp1/
# stop DAQ (dm@dmdaq)
dm-stop-daq -h
dm-stop-daq --experiment exp1
# DM Functionality: Upload
# ========================
# prepare data directory we want to upload (dm@dmdaq)
mkdir -p /tmp/data/exp1/2015/04/21
echo "this is file 2" > /tmp/data/exp1/2015/04/21/file2
echo "this is file 3" > /tmp/data/exp1/2015/04/21/file3
# check directory content on the storage node (dm@dmstorage)
ls -l /opt/DM/data/ESAF/exp1/
# upload data (dm@dmdaq)
dm-upload -h
dm-upload --experiment exp1 --data-directory /tmp/data/exp1
# check directory content on the storage node (dm@dmstorage)
ls -l /opt/DM/data/ESAF/exp1/
ls -l /opt/DM/data/ESAF/exp1/2015/04/21/
cat /opt/DM/data/ESAF/exp1/2015/04/21/file3
# stop experiment (sveseli@dmstorage)
dm-stop-experiment --name exp1
# at this point we can log into the portal to see modified experiment
# observe that end time is entered correctly
demo/sprint-20150630/demo_notes.sv.txt deleted 100644 → 0
View file @ e561021c
# Demo environment consists of two linux VMs:
# - data acquisition (DAQ) and data storage (DS) nodes
# - CentOS 6.6, 64-bit
# - no shared storage
# - DS node runs database server, Web Portal and DS Web Service
# - DAQ node runs DAQ Web Service
# Machine Preparation
# ===================
# install dependencies (both machines)
yum install -y gcc libgcc expect zlib-devel openssl-devel openldap-devel subversion make sed gawk autoconf automake wget readline-devel
# Download globus RPM repo and install gridftp (both machines)
# http://toolkit.globus.org/ftppub/gt6/installers/repo/globus-toolkit-repo-latest.noarch.rpm
yum install globus-gridftp
# Disable requiredtty in /etc/sudoers
# Prepare gridftp server to use sshd (dmstorage machine)
globus-gridftp-server-enable-sshftp
# create system (dm) account on both machines, configure ssh-keys and
# authorized_keys files
# create several user accounts (dmstorage machine): dmuser1, dmuser2, dmuser3
# build and install epics base and SDDS/SDDSepics extensions under
# /opt/epics (dmstorage machine)
# configure /opt/DM area for software installation
mkdir -p /opt/DM
chown -R dm.dm /opt/DM
chmod 755 /opt/DM
# configure (or disable) firewall (both machines)
/etc/init.d/iptables stop
# DM Deployment: DS Machine
# =========================
# Log into dmstorage node and create local DM deployment directory
# in dm user home area
cd /opt/DM
ls -l
# Checkout code as release 0.2
svn co https://subversion.xray.aps.anl.gov/DataManagement/tags/20150630 dm-0.2
# Build support area
cd dm-0.2
make support
# Source setup
source setup.sh
# Create db
make db
# Configure Web Portal
# Note:
# - this needs to be done only during the first portal deployment,
# or after portal has been unconfigured explicitly
# - this step configures DB access
# - adds initial DM system user to the DB
make configure-web-portal
# The above step used two new utilities that go directly to the db:
dm-add-user -h
dm-add-user-system-role -h
# Add few users
dm-add-user --username dmuser1 --first-name Test --last-name User1
dm-add-user --username dmuser2 --first-name Test --last-name User2
dm-add-user --username dmuser3 --first-name Test --last-name User3
# Deploy Web Portal
# Note:
# - deploys portal war file into glassfish
# - after this step, users can access portal at
# https://dmstorage.svdev.net:8181/dm
make deploy-web-portal
# Show no sudo functionality for DM account
sudo -l
# Deploy DS Web Service
# Note:
# - generates SSL certificates and configuration files
# - after this step, DS web service is accessible at port 22236
# - log files are under DM/var/log
# - configuration files are under DM/etc
# - user setup file is DM/etc/dm.setup.sh
# - service control script is under DM/dm-0.2/etc/init.d
make deploy-ds-web-service
# Show sudo functionality for DM account that enables group/permission
# management
sudo -l
# Check functionality. Open second terminal and log into dmstorage node
# as user sveseli
# Source setup file to get access to DM commands
source /opt/DM/etc/dm.setup.sh
# Get user list as administrator (dm) account
dm-get-users
# DM Deployment: DAQ Machine
# ==========================
# Log into dmdaq node and create local DM deployment directory
# in dm user home area
cd /opt/DM
ls -l
# Checkout code as release 0.2
svn co https://subversion.xray.aps.anl.gov/DataManagement/tags/20150630 dm-0.2
# Build support area
# Note the following:
# - since demo machines are identical, we could simply copy support/dm code
# from the storage node; this is not necessarily the case in general
# - support area and DM code distribution can be shared between DAQ and DS
# nodes
# - support area on the daq node is much lighter (i.e., no need
# for glassfish, etc.)
cd dm-0.2
make support-daq
# Source setup
source setup.sh
# Deploy DAQ Web Service
# Note:
# - requires storage node to be installed
# - generates SSL certificates and configuration files
# - after this step, DAQ web service is accessible at port 33336
# - log files are under DM/var/log
# - configuration files are under DM/etc
# - user setup file is DM/etc/dm.setup.sh
make deploy-daq-web-service
# DM Functionality: DAQ
# =====================
# add new experiment and couple of users (sveseli@dmstorage)
dm-add-experiment --name exp1 --type-id 1 --description test
dm-add-user-experiment-role --username dmuser1 --experiment exp1 --role=User
dm-add-user-experiment-role --username dmuser2 --experiment exp1 --role=User
# Note that dmuser1 and 2 are on the list of experiment users
dm-get-experiments
dm-get-experiment --name exp1 --display-keys=__all__
# check directory content on the storage node (dm@dmstorage)
ls -l /opt/DM/data
# Show that unix account corresponding to dmuser1 has no special groups
# associated with it
id dmuser1
# Show there is no exp1 unix group
grep exp1 /etc/group
# start experiment (sveseli@dmstorage)
dm-start-experiment --name exp1
# Show there is now exp1 unix group
grep exp1 /etc/group
# check directory content on the storage node (dm@dmstorage)
# note that experiment directory permissions are restricted
ls -l /opt/DM/data/ESAF
ls -l /opt/DM/data/ESAF/exp1/
# Check experiment user groups: only 1 and 2 should have new group assigned
# to them
id dmuser1
id dmuser2
id dmuser3
# in the first terminal on the storage node, tail log file (dm@dmdstorage)
tail -f /opt/DM/var/log/dm.ds-web-service.log
# in the first terminal on the daq node, tail log file (dm@dmdaq)
tail -f /opt/DM/var/log/dm.daq-web-service.log
# open second terminal for daq node, login as system (dm) user
# source setup file (dm@dmdaq)
source /opt/DM/etc/dm.setup.sh
# prepare DAQ directory for this experiment (dm@dmdaq)
mkdir -p /tmp/data/exp1
# create test file in the DAQ directory (daq node)
# observe log file entries, point out file transfer
echo "Hello there, data management is here" > /tmp/data/exp1/file1
# check directory content on the storage node (dm@dmstorage)
# file1 should be transferred
ls -l /opt/DM/data/ESAF/exp1/
# upload data (dm@dmdaq)
dm-upload --experiment exp1 --data-directory /tmp/data/exp1
# check directory content on the storage node (dm@dmstorage)
# file1 should be transferred
# note permissions
ls -l /opt/DM/data/ESAF/exp1/
# as root@dmstorage, su into dmuser1 account and try to read data
# should work
cat /opt/DM/data/ESAF/exp1/file1
# as root@dmstorage, su into dmuser3 account and try to read data
# should fail
cat /opt/DM/data/ESAF/exp1/file1
# Demonstrate retries: show config file
vi /opt/DM/etc/dm.daq-web-service.conf
# As root@dmdaq, temporarily move rsync
mv /usr/bin/rsync /usr/bin/rsync.orig
# upload new data (dm@dmdaq), observe how transfer fails
echo "Hello there, data management is here again" > /tmp/data/exp1/file1
dm-upload --experiment exp1 --data-directory /tmp/data/exp1
# As root@dmdaq, restore rsync, observe how transfer succeeds
mv /usr/bin/rsync.orig /usr/bin/rsync
# check directory content on the storage node (dm@dmstorage)
# file1 should be transferred
ls -l /opt/DM/data/ESAF/exp1/
# Demonstrate gridftp plugin
# Edit config file as dm@dmdaq, comment out rsync plugin, uncomment gridftp
# plugin; restart service
vi /opt/DM/etc/dm.daq-web-service.conf
./etc/init.d/dm-daq-web-service restart
tail -f /opt/DM/var/log/dm.daq-web-service.log
# upload new data (dm@dmdaq), observe how transfer succeeds
echo "Hello there, data management is here yet again" > /tmp/data/exp1/file1
dm-upload --experiment exp1 --data-directory /tmp/data/exp1
# stop experiment (sveseli@dmstorage)
dm-stop-experiment --name exp1
getting_started/getting-started.md 0 → 100644
View file @ ae1437b1
Document can now be found [here](https://git.aps.anl.gov/DM/dm-docs/-/wikis/DM/HowTos/Getting-Started) on the [DM Wiki](https://git.aps.anl.gov/DM/dm-docs/-/wikis/home).
getting_started/gettingDataFromGlobus.md 0 → 100644
View file @ ae1437b1
Document can now be found [here](https://git.aps.anl.gov/DM/dm-docs/-/wikis/DM/HowTos/Getting-Data-From-Globus) on the [DM Wiki](https://git.aps.anl.gov/DM/dm-docs/-/wikis/home).
sphinx/.svnignore deleted 100644 → 0
View file @ e561021c
build
sphinx/Makefile deleted 100644 → 0
View file @ e561021c
# Minimal makefile for Sphinx documentation
#
# You can set these variables from the command line.
SPHINXOPTS =
SPHINXBUILD = sphinx-build
SPHINXPROJ = APSDataManagement
SOURCEDIR = source
BUILDDIR = build
# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
.PHONY: help Makefile
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
sphinx/source/conf.py deleted 100644 → 0
View file @ e561021c
# -*- coding: utf-8 -*-
#
# APS Data Management documentation build configuration file, created by
# sphinx-quickstart on Thu Feb 23 09:20:39 2017.
#
# This file is execfile()d with the current directory set to its
# containing dir.
#
# Note that not all possible configuration values are present in this
# autogenerated file.
#
# All configuration values have a default; values that are commented out
# serve to show the default.
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
# import os
# import sys
# sys.path.insert(0, os.path.abspath('.'))
# -- General configuration ------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
#
# needs_sphinx = '1.0'
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = ['sphinx.ext.autodoc',
'sphinx.ext.coverage',
'sphinx.ext.ifconfig',
'sphinx.ext.viewcode']
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# The suffix(es) of source filenames.
# You can specify multiple suffix as a list of string:
#
# source_suffix = ['.rst', '.md']
source_suffix = '.rst'
# The master toctree document.
master_doc = 'index'
# General information about the project.
project = u'APS Data Management'
copyright = u'2015-2019, Argonne National Laboratory'
author = u'APS/SDM'
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
# built documents.
#
# The short X.Y version.
version = u'1.0'
# The full version, including alpha/beta/rc tags.
release = u'1.0'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#
# This is also used if you do content translation via gettext catalogs.
# Usually you set "language" from the command line for these cases.
language = None
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This patterns also effect to html_static_path and html_extra_path
exclude_patterns = []
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
# If true, `todo` and `todoList` produce output, else they produce nothing.
todo_include_todos = False
# -- Options for HTML output ----------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
#html_theme = 'alabaster'
html_theme = "sphinx_rtd_theme"
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
#
# html_theme_options = {}
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
# -- Options for HTMLHelp output ------------------------------------------
# Output file base name for HTML help builder.
htmlhelp_basename = 'APSDataManagementdoc'
# -- Options for LaTeX output ---------------------------------------------
latex_elements = {
# The paper size ('letterpaper' or 'a4paper').
#
# 'papersize': 'letterpaper',
# The font size ('10pt', '11pt' or '12pt').
#
# 'pointsize': '10pt',
# Additional stuff for the LaTeX preamble.
#
# 'preamble': '',
# Latex figure (float) alignment
#
# 'figure_align': 'htbp',
}
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
latex_documents = [
(master_doc, 'APSDataManagement.tex', u'APS Data Management Documentation',
u'APS/SDM', 'manual'),
]
# -- Options for manual page output ---------------------------------------
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
(master_doc, 'apsdatamanagement', u'APS Data Management Documentation',
[author], 1)
]
# -- Options for Texinfo output -------------------------------------------
# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
(master_doc, 'APSDataManagement', u'APS Data Management Documentation',
author, 'APSDataManagement', 'One line description of project.',
'Miscellaneous'),
]
sphinx/source/dm.aps_db_web_service.api.rst deleted 100644 → 0
View file @ e561021c
.. automodule:: dm.aps_db_web_service.api
.. currentmodule:: dm.aps_db_web_service.api
ApsDbApiFactory
---------------
.. autoclass:: dm.aps_db_web_service.api.apsDbApiFactory.ApsDbApiFactory()
:show-inheritance:
:members: getBssApsDbApi, getEsafApsDbApi, getServiceMonitorApsDbApi
BssApsDbApi
-----------
.. autoclass:: dm.aps_db_web_service.api.bssApsDbApi.BssApsDbApi()
:show-inheritance:
:members: __init__, listRuns, listBeamlines, getCurrentRun, listProposals, getProposal
EsafApsDbApi
------------
.. autoclass:: dm.aps_db_web_service.api.esafApsDbApi.EsafApsDbApi()
:show-inheritance:
:members: __init__, listEsafs, getEsaf
ServiceMonitorApsDbApi
----------------------
.. autoclass:: dm.aps_db_web_service.api.serviceMonitorApsDbApi.ServiceMonitorApsDbApi()
:show-inheritance:
:members: __init__, getServiceStatus, clearServiceStatus
sphinx/source/dm.cat_web_service.api.rst deleted 100644 → 0
View file @ e561021c
.. automodule:: dm.cat_web_service.api
.. currentmodule:: dm.cat_web_service.api
CatApiFactory
-------------
.. autoclass:: dm.cat_web_service.api.catApiFactory.CatApiFactory()
:show-inheritance:
:members: getDatasetCatApi, getFileCatApi, getServiceMonitorCatApi
DatasetCatApi
-------------
.. autoclass:: dm.cat_web_service.api.datasetCatApi.DatasetCatApi()
:show-inheritance:
:members: __init__, addExperimentDataset, updateExperimentDataset, updateDatasetById, getDatasets, getExperimentDatasets, getDatasetById, getExperimentDataset, getExperimentDatasetFiles
FileCatApi
----------
.. autoclass:: dm.cat_web_service.api.fileCatApi.FileCatApi()
:show-inheritance:
:members: __init__, addExperimentFile, updateExperimentFile, updateExperimentFileById, updateExperimentFiles, updateExperimentFilesAsync, getAsyncUpdateStatus, getExperimentFiles, getExperimentFileById, getExperimentFile, deleteExperimentFile, deleteExperimentFileById, getExperimentFileCollectionStats, renameExperimentFileCollection, deleteExperimentFileCollection
ServiceMonitorCatApi
--------------------
.. autoclass:: dm.cat_web_service.api.serviceMonitorCatApi.ServiceMonitorCatApi()
:show-inheritance:
:members: __init__, getServiceStatus, clearServiceStatus
sphinx/source/dm.common.api.rst deleted 100644 → 0
View file @ e561021c
.. automodule:: dm.common.api
.. currentmodule:: dm.common.api
DmApiFactory
--------------
.. autoclass:: dm.common.api.dmApiFactory.DmApiFactory()
:show-inheritance:
:members: USERNAME_KEY, PASSWORD_KEY, PASSWORD_FILE_KEY, LOGIN_FILE_KEY, HOST_KEY, PORT_KEY, PROTOCOL_KEY, setDefaults, setConfigFile, getConfiguration
sphinx/source/dm.daq_web_service.api.rst deleted 100644 → 0
View file @ e561021c
.. automodule:: dm.daq_web_service.api
.. currentmodule:: dm.daq_web_service.api
DaqApiFactory
-------------
.. autoclass:: dm.daq_web_service.api.daqApiFactory.DaqApiFactory()
:show-inheritance:
:members: getExperimentDaqApi, getServiceMonitorDaqApi
ExperimentDaqApi
-----------------
.. autoclass:: dm.daq_web_service.api.experimentDaqApi.ExperimentDaqApi()
:show-inheritance:
:members: __init__, startDaq, stopDaq, listDaqs, getDaqInfo, clearDaq, upload, stopUpload, listUploads, getUploadInfo, clearUpload, listProcessingPlugins
ServiceMonitorDaqApi
--------------------
.. autoclass:: dm.daq_web_service.api.serviceMonitorDaqApi.ServiceMonitorDaqApi()
:show-inheritance:
:members: __init__, getServiceStatus, clearServiceStatus