From 26e4ac6b76b3ae9f1544bbd09e74cbf99e81ca64 Mon Sep 17 00:00:00 2001
From: hammonds <JPHammonds@anl.gov>
Date: Fri, 26 Mar 2021 15:38:48 -0500
Subject: [PATCH] Changes to reflect changes of dm repository
---
.../DataManagementSplitSystemSetup.md | 33 ++++++++++---------
1 file changed, 17 insertions(+), 16 deletions(-)
diff --git a/Installation/DataManagementSplitSystemSetup.md b/Installation/DataManagementSplitSystemSetup.md
index 76652171..048a40e2 100644
--- a/Installation/DataManagementSplitSystemSetup.md
+++ b/Installation/DataManagementSplitSystemSetup.md
@@ -46,7 +46,7 @@ __After these ports are added select__ `Reload Firewall` __from the Options menu
### Support Tools Installation
Before installation of the APS Data Management System a number of tools need to be installed on the server nodes. The __DM__ system depends on tools such as Java, Python, Postgresql, MongoDB, ZeroMQ, etc. A set of scripts have been established which will download, build (when necessary) and install these tools for use with the __DM__ system. While it is possible to install most of these tools using more conventional means (e.g. RPM on Linux) the install scripts provided here builds and installs these tools specifically for use with the __DM__ system.
-For the purposes of this tutorial, we will are creating two nodes which will contain different piesces of the __DM__. One node will be referred to as the data-storage node this will contain the data storage web service and the Postgresql database which conatains the user database. The second node will b reffered to as the exp-station node. This node will provide the cat web service (a catalog of the stored data), the daq web service (provides a way to move collected data) and the proc web service (provides a means to process data).
+For the purposes of this tutorial, we will are creating two nodes which will contain different pieces of the __DM__. One node will be referred to as the data-storage node. This will contain the data storage web service and the Postgresql database which conatains the user database. The second node will be reffered to as the exp-station node. This node will provide the cat web service (a catalog of the stored data), the daq web service (provides a way to move collected data) and the proc web service (provides a means to process data).
These scripts can be found in the APS git repository at:
@@ -56,12 +56,13 @@ On both Nodes:
* Select an account (such as dmadmin) which will build, install and manage the __DM__ system.
* Select a parent location to install the system and create a subdirectory __DM__ to contain the __DM__ system and the support tools. We will refer to this directory in future sections as DM\_INSTALL\_DIR
- * Install a copy of the code from the _support_ git repository in DM\_INSTALL\_DIR. This can be done in a variety of ways (3rd an 4th should be the most common)
+ * Install a copy of the code from the _support_ git repository in DM\_INSTALL\_DIR. This can be done in a variety of ways (3rd should be the most common for development)
- Grab a zip file from the APS Gitlab website (from URLs above) and unzip the file.
- Clone the repositories directly into DM\_INSTALL\_DIR (basically like cloning a forked repo shown below)
- Fork the repository following the fork link in the top right of the project page and then clone the repository as shown below. The example shown clones the dm-support repository into a directory __support__ and the __DM__ repository into a directory __dev__. In each case the clone is pulled from the user _USERNAME_'s fork of the repository.
> git clone https://git.aps.anl.gov/_USERNAME_/dm-support.git __support__ (Assumes forking repository)
+
* Change directory to the _support_ directory
> cd support
@@ -70,30 +71,30 @@ On both Nodes:
We will install support tools needed by the data-storage node. Again these tools will support the data storage service, a central database (which stores information on users, experiments, and beamline deployments) and Web Portal that allows some management of the sytem. For these services, this step will install postgresql, openjdk, ant, payara, python and a number of needed python modules.
- * Run the command `./sbin/install_support_ds.sh`. This installation will take some time to complete as this will download, compile and configure a number of key tools. NOTE: to later wipe out this step of the install run `./sbin/clean_support_all.sh`.
+ * Run the command `./sbin/install_support_with_conda_ds.sh`. This installation will take some time to complete as this will download, compile and configure a number of key tools. NOTE: to later wipe out this step of the install run `./sbin/clean_support_all.sh`.
* As this script runs, you will be prompted to provide passwords for the master and admin accounts for the Payara web server. Choose appropriate passwords & record these for later use. These will be used to manage the Payara server, which will provide a portal for managing some parts of the DM.
##### On exp-station node
Similar to the data-storage node, we will install support tools for the experiment station node. These tools will support the daq, proc & cat web services. This will facilitate managing file transfers during or after acquisition, processing data after collection and managing experiment meta-data. To support this this will download & install Python 2 and a number of associated modules and Python 3 and the same modules. Note, in the near future this should be just Python 3 versions.
- * Run the command `./sbin/install_support_daq.sh`. This will take a some time as it downloads & compiles from source. NOTE: Again, to later wipe out this step of the install run `./sbin/clean_support_all.sh`.
+ * Run the command `./sbin/install_support_with_conda_daq.sh`. This will take a some time as it downloads & compiles from source. NOTE: Again, to later wipe out this step of the install run `./sbin/clean_support_all.sh`.
### Data Management component installation
Once again, we are installing two different systems, each with different parts of the system to provide different features on each. Also, scripts have been developed to install and configure the components of the system. These scripts can be found at
-[https://git.aps.anl.gov/DM/dm.git](https://git.aps.anl.gov/DM/dm.git)
+[https://git.aps.anl.gov/DM/data-management.git](https://git.aps.anl.gov/DM/data-management.git)
-The installation scripts for the DM System assume a particular directory structure. The contents of this repository should be cloned in the DM\_INSTALL\_DIR into a directory corresponding to a version tag. This allows the system to be updated in a way that allows updating the system in operation with a new versioned directory. Initially, and as the system is updated, a symbolic link called _production_, in DM\_INSTALL\_DIR, should be directed to the version tagged directory of _dm_. Similarly, if it is discovered that fallback is necessary, then the link will be moved back to an older version. An example of this, is shown in the image below.
+The installation scripts for the DM System assume a particular directory structure. The contents of this repository should be cloned in the DM\_INSTALL\_DIR into a directory corresponding to a version tag. This allows the system to be updated in a way that allows updating the system in operation with a new versioned directory. Initially, and as the system is updated, a symbolic link called _production_, in DM\_INSTALL\_DIR, should be directed to the version tagged directory of _data-management_. Similarly, if it is discovered that fallback is necessary, then the link will be moved back to an older version. An example of this, is shown in the image below.
-A stepped instruction for this, assuming as with the support module a fork of the _dm_ repository has been forked by a user, follows. These steps should be followed on both _data-storage_ and _exp-station_ nodes.
+A stepped instruction for this, assuming as with the support module a fork of the _data-management_ repository has been forked by a user, follows. These steps should be followed on both _data-storage_ and _exp-station_ nodes.
* Change directory to DM\_INSTALL\_DIR
* clone the forked repository into a version_tagged directory
-> git clone https://git.aps.anl.gov/_USERNAME_/dm.git dm\_version\_tag
+> git clone https://git.aps.anl.gov/_USERNAME_/data-management.git dm\_version\_tag
* create a link of the cloned directory to _production_
@@ -103,15 +104,15 @@ A stepped instruction for this, assuming as with the support module a fork of th
This node will be responsible for providing the data storage web service, the postgresql database (which stores information on users, experiments, and beamline deployments), and the payara web server (provides portal for management).
-To install _dm_ compnents for the data-storage node
+To install _data-management_ compnents for the data-storage node
* cd DM\_INSTALL\_DIR/production
- * edit etc/dm.deploy.conf to change DM\_CA\_HOST to data-storage
+ * edit etc/dm.deploy.conf to change DM\_CA\_HOST to data-storage (certificates contained to this development)
* ./sbin/dm\_deploy\_data\_storage.sh
- This deploy process will install components and prompt for user input as necessary. Prompts will ask for a number of system passwords, some existing and some being set by this process, node names for the DS web service node and file locations. These include
- __postgres__ admin account - This will be used to manage the postgres itself. Each developer can set this to a unique value.
- __dm__ db management account - This will be for mananging the 'dm' database in postgres. Each developer can set this to a unique value.
- data storage directory - this directory will serve as the root directory for storage of data in the system. During transfers initiated by the daq web service, files will be moved into subdirectories of this system. The subdirectory paths will be constructed from beamline name, experiment name and a path specified by the user in the transfer setup.
- - __dm__ system account - This is user __dm__ in the Data Management system. This user has administrative priviledge in the Data Management system. This is a user in the 'dm' user table. Each developer can set this to a unique value.
+ - __dm__ system account - This is user __dm__ in the Data Management system. This user has administrative priviledge in the Data Management system. This is a user in the 'dm' user\_info table. Each developer can set this to a unique value.
- __dmadmin__ LDAP password - This password provides the Data Management software access to the APS/ANL LDAP system to gather reference to that database. This is a password to an external system and and is therefore a pre-existing password that developers will need to get from the Data Management system administrator.
#### exp-station Node Installation
@@ -123,8 +124,8 @@ To install _dm_ components on the exp-station:
* Edit the file etc/dm.deploy.conf to ensure that the DM\_CA\_HOST is set to the data-storage node.
* ./sbin/dm\_deploy\_exp\_station.sh
- This will start the installation process which will prompt for
- - DM DS Web Service Host (data-storage in this case
- - DM DS Web Servervice Installation directory (where the web service is installed on node data-storage)
+ - DM DS Web Service Host (data-storage in this case)
+ - DM DS Web Service Installation directory (where the web service is installed on node data-storage)
- DM DAQ station name. TEST in this instance, something like 8-ID-I on the real system. Oficial name of station in facility system such as our proposal/ESAF/Scheduling systems.
@@ -222,7 +223,7 @@ You should at this point have a directory installed which has both the _Data Man
- Re-source the setup file from step 1. This is only necessary on exp-station.
- source etc/dm.setup.sh
-At this point we will are more in a position to start using the sytem. As a first test we will add a few test users to the system and then run the command dm-test-upload which will
+At this point we are more in a position to start using the sytem. As a first test we will add a few test users to the system and then run the command dm-test-upload which will
* create a new experiment
* attach a list of users to the experiment
* define a location where data exists
@@ -231,7 +232,7 @@ At this point we will are more in a position to start using the sytem. As a firs
To accomplish this we use the following
-To add 3 users
+To add 3 users (must be done on the data-storage node. dm-add-user is done by direct DB access. This is not an interaction one would normally have with the system)
```
dm-add-user --username jprofessor --last-name Professor --first-name John
@@ -239,7 +240,7 @@ dm-add-user --username gpostdoc --last-name Postdoc --first-name George
dm-add-user --username jgradstudent --last-name Gradstudent --first-name Jane
```
-To add an experiment, define the users, and kick off an upload:
+To add an experiment, define the users, and kick off an upload: (This and all following should be done on exp-station as this would normally have access to user experiment data)
```
dm-test-upload --experiment=e1 --data-directory=/home/dmadmin/testData --dest-directory=MyFirstExperiment --users=jprofessor,gpostdoc,jgradstudent
--
GitLab