@@ -109,7 +111,7 @@ To install _dm_ compnents for the data-storage node
- __postgres__ admin account - This will be used to manage the postgres itself. Each developer can set this to a unique value.
- __dm__ db management account - This will be for mananging the 'dm' database in postgres. Each developer can set this to a unique value.
- data storage directory - this directory will serve as the root directory for storage of data in the system. During transfers initiated by the daq web service, files will be moved into subdirectories of this system. The subdirectory paths will be constructed from beamline name, experiment name and a path specified by the user in the transfer setup.
- __dm__ system account - This is user __dm__in the Data Management system. This user has administrative priviledge in the Data Management system. This is a user in the 'dm' user table. Each developer can set this to a unique value.
- __dm__ system account - This is user __dm__in the Data Management system. This user has administrative priviledge in the Data Management system. This is a user in the 'dm' user table. Each developer can set this to a unique value.
- __dmadmin__ LDAP password - This password provides the Data Management software access to the APS/ANL LDAP system to gather reference to that database. This is a password to an external system and and is therefore a pre-existing password that developers will need to get from the Data Management system administrator.
#### exp-station Node Installation
...
...
@@ -130,12 +132,12 @@ For initial test/development purposes, a few changes are necessary to short-circ
##### On the data-storage Node
* dm-aps-db-web-service.conf (_if included_)
- Comment out the entry for the principalAuthenticator2 which uses the LDAP authenticator
- Comment out the entry for the `principalAuthenticator2` which uses the LDAP authenticator
* dm-ds-web-service.conf
- Comment out the entry for the principalAuthenticator2 which uses the LDAP authenticator
- comment out the two lines for platformUtility which use LinuxUtility and LdapLinuxPlatformUtility
- Add a new platformUtility line in place of the other two
- Comment out the entry for the `principalAuthenticator2` which uses the LDAP authenticator
- comment out the two lines for `platformUtility` which use LinuxUtility and LdapLinuxPlatformUtility
- Add a new 'platformUtility` line in place of the other two
- Change value for `manageStoragePermissions` in ExpermentManager section to False
...
...
@@ -144,13 +146,13 @@ For initial test/development purposes, a few changes are necessary to short-circ
##### On the exp-station Node
* dm-cat-web-service.conf
- Comment out the entry for the principalAuthenticator2 which uses the LDAP authenticator
- Comment out the entry for the `principalAuthenticator2` which uses the LDAP authenticator
* dm-daq-web-service.conf
- Comment out the entry for the principalAuthenticator2 which uses the LDAP authenticator
- Comment out the entry for the `principalAuthenticator2` which uses the LDAP authenticator
* dm-proc-web-service.conf
- Comment out the entry for the principalAuthenticator2 which uses the LDAP authenticator
- Comment out the entry for the `principalAuthenticator2` which uses the LDAP authenticator
* dm-ds-web-service.conf
- Comment out the entry for the principalAuthenticator2 which uses the LDAP authenticator
- Comment out the entry for the `principalAuthenticator2` which uses the LDAP authenticator
After these modifications the services should be restarted:
...
...
@@ -161,14 +163,15 @@ After these modifications the services should be restarted:
### Overview of the sytem & tools
The installed development system has a few tools for managing the system. This section describes some of the available tools and process ideas for the system. The next section will describe some steps to walk through final setup and use.
- A web portal which should now be up and running at the URL https://localhost:8181/dm. This portal is powered by a Payara application server which has its own setup page at https://localhost:4848 (once configured above, you may not need to do much with the Payara config page).
- A PyQt app installed dm-station-gui which can be used to setup/monitor experiment definition, file trasfers and data workflows.
- A set of command-line scripts for manipulating the system. THese commands are made accessible by sourcing the file DM_INSTALL_DIR/etc/dm.setup.sh (Note there are some definitions that are blank in the default version of this file).
- A web portal which should now be up and running at the URL https://data-storage:8181/dm. This portal is powered by a Payara application server which has its own setup page at https://localhost:4848. Once configured above, you may not need to do much with the Payara config page.
- A set of command-line scripts for manipulating the system. These commands are made accessible by sourcing the file DM_INSTALL_DIR/etc/dm.setup.sh on the exp-station. (Note there are some definitions that are blank in the default version of this file).
- A PyQt app installed on the exp-station, dm-station-gui which can be used to setup/monitor experiment definition, file trasfers and data workflows.
- There are also a couple of underlying databases holding the data.
- A postgresql database which holds standard data such as user info, beamline/station definitions, experiments, access info linking users to experiments and data.
- A mongo database, which allows a bit more flexibility. This stores info on workflows and file information.
- An interface to the mongo database is available via a mongo-express web server on https://exp-station:18182
To start with the Data Management (DM) System is configured with one user __dm__ which is a management account, the third account listed above. One of the first items to handle is to create accounts that will be associated with managing the beamline setup and some (possibly the same accounts) that will be associated with experiments. In practice, the DM system is loosely linked to the list of users in the APS Proposal/ESAF system. Accounts on the ESAF system are coordinated with a list of users on the DM system. This is done by using the dm-update-users-from-aps-db. This will require a configuration file (find a good place to put the file). One other possibility is to create users manually from the supplied web portal. Note that, in the ESAF system, the user name is the badge number of the individual, while in the DM system a 'd' is prepended to the badge number for the user name.
To start with the Data Management (DM) System is configured with one user __dm__ which is a management account. One of the first items to handle is to create accounts that will be associated with managing the beamline setup and some (possibly the same accounts) that will be associated with experiments. At APS, the DM system is loosely linked to the list of users in the APS Proposal/ESAF system. Accounts on the ESAF system are coordinated with a list of users on the DM system. This is done by using the dm-update-users-from-aps-db. This will require a configuration file. One other possibility is to create users manually from the supplied web portal. Note that, in the ESAF system, the user name is the badge number of the individual, while in the DM system a 'd' is prepended to the badge number for the user name.
Once users have been added to the system, the DM web portal can be used to associate users with a beamline or with experiments that are created. The __dm__ user can be used to log into the web portal and from the _Experiment Stations_ tab new stations can be added or existing stations, such as the test station, can be edited and station managers can be added. To create experiments, station managers can log into the system and add/manage experiments for that station. From the test installation the user can manually create experiments & add users to the experiment. In practice, at the APS, when a user adds an experiment they are provided with a list of experiments from the proposal system and the list of users is populated from the (Proposal/ESAF ??) info. Note that it is also possible to add/modify experiments either through the dm-station-gui or through the command line interface with commands such as dm-add-experiment or dm-update-experiment.
...
...
@@ -189,9 +192,11 @@ Each step in a workflow can define inputs and outputs which can then be used in
If needed the test system can be restarted running a couple of startup commands. Change directory the DM install directory and then
This may be necessary if, for instance, the system has been rebooted. These commands restart several services in the install directory. If you have modified something in only one of these services you may be able to restart that service. For instance if only the data storage web service needs to be rebooted then you can run
...
...
@@ -202,18 +207,18 @@ This may be necessary if, for instance, the system has been rebooted. These com
As mentioned earlier, after the inital install we have one user __dm__ which is intended to be for the overall system. We now need to set up a user for administration of a beamline and start some steps to use the sytem.
You should at this point have a directory installed which has both the _Data Manangement_ and _support_ software installed. After doing the installs described above there should be a number of other directories as well such as etc, log and var. We are now going to walk through changes needed in the etc directory which will allow us to interact with the system.
1. source the file _etc/dm.setup.sh_. This defines a number of environment variables and modifies the path to include, in particular, a number of commands beginning with __dm-__ which interact with the underlying system to add/modify users, experiments, upload and daq (both to move files) and workflows and processes (to define & monitor processing of the collected data).
1. source the file _etc/dm.setup.sh_. For now, this will be done on both nodes. This defines a number of environment variables and modifies the path to include, in particular, a number of commands beginning with __dm-__ which interact with the underlying system to add/modify users, experiments, upload and daq (both to move files) and workflows and processes (to define & monitor processing of the collected data). Normally, you will only do this on exp-station since most operations will be done there.
- source etc/dm.setup.sh
2.add a user __dmtest__ to the system which will assume the role of manage what is going on in the system.
2. Create a user __dmtest__ and add a system role to make this user a manager of the station __TEST__. This will need to be done on the data-storage node since these commands access the postgresql database directly.
- dm-add-user --username dmtest --first-name DM --last-name Test --password dmtest
3. add a systemrole to the created user __dmtest__ to make this a manager of the station TEST which is already defined in the system. You will be asked to provide username & password. Use username __dm__ system account and the password given during setup above.
- dm-add-user-system-role --role Manager --station TEST --username dmtest
4. create a file, _etc/.dmtest.system.login_, in the same directory as the dm.setup.sh). This will contain the username & password.
- dmtest|dmtest (example contents)
5. Edit the file _etc/dm.setup.sh_, the one from step 1, to modify the line DM\_LOGIN\_FILE to point at the file created in step 4.
- dm-add-user-system-role --role Manager --station TEST --username dmtest
3. Make the dmtest user the default account used to execute the dm system commands on exp-station.
- create a file, _etc/.dmtest.system.login_, in the same directory as the dm.setup.sh). This will contain the username & password.
- dmtest|dmtest (example contents)
- Edit the file _etc/dm.setup.sh_, the one from step 1, to modify the line DM\_LOGIN\_FILE to point at the file created in step 4.
- DM\_LOGIN\_FILE=/home/dmadmin/etc/.dmtest.system.login (modified in file)
6. Re-source the setup file from step 1.
- source etc/dm.setup.sh
- Re-source the setup file from step 1. This is only necessary on exp-station.
- source etc/dm.setup.sh
At this point we will are more in a position to start using the sytem. As a first test we will add a few test users to the system and then run the command dm-test-upload which will
