From dc9bebfbe0944c433095d83b1e41814471d7a7cc Mon Sep 17 00:00:00 2001 From: JPHammonds <JPHammonds@anl.gov> Date: Mon, 9 Sep 2019 13:28:15 -0500 Subject: [PATCH] Add some things to check out after the test system is up and running. --- doc/Installation/APSDeveloperInstallation.md | 25 ++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/doc/Installation/APSDeveloperInstallation.md b/doc/Installation/APSDeveloperInstallation.md index e6cf0432..ded9b85f 100644 --- a/doc/Installation/APSDeveloperInstallation.md +++ b/doc/Installation/APSDeveloperInstallation.md @@ -62,3 +62,28 @@ An example of setting up the Data Management system for a developer is described ### Removing Test system Often in the development of Data Management system components it will be necessary to remove/reload components of the system. The script _dm/_remove/_test/_test/_system.sh_ in the sbin directory of the 'dm' repository (/local/DataManagement/dev/sbin from the directory describe above) issues commands to clear out database & configurations to allow creating a clean installation of the system. +### Testing the Installed System +The installed development system has a few tools for managing the system. + - A web portal which should now be up and running at the URL https://localhost:8181/dm. This portal is powered by a Payara application server which has its own setup page at https://localhost:4848 (once configured above, you may not need to do much with the Payara config page). + - A PyQt app installed dm-station-gui which can be used to setup/monitor experiment definition, file trasfers and data workflows. + - A set of command-line scripts for manipulating the system. THese commands are made accessible by sourcing the file DM_INSTALL_DIR/etc/dm.setup.sh (Note there are some definitions that are blank in the default version of this file). + - There are also a couple of underlying databases holding the data. + - A postgresql database which holds standard data such as user info, beamline/station definitions, experiments, access info linking users to experiments and data. + - A mongo database, which allows a bit more flexibility. This stores info on workflows and file information. + +To start with the Data Management (DM) System is configured with one user __dm__ which is a management account, the third account listed above. One of the first items to handle is to create accounts that will be associated with managing the beamline setup and some (possibly the same accounts) that will be associated with experiments. In practice, the DM system is loosely linked to the list of users in the APS Proposal/ESAF system. Accounts on the ESAF system are coordinated with a list of users on the DM system. This is done by using the dm-update-users-from-aps-db. This will require a configuration file (find a good place to put the file). One other possibility is to create users manually from the supplied web portal. Note that, in the ESAF system, the user name is the badge number of the individual, while in the DM system a 'd' is prepended to the badge number for the user name. + +Once users have benn added to the system, the DM web portal can be used to associate users with a beamline or with experiments that are created. The __dm__ user can be used to log into the web portal and from the _Experiment Stations_ tab new stations can be added or existing stations, such as the test station, can be edited and station managers can be added. To create experiments, station managers can log into the system and add/manage experiments for that station. From the test installation the user can manually create experiments & add users to the experiment. In practice, at the APS, when a user adds an experiment they are provided with a list of experiments from the proposal system and the list of users is populated from the (Proposal/ESAF ??) info. Note that it is also possible to add/modify experiments either through the dm-station-gui or through the command line interface with commands such as dm-add-experiment or dm-update-experiment. + +After defining an experiment, it is possible to then manage tasks such as file transfers (daq or upload) or workflows & processing jobs. These tasks can be done using either the dm-station-gui or by the command line interface. + +'daq' transfers monitor selected directories for files from a live data acquisition process from the collected location to a 'storage' location. 'upload' tranfers copy any existing files from the collected location to the 'storage' location. As file are transfered, they are placed into a storage directory with subdirectories for the _(station name)/(storage root path)/(experiment name)_. + +DM workflows define a sequence of commands that would operate on data sets to: + + - Stage data + - Move the data to a particular location such as a transfer between globus endpoints + - Process data for using reduction/analysis algorithms + - Add results to files that are tracked by Data Management + +Each step in a workflow can define inputs and outputs which can then be used in subsequent steps. -- GitLab