create wiki template

71727386 · hparraga · ae1437b1 · 71727386
Commit 71727386 authored 2 years ago by hparraga
--- a/BeamlineDeploymentWikiTemplate.md
+++ b/BeamlineDeploymentWikiTemplate.md
+### To generate a wiki, find and replace these 6 lines, then delete this section
+BEAMLINE_NAME = 12-ID-D
+INSTALL_DIR = /home/dm_idd
+STATION_NAME = 12IDD
+DEPLOYMENT_MACHINE = s12idddm.xray.aps.anl.gov
+STORAGE_MACHINE = xraydtn02.xray.aps.anl.gov
+BEAMLINE_ACCOUNT = user12idd
+
+# BEAMLINE_NAME Deployment
+
+## Quick Start Guide
+In this section we illustrate typical system usage via the DM Station GUI (see the [Getting Started](DM/HowTos/Getting-Started) guide for more details). Note that INSTALL_DIR directory should be accessible on all BEAMLINE_NAME machines.
+
+1. Setup environment by sourcing setup file.
+      `$ # After sourcing setup file, all DM commands will be in your path`
+
+      `$ # For users using C shell`
+
+      `$ source INSTALL_DIR/etc/dm.setup.csh`
+
+      `$ # For users using bash shell`
+
+      `$ source INSTALL_DIR/etc/dm.setup.sh`
+
+2. Run the "dm-station-gui" command. This will allow you to create an experiment, upload data to the central storage system, see experiment files, etc.
+      `$ dm-station-gui`
+3.    Access files using Globus Online (see the [Getting Data From Globus](DM/HowTos/Getting-Data-From-Globus) guide for more details). Note that experiment files can be read only by those DM usernames that have been assigned a role within a given experiment.
+  a. Using your browser, open https://www.globus.org/
+  b. Sign in with your Globus username
+  c. Under File Manager, chose "aps#data" as Collection and click "Continue" in order to authenticate.
+  d. Authenticate with your DM username (typically "d<badge number>"). Use the same password (APS Web Password) that you use to access APS Beamline User Portal.
+  e. Navigate to experiment folder. All experiments will be under the "/gdata/dm/STATION_NAME" directory (e.g. "/gdata/dm/STATION_NAME/test-STATION_NAME-01)".
+  f. At this point you should be able to transfer files to any Globus endpoint that you have access to. Note that downloading files to your desktop or laptop requires installing [Globus Connect Personal](https://www.globus.org/globus-connect-personal) endpoint.
+
+## Command Lines
+
+In this section we list few most important DM commands that can be used from a terminal session or in scripts.
+
+1. Create an experiment and upload data to the central storage system:
+```
+    $ # Experiment name must be unique and start with a letter.
+    $ # Users can be added to experiment by specifying Beamline Scheduling System proposal id (--proposal-id option),
+    $ # or a list of users directly (--users option)
+    $ dm-STATION_NAME-upload --experiment=test-STATION_NAME-01 --data-directory=INSTALL_DIR/test
+```
+2. Check status of your uploads:
+```
+    $ dm-list-uploads
+```
+3. Start monitoring directory for new files (DAQ mode):
+```
+    $ # Any files already present in the specified data directory will be ignored.
+    $ # The --dest-directory option indicates directory path relative to experiment root directory in storage
+    $ # Users can be added to experiment by specifying Beamline Scheduling System proposal id (--proposal-id option)
+    $ # or a list of users directly (--users option)
+    $ dm-STATION_NAME-daq --data-directory=INSTALL_DIR/test --experiment=test-STATION_NAME-01 --dest-directory=daqdata --duration=1h
+```
+4. Check status of your DAQs:
+```
+    $ dm-list-daqs
+```
+5. Retrieve list of your cataloged file metadata, etc. Alternative way of viewing metadata catalog is via Mongo Express application ([https://DEPLOYMENT_MACHINE:8182/db/dm](https://DEPLOYMENT_MACHINE:8182/db/dm)).
+```
+    $ dm-list-experiment-files --experiment=test-STATION_NAME-01
+```
+6. Add or remove users associated with your experiment.
+```
+    $ # Use d<badge> when specifying usernames
+    $ dm-add-user-experiment-role --username=d225159 --experiment=test-STATION_NAME-01 --role=User
+    $ dm-delete-user-experiment-role --username=d225159 --experiment=test-STATION_NAME-01 --role=User
+```
+
+## Python API
+
+The DM Python API documentation is located in the INSTALL_DIR/production/doc/html folder, while some simple examples can be found under the INSTALL_DIR/production/examples directory.
+
+
+1. After sourcing the DM setup file, DM Python will be in user's path and DM Python APIs will be in user's PYTHONPATH environment variable.
+```
+    $ # This command brings up the DM API documentation
+    $ firefox INSTALL_DIR/production/doc/html/index.html
+    $ # Retrieve list of existing experiments
+    $ python -c "from dm import ExperimentDsApi; api = ExperimentDsApi(); print(api.getExperimentsByStation())"
+```
+2. DM Python APIs can also be installed in a Conda environment:
+```
+    $ # Install conda in a fresh terminal with a clean environment.
+    $ # Do NOT source regular DM setup file.
+    $ curl -o Miniconda3-latest-Linux-x86_64.sh  https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
+    $ sh Miniconda3-latest-Linux-x86_64.sh -b -p $PWD/CONDA
+
+    $ # Install conda DM API package
+    $ export PATH=$PWD/CONDA/bin:$PATH
+    $ conda install -c aps-anl-tag aps-dm-api
+
+    $ # Source dm.conda.setup.sh file, which does not modify PYTHONPATH environment variable
+    $ source INSTALL_DIR/etc/dm.conda.setup.sh
+
+    $ # Retrieve list of existing experiments
+    $ python -c "from dm import ExperimentDsApi; api = ExperimentDsApi(); print(api.getExperimentsByStation())"
+```
+
+
+## System Administration
+
+DM services are controlled using standard RHEL daemon scripts located under the INSTALL_DIR/production/etc/init.d directory. Relevant scripts are:
+
+- dm-daq-web-service (DAQ web service)
+- dm-cat-web-service (Cataloging web service)
+- dm-proc-web-service (Processing web service)
+- dm-mongodb (MongoDB server)
+- dm-mongo-express (Mongo Express web application)
+- dm-nginx (Nginx server)
+- dm-daq-services (controls all DAQ services)
+
+Shown below is an example of DAQ Web Service restart:
+```
+   $ # Control scripts can be found in INSTALL_DIR/production/etc/init.d directory.
+   $ # Usage:
+   $ INSTALL_DIR/production/etc/init.d/dm-daq-web-service
+   Usage: INSTALL_DIR/production/etc/init.d/dm-daq-web-service {start|stop|status|restart}
+
+   $ # Restart DAQ Web Service
+   $ INSTALL_DIR/production/etc/init.d/dm-daq-web-service restart
+   Stopping dm.DaqWebService daemon:                          [  OK  ]
+   Starting dm.DaqWebService daemon:                          [  OK  ]
+   $ INSTALL_DIR/production/etc/init.d/dm-daq-web-service status
+   dm.DaqWebService (pid  29314) is running...
+```
+
+
+## Deployment Details
+
+- DM Station Name: STATION_NAME
+- DM Software Version: [4.1.1](https://git.aps.anl.gov/DM/data-management/-/tree/4.1.1) ([Release Notes](https://git.aps.anl.gov/DM/data-management/blob/4.1.1/doc/RELEASE_NOTES.txt))
+- Deployment Machine: DEPLOYMENT_MACHINE
+- Deployed Services:
+  - Data Acquisition Service (port 33336)
+  - Cataloging Service (port 44436)
+  - Processing Service (ports 55536)
+  - MongoDB Server (localhost, port 27017)
+  - Nginx (port 8182)
+  - Mongo Express (localhost, port 18182)
+- DM Admin Unix Account: dmadmin
+- DM Beamline System Account: BEAMLINE_ACCOUNT
+- Deployment Directory: INSTALL_DIR
+- Support Software Directory: INSTALL_DIR/support
+- System Configuration Files: INSTALL_DIR/etc
+- Service Log Directories:
+   - DM Backend Services: INSTALL_DIR/var/log
+- MongoDB Directory: INSTALL_DIR/support/opt/mongodb/linux-x86_64/data
+- User Environment Setup File: INSTALL_DIR/etc/dm.setup.sh
+- Service Control Scripts: INSTALL_DIR/production/etc/init.d
+- DM Storage Service: STORAGE_MACHINE:22236
+- DM APS DB Service: STORAGE_MACHINE:11236
+- DM Web Portal: [https://STORAGE_MACHINE:8181/dm](https://STORAGE_MACHINE:8181/dm)
+- Mongo Express: [https://DEPLOYMENT_MACHINE:8182/db/dm](https://DEPLOYMENT_MACHINE:8182/db/dm)
+- Globus Endpoint: aps#data
+- Globus Folder: /gdata/dm/STATION_NAME