Newer
Older
## APS Data Management System Developer Installation
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
The APS Data Management system makes use of many tools (Java, Python, Postgress, MongoDB, ZeroMQ, etc.). The Management System itself is built on top of these tools. While it is possible to install and user the underlying tools using more conventional means (e.g. RPM installation on Linux) scripts are provided that installs these tools from either source or binary builds and installs & configures them specifically for use with the Data Management System. These scripts can be found in a git repository at:
[https://git.aps.anl.gov/DM/dm-support.git](https://git.aps.anl.gov/DM/dm-support.git)
while the code that makes up the Data Management System itself can be found in a repository at:
[https://git.aps.anl.gov/DM/dm.git](https://git.aps.anl.gov/DM/dm.git)
### Installation for development
An example of setting up the Data Management system for a developer is described here.
- Create a directory (referred here as DM\_INSTALL\_DIR
> mkdir -p /local/Data Management
- Change directory into DM\_INSTALL\_DIR
> cd /local/Data Management
- Install a copy of the code from each of these git repositories in DM\_INSTALL\_DIR. This can be done in a variety of ways (3rd an 4th should be the most common)
- Grab a zip file from the APS Gitlab website (from URLs above) and unzip the file.
- Clone the repositories directly into DM\_INSTALL\_DIR (basically like cloning a forked repo shown below)
- Fork the repository following the fork link in the top right of the project page and then clone the repository as shown below. The example shown clones the dm-support repository into a directory __support__ and the dm repository into a directory __dev__. In each case the clone is pulled from the user _USERNAME_'s fork of the repository.
- Create a branch of the repository from the project web page and then clone the repository from the branch (similar to clown shown below)
> git clone https://git.aps.anl.gov/_USERNAME_/dm-support.git __support__
>
> git clone https://git.aps.anl.gov/_USERNAME_/dm.git __dev__
- Change directory into the __support__ directory
> cd support
- Install & build all of the components needed to build the development system running the script _install/_support/_all.sh_ in the _sbin_ directory.
- During this install/build you will need to provide two passwords for the adminstration of the __Payara__ application server. These passwords are for the _master_ (for administration of the keystore) and _admin_ (for administration of the application server properties) user accounts.
- Note that a number of the installed applications/libraries are built during the process so it is common that this process will possibly take a couple of hours to complete, but this is a one time installation process, although individual components can then be updated separately later.
- There is a configuration build_env.sh file which allows changing things like which version of each package will be installed. This is executed at the beginning of each script that will be run by install_support_all.sh. At any time, the current version of these tools may change to adapt for a new provided feature or to just ensure that new builds use the latest possible version of a tool to avoid a stale environment which falls far behind the current version of each tool.
> ./sbin/install/_support/_all.sh
- Change directory to the root of the Data Management components
- Note some configuration can be changed before processing the as discussed below. There are two files **dm_dev.deploy.conf** and **dm.deploy.conf** which define some environment variables used in the scripts used to install & configure. For the test deployment, **dm_dev.deploy.conf** is used.
> cd ../dev
- Execute the dm/_deploy/_test/_system.sh file in the sbin directory
- Like installing the support tools, this script builds and installs several components of the DM system so it will take some time to complete.
- This deploy process will prompt for user input at several points in the process.
- passwords for several accounts
- __postgres__ admin account - This will be used to manage the postgres itself. Each developer can set this to a unique value.
- __dm__ db management account - This will be for mananging the 'dm' database in postgres. Each developer can set this to a unique value.
- __dm__ system account - This is user __dm__in the Data Management system. This user has administrative priviledge in the Data Management system. This is a user in the 'dm' user table. Each developer can set this to a unique value.
- __dmadmin__ LDAP password - This password provides the Data Management software access to the APS/ANL LDAP system to gather reference to that database. This is a password to an external system and and is therefore a pre-existing password that developers will need to get from the Data Management system administrator.
- __dmadmin__ BSS login password. This is a password to allow the Data Management system access to the APS Beamline Scheduling system. This is a password to an external system and and is therefore a pre-existing password that developers will need to get from the Data Management system administrator.
- __dmadmin__ ESAF DB password. This is a password to allow the Data Management system access to the ESAF system. This is a password to an external system and and is therefore a pre-existing password that developers will need to get from the Data Management system administrator.
- Scripts in the Data Management system need a location for the data storage directory. Files will be moved to/from this directory.
### Removing Test system
Often in the development of Data Management system components it will be necessary to remove/reload components of the system. The script _dm/_remove/_test/_test/_system.sh_ in the sbin directory of the 'dm' repository (/local/DataManagement/dev/sbin from the directory describe above) issues commands to clear out database & configurations to allow creating a clean installation of the system.