## Setup of Development/Test Data Management System on Multiple Nodes
In a typical setup, it is necessary to install the Data Mangement System on multiple nodes. Centralizining overall long term data storage for instance would argue that the Data Storage Service on one, or possibly a small set of, server(s). On a given experiemnt, it may be necessary to have more than one DAQ node to deal with different detectors. This document will describe a two node setup. These nodes will be
* The data-storage node. This will provide the data storage service and Web Portal
* The data-storage node. This will provide the data storage service, a central database (which stores information on users, experiments, and beamline deployments) and Web Portal that allows some management of the sytem.
* The exp-station node. This will provide the _daq_, _proc_ and _cat_ web services which will manage moving data from the collection system to the storage system, processing the data as needed and cataloging steps in storage and processing.
### Computer setup.
In production at APS we are using RedHat Enterprise Linux 7 on all machines. For development we are using either RHEL 7 machines or CentOS 7 machines. In the case of CentOS, this work in typically done on a virtual machine using VirtualBox. When installing, we are typically selecting a devolopment workstation configuration as a starting point for work. In addition to this, a number of requirements have been put together and can be found [here](https://confluence.aps.anl.gov/display/DMGT/DM+Station+System+Requirements). When using VirtualBox, once the OS has completed this system can be cloned to make additional machines with the same configuration. It is therefore recommended to keep a copy of the VM to use as a starting point to repeat the work done.
In production at APS we are using RedHat Enterprise Linux 7 on all machines. For development we are using either RHEL 7 (centrally managed by IT group) machines or CentOS 7 machines (user managed and installed as a VirtualBox VM). When installing, we are typically selecting a devolopment workstation configuration as a starting point for work. In addition to this, a number of requirements have been put together and can be found [here](https://confluence.aps.anl.gov/display/DMGT/DM+Station+System+Requirements). When using VirtualBox, once the OS has completed this system can be cloned to make additional machines with the same configuration. It is therefore recommended to keep a copy of the VM to use as a starting point to repeat the work done.
The typical multiple node VM setup uses two network interfaces. These interfaces are configured in the VirtualBox setup. The first network interface is configured as a generic NAT connection which will allow the VM to access the public network in order to facilitate support tool downloads during installation and access to facility resources if it is required to extend the __DM__ system to connect to facility resources as the aps\_db\_web\_service does to provide access to systems such as the APS Experiment Safety Assment Form (ESAF), System and Beamline Scheduling System (BSS). The second network interface is configured as a 'Host-only Adapter' on the 'vboxnet0' network. This interface will be used to set up the systems to communicate with each other.
The typical multiple node VM setup uses two network interfaces. These interfaces are configured in the VirtualBox setup. The first network interface is configured as a generic NAT connection which will allow the VM to access the public network in order to facilitate support tool downloads during installation. This would allow also access to facility resources if it is required. This could be used to extend the __DM__ system to connect to facility resources such as the aps\_db\_web\_service which provides access to systems such as the APS Experiment Safety Assment Form (ESAF), System and Beamline Scheduling System (BSS). The second network interface is configured as a 'Host-only Adapter' on the 'vboxnet0' network. This interface will be used to set up the systems to communicate with each other.
The __DM__ System installation process will use the 'hostname -f' command to get the system name. The host name is used by the __DM__ system when launching services to make services available 'publicly' on the 'Host-only Adapter' network. This makes services available to the other VMs running on the 'vboxnet0' network. In order to recieve names for each system during network setup, the hostname must be set for each system. The system hostname on a CentOS system can be set with the hostnamectl command.
The __DM__ System installation process will use the 'hostname -f' command to get the system name. The host name is used by the __DM__ system when configuring services to make them available 'publicly' on the 'Host-only Adapter' network. This makes services available to the other VMs running on the 'vboxnet0' network. In order for the to recieve names for each system during network setup, the hostname must be set for each system. The system hostname on a CentOS system can be set with the hostnamectl command. In a multiple node environment VMs will also need some form of name resolution for the VM nodes in the system. This can be acheived by adding node entries in /etc/hosts file.
The system installed fro
...
...
@@ -21,6 +21,8 @@ These scripts can be found in the APS git repository at:
* Select an account (such as dmadmin) which will build, install and manage the __DM__ system.
* Select a parent location to install the system and create a subdirectory __DM__ to contain the __DM__ system and the support tools. We will refer to this directory in future sections as DM\_INSTALL\_DIR
* Install a copy of the code from the _support_ git repository in DM\_INSTALL\_DIR. This can be done in a variety of ways (3rd an 4th should be the most common)
* First we will install support tools needed by the data-storage node. To do this run the command `./sbin/install_support_ds.sh`. This will install postgresql, openjdk, ant, payara, python and a number of needed python modules. As this script runs, you will be prompted to provide passwords for the master and admin accounts for the Payara web server. These will be used to manage the Payara server, which will provide a portal for managing some parts of the DM.
* Next we will install support tools needed by the exp-station. To do this run the command `./sbin/install_support_daq.sh`. This will install python, a number of needed python modules, and Python 3 and the same associated python modules. Note that in the near future, this should become just Python 3 and Python 3 modules.
##### On data-storage node
Here will install support tools needed by the data-storage node. Again these tools will support the data storage service, a central database (which stores information on users, experiments, and beamline deployments) and Web Portal that allows some management of the sytem. To this extent, this will install postgresql, openjdk, ant, payara, python and a number of needed python modules.
* Run the command `./sbin/install_support_ds.sh`. This installation will take some time to complete as this will download, compile and configure a number of key tools.
* As this script runs, you will be prompted to provide passwords for the master and admin accounts for the Payara web server. Choose appropriate passwords & record these for later use. These will be used to manage the Payara server, which will provide a portal for managing some parts of the DM.
##### On exp-station node
Similar to the data-storage node, we will install support tools for the experiment station node. These tools will support the daq, proc & cat web services. This will facilitate managing file transfers during or after acquisition, processing data after collection and managing experiment meta-data. To support this this will download & install Python 2 and a number of associated modules and Python 3 and the same modules. Note, in the near future this should be just Python 3 versions.
* Run the command `./sbin/install_support_daq.sh`. This will take a some time as it downloads & compiles from source.
### Data Management component installation
...
...
@@ -42,7 +53,9 @@ Once again, we are installing two different systems, each with different parts o
To work along with the installation scripts in this package, this should be installed with a given structure. The contents of this repository should be cloned in the DM\_INSTALL\_DIR into a directory corresponding to a version tag. This allows the system to be updated in a way that allows updating the system in operation with a new versioned directory. Initially and as the system is updated a symbolic link directory in called _production_, in DM\_INSTALL\_DIR, should be redirected to a new version tag. Similarly, if it is discovered that fallback is necessary, then the link will be moved back to an older version. An example of this, is as follows, assuming as with the support module a fork of the _dm_ repository has been forked by a user. These steps should be followed on both _data-storage_ and _exp-station_ nodes.
The installation scripts for the DM System assume a particular directory structure. The contents of this repository should be cloned in the DM\_INSTALL\_DIR into a directory corresponding to a version tag. This allows the system to be updated in a way that allows updating the system in operation with a new versioned directory. Initially, and as the system is updated, a symbolic link called _production_, in DM\_INSTALL\_DIR, should be directed to the version tagged directory of _dm_. Similarly, if it is discovered that fallback is necessary, then the link will be moved back to an older version. An example of this, is shown in the image below. A stepped instruction for this, assuming as with the support module a fork of the _dm_ repository has been forked by a user, follows. These steps should be followed on both _data-storage_ and _exp-station_ nodes.
