2-IOC-deployment.md

IOC Deployment and Troubleshooting

[previous section] [next section]

2 Deploy a new IOC

2.1 Clone xxx

$ cd ${IOC_DIR}
$ git clone https://git.aps.anl.gov/practical_beamline_controls_training/session_2/xxx.git

output

Cloning into 'xxx'...
remote: Enumerating objects: 222, done.
remote: Counting objects: 100% (222/222), done.
remote: Compressing objects: 100% (141/141), done.
remote: Total 222 (delta 65), reused 222 (delta 65), pack-reused 0
Receiving objects: 100% (222/222), 177.75 KiB | 292.00 KiB/s, done.
Resolving deltas: 100% (65/65), done.

2.2 Confirm the directories exist

$ cd ${IOC_DIR}/xxx
$ tree -d .

output

.
├── configure
├── docs
├── iocBoot
│   └── iocxxx
│       ├── autosave
│       ├── examples
│       │   └── detectors
│       ├── softioc
│       │   └── commands
│       └── substitutions
└── xxxApp
    ├── Db
    ├── op
    │   ├── adl
    │   ├── bob
    │   │   └── autoconvert
    │   ├── burt
    │   ├── edl
    │   │   └── autoconvert
    │   ├── opi
    │   │   └── autoconvert
    │   ├── python
    │   └── ui
    │       └── autoconvert
    └── src

25 directories

2.3 Choose a new IOC prefix

A desired prefix can be specified as an argument to newPrefix.sh

$ ./newPrefix.sh kmp

output

New IOC name   = kmp
New IOC prefix = kmp:
Creating a custom caQtDM screen for kmp
Creating a custom MEDM screen for kmp
Creating override.iocsh to override the IOC prefix

If no prefix is specified, one will be chosen at random.

$ ./newPrefix.sh

output

New IOC name   = aoi
New IOC prefix = aoi:
Creating a custom caQtDM screen for aoi
Creating a custom MEDM screen for aoi
Creating override.iocsh to override the IOC prefix

2.4 Build the IOC (Attempt #1)

$ make

output

configure/CONFIG:18: *** EPICS_BASE must be set in a configure/RELEASE file.  Stop.

what does this mean?

This error message indicates that the EPICS_BASE environment variable is not defined in your configure/RELEASE file.

EPICS_BASE is a mandatory environment variable that points to the location of the EPICS base installation directory. The configure/RELEASE file is used to define EPICS-related environment variables required for building an IOC.

Here's an example of how to set the EPICS_BASE variable in the configure/RELEASE file:

EPICS_BASE=/path/to/epics/base

where "/path/to/epics/base" points to your EPICS base installation directory.

Once you have set the EPICS_BASE variable in the configure/RELEASE file, try building your IOC again and the error should be resolved.

2.5 Resolve build error #1

Modify configure/RELEASE so that EPICS_BASE is defined.

$ gedit configure/RELEASE &

solution

EPICS_BASE doesn't need to be defined explicitly; The synApps configure/RELEASE file defines the version of base used to build it, so only the SUPPORT path needs to be corrected.

$ git diff
diff --git a/configure/RELEASE b/configure/RELEASE
index 474fd69..e7aba0e 100644
--- a/configure/RELEASE
+++ b/configure/RELEASE
@@ -26,7 +26,7 @@
 ##############################################################################
 
 # Define the path to the synApps support directory and uncomment.
-SUPPORT=/corvette/home/epics/devel
+SUPPORT=/APSshare/epics/synApps_6_2_1/support
 
 # Specify the standard synApps configuration.
 -include $(SUPPORT)/configure/RELEASE
$

what does this mean?

We are using a synApps release of EPICS, which is a set of pre-built support modules for EPICS. The SUPPORT variable is defined to point to the directory where the synApps support modules are installed: /APSshare/epics/synApps_6_2_1/support.

The -include directive is used to include the configure/RELEASE file from the synApps support module directory, which sets various EPICS-related environment variables, including EPICS_BASE, as needed for building the IOC.

Therefore, we do not need to explicitly set the EPICS_BASE variable in your configure/RELEASE file because it is already set in the synApps support module's configure/RELEASE file.

2.6 Build the IOC (attempt #2)

If following the build succeeds, skip to step 2.9

$ make

output

If you're logged into the bcda VMs, it is likely you got the following error.

<snip>
/bin/ld: cannot find -lnet
/bin/ld: cannot find -lpcap
collect2: error: ld returned 1 exit status
make[3]: *** [xxx] Error 1
make[3]: Leaving directory `/home/beams/USERNAME/PET-S2/epics/synApps_6_2_1/ioc/xxx/xxxApp/src/O.rhel7-x86_64'
make[2]: *** [install.rhel7-x86_64] Error 2
make[2]: Leaving directory `/home/beams/USERNAME/PET-S2/epics/synApps_6_2_1/ioc/xxx/xxxApp/src'
make[1]: *** [src.install] Error 2
make[1]: Leaving directory `/home/beams/USERNAME/PET-S2/epics/synApps_6_2_1/ioc/xxx/xxxApp'
make: *** [xxxApp.install] Error 2

what does this mean?

This error message indicates that the linker (ld) is unable to find two libraries: net and pcap. When building your IOC (Input/Output Controller) using the make command, the linker is responsible for resolving and linking all the required libraries. The -l flag is used to specify libraries that need to be linked. In this case, the linker is looking for net and pcap. However, it is unable to locate these libraries, which results in the error.

2.7 Resolve build error #2

Tell the build system the net (and pcap) libraries aren't installed.

$ echo "LINUX_NET_INSTALLED = NO" > configure/CONFIG_SITE.local

what does this mean?

The configure/CONFIG_SITE.local file is used to specify local configuration settings for building an EPICS IOC. By setting LINUX_NET_INSTALLED to NO, you are disabling the use of the net library by telling the build system that this library is not installed on your system, which was causing the linker errors.

The IOC build system will now use an alternative networking library instead of libnet, allowing the IOC to be built without requiring the installation of the net library.

Note: the linux command $ echo "LINUX_NET_INSTALLED = NO" > configure/CONFIG_SITE.local writes the text "LINUX_NET_INSTALLED = NO" to the configure/CONFIG_SITE.local file. If the file does not exist, the > sign creates the file and writes the output to it. If the file already exists, the > sign overwrites its contents with the new output.

2.8 Build the IOC

We do a clean build at this point to ensure the fix for the previous failed build takes effect. The value that was set in CONFIG_SITE.local causes some software to be exluded from the IOC. Without the make distclean the excluded software can be partially included in the IOC, which results in build problems.

$ make distclean
$ make

2.9 Run the IOC

Run the IOC.

$ cd ${IOC_DIR}/xxx/iocBoot/iocxxx
$ ./softioc/xxx.sh run

2.10 Resolve the runtime errors

Errors that occur when commands in an IOC's startup script are executed are usually critical.

2.10.1 Identify critical runtime errors

Scroll back until the you see the first error message. Resolving the first error message can often eliminate many other error messages that appear unrelated.

2.10.2 Stop the IOC

Exit the IOC from the IOC shell:

kmp> exit

Note: typing ctrl+c will also stop the IOC.

2.10.3 Resolve critcial runtime errors

Resolve the problem that is causing the first error message, which accurately describes the problem.

hint

The envPaths file is generated by the build system and is affected by the Makefile in the IOC's startup directory, ${IOC_DIR}/xxx/iocBoot/iocxxx, which is also the current working directory.

solution

Edit the Makefile and change linux-x86_64 to the EPICS_HOST_ARCH that was set in step 1.1.2.

$ gedit Makefile &

Git can show you the change you made to the file.

$ git diff Makefile 
diff --git a/iocBoot/iocxxx/Makefile b/iocBoot/iocxxx/Makefile
index fa87a45..1e2f07a 100644
--- a/iocBoot/iocxxx/Makefile
+++ b/iocBoot/iocxxx/Makefile
@@ -7,10 +7,11 @@ include $(TOP)/configure/CONFIG
 #ARCH = windows-x64
 #ARCH = cygwin-x86
 #ARCH = linux-x86
-ARCH = linux-x86_64
+#ARCH = linux-x86_64
 #ARCH = vxWorks-ppc32
 #ARCH = vxWorks694-ppc32
 #ARCH = vxWorks-ppc603
+ARCH = $(EPICS_HOST_ARCH)
 
 #TARGETS = cdCommands
 TARGETS = envPaths

Once the Makefile is corrected, issue a 'make' in the current working directory, ${IOC_DIR}/xxx/iocBoot/iocxxx, to create the envPaths file.

$ make
perl -CSD /APSshare/epics/base-7.0.4.1/bin/rhel8-x86_64/convertRelease.pl -t /home/beams/USERNAME/PET-S2/epics/synApps_6_2_1/ioc/xxx envPaths

what does this mean?

Perl is a programming language commonly used for scripting and text processing. The command is essentially invoking the convertRelease.pl script and passing the path to the envPaths file as an argument. The script will process the envPaths file and perform any necessary conversions to set up the necessary environment variables.

2.10.4 Run the IOC

Start the IOC and confirm the error is resolved.

$ ./softioc/xxx.sh run

There will be autosave errors due to missing .sav files. Let the IOC run for at least 30 seconds and it will create the .sav files. Restart the IOC and the autosave errors will be gone.

Note: at this point the IOC is usable.

2.11 Resolve iocInit errors

Errors that occur during iocInit are not always critical and can sometimes be ignored, but it is good to eliminate them if possible.

2.11.1 Identify iocInit errors

These findUioAddr errors occur at iocInit:

findUioAddr: Failed to open '/sys/class/uio/uio0/name'
findUioAddr: Failed to open '/sys/class/uio/uio0/name'
findUioAddr: Failed to open '/sys/class/uio/uio0/name'

The errors come from softGlueZynq, which isn't needed in general purpose Linux IOCs.

2.11.2 Stop the IOC

Exit the IOC from the IOC shell:

kmp> exit

Note: typing ctrl+c will also stop the IOC.

2.11.3 Resolve iocInit errors

Remove softGlueZynq by doing the following: