6-asyn-troubleshooting.md

IOC Deployment and Troubleshooting

6 asyn troubleshooting

The EPICS asyn module provides a trace facility that allows communication between an IOC and a controller to be displayed or logged.

6.1 Start the IOC

$ cd ${IOC_DIR}/xxx/iocBoot/iocxxx
$ ./softioc/xxx.sh start

output

The script will prevent you from starting a second copy of an IOC.

xxx is already running (pid=1963381) in a screen session (pid=1963380)

6.2 How to enable asyn traces

Asyn traces can be enabled from the IOC's shell. An example of how to do this can be found in the example virtual motor controller configuration that was copied in a previous step:

# Only show errors
asynSetTraceMask("VMC_ETH", 0, 1)
# Leave ascii selected so traces can be turned on with a single click
asynSetTraceIOMask("VMC_ETH", 0, 1)

It is much easier, however, to enable asyn traces from MEDM/caQtDM if the IOC loaded at least one asyn record.

Do the following:

1. Launch caQtDM if it isn't already running

$ ./softioc/xxx.sh caqtdm

This launches my-xxx.ui:

2. Switch to the Direct I/O section of my-xxx.ui

3. From the Serial 1-8 menu of my-xxx.ui, choose asyn record serial1

This launches asynOctet.ui:

Note: the terminators should remain empty for now. They'll be corrected in step 6.4

4. From the More... menu of asynOctet.ui, choose Record Parameters

This launches asynRecord.ui:

The traceMask determines which messages are printed on the IOC's console:

• traceError should always be on
• traceIODriver is useful for looking at serial/ethernet communication (when connected to the correct asyn port)

The traceIOMask determines how the messages are printed on the IOC's console:

• traceIOASCII prints the messages as ASCII strings (this works well for messages that only include symbols that appear on a keyboard)
• traceIOEscape prints the messages and unprintable characters using escape sequences (starting with a backslash)
• traceIOHex prints the messages as a series of byte values in hexadecimal (without the 0x prefix)

Note: input and output message terminators are usually unprintable characters like carriage return, \r or 0x13, and line feed, \n or 0x10.

6.3 Observe controller communication

The commands used by the virtual motor controller are documented here.

Connect to the IOC's console

$ ./softioc/xxx.sh console

Turn traceIODriver on:

Do the following and observe the commands on the IOC's console:

• Note the commands sent while idle polling: POS and ST
• Move a motor and look for the MV command
• Stop a motor and look for the AB command

Typing ctrl+a, [ will cause screen to enter copy/scrollback mode at which point the arrow keys can be used to navigate up to commands that scrolled off the screen. More info about screen's copy/scrollback mode can be found here.

Note: The Abort button at the bottom of motorx_all.ui writes to an allstop PV, NOT directly to the motor record's STOP field; the motor has to be moving for the Abort button to cause the AB command to be sent to the controller.

6.4 Synchronize the terminators on asynOctet.ui

The terminator fields on asynOctet.ui appear to be blank, but we have evidence that they are not actually blank (we just saw successful communication between the IOC and virtual motor controller).

We can read the terminator values and see that they have the same values that were set in vmc.cmd:

kmp> asynOctetGetOutputEos VMC_ETH
"\r"
kmp> asynOctetGetInputEos VMC_ETH
"\r\n"

Set the output terminator to \r on asynOctet.ui and the input terminator will update automatically:

6.5 Manually send a command to the controller

The asynOctet screen allows commands to be sent to the controller manually.

Send the 1 POS? command to the controller to query m1's position (in controller units):

6.6 Stop the IOC

Exit the IOC from the IOC shell:

kmp> exit

alternate approach

Disconnect from the IOC:

kmp> ctrl+a, ctrl+d

Then stop the IOC:

$ ./softioc/xxx.sh stop

6.7 "Upgrade" the Virtual Motor Controller to a newer firmware.

6.7.1 Kill server.py

Type ctrl+c in the xterm window or click the X in the corner of the window.

6.7.2 Run server-new-fw.py

Reminder: the port argument passed to the -p option must match the value of VMC_PORT chosen in step 5.7.2

$ xterm -e "${SUPPORT_DIR}/motorVMC/python/server-new-fw.py -p 33333" &

output

Listening on port 33333

6.8 Run the IOC

Run the IOC

$ ./softioc/xxx.sh run

6.9 Observe the motor behavior with the new firmware

What is the first error that occurs when the IOC starts?

answer

If all the motor positions were zero, there may not be any errors when the IOC starts.

If a motor has a non-zero position, the first error is an asyn writeFloat64 error that occurs at iocInit when the motor's autosaved position is being restored:

2023/02/24 13:44:00.287 asynMotorController:writeFloat64 error, status=1 axis=0, function=8, value=400.000000
2023/02/24 13:44:00.287 devMotorAsyn::asynCallback: kmp:m1 pasyn{Float64,Int32}->write returned

The IOC doesn't autosave the asyn trace settings, so turn traceIODriver on again:

Other questions to answer:

• Is controller responding to the commands sent by the IOC?

answer

• Yes, the controller is responding to commands sent by the IOC, but each response takes much longer than it should.

• If the motor record's VAL field is changed, does the motor move?

answer

• Yes, but you can't tell from the motor screen. A controller error appears when a move is attempted, but the asyn trace shows that the MV command does get sent to the controller and the queried position is correct. The motor record does retries because the position query is not returning asynSuccess to the EPICS motor driver.

6.10 Enable traceIOEscape

Sometimes communication problems are caused by the characters you can't see. Enable traceIOEscape instead of (or in addition to) traceIOASCII

Both the printable and unprintable characters that are sent to/from the controller should now be visible on the IOC's shell.

6.11 Manually send a command to the controller

Send the 1 POS? command to the controller to query m1's position (in controller units) using asynOctet.ui.

How does the controller's response differ from before the firmware upgrade?

answer

• The trailing carriage return, \r, in the reply from the controller is new; it wasn't present before the firmware upgrade.
• The asyn record's STAT (status) field is no longer NO_ALARM; it is now READ, which indicates a read error.
• The asyn record's SEVR (severity) field is no longer NO_ALARM; it is now MAJOR, which indicates a significant error has occured.
• The asyn record's EOMR (end of message reason) field was previously Eos after a message was received; now it is None, which indicates the end of string characters, \r\n were not found.

6.12 Diagnose the communication problem

Why doesn't the controller communicate properly with EPICS after the firmware upgrade?

answer

The firmware upgrade changed the message terminator of the controller's replies. The asyn port's input terminator, IEOS, is set to \r\n but the controller now responds with only \r.

6.13 Confirm the communication problem

Correct the IEOS in on asynOctet.ui

solution

Changing the input terminator from \r\n to \r has an immmediate effect on the messages displayed on the IOC's console: the communication is no longer delayed and the driver resumes polling at the idle polling rate.

When the position is queried manually, the I/O status and severity return to NO_ALARM on asynOctet.ui:

6.14 Solve the communication problem

Autosave is saving the new input terminator, but the IOC's configuration should still be modified so the configuration is consistent with the autosaved values. Stop the IOC to get the command prompt back.

kmp> exit
$ gedit iocsh/vmc.cmd &

solution

Note: this is a situation where it is very important to include a comment explaining the change so that the next person who maintains this IOC doesn't need to wonder why the input terminator doesn't match the virtual motor controller documentation.

diff --git a/iocBoot/iocxxx/iocsh/vmc.cmd b/iocBoot/iocxxx/iocsh/vmc.cmd
index 3754207..b39d863 100644
--- a/iocBoot/iocxxx/iocsh/vmc.cmd
+++ b/iocBoot/iocxxx/iocsh/vmc.cmd
@@ -12,7 +12,10 @@ asynSetTraceMask("VMC_ETH", 0, 1)
 asynSetTraceIOMask("VMC_ETH", 0, 1)
 
 # Set end-of-string terminators
-asynOctetSetInputEos("VMC_ETH",0,"\r\n")
+# Original Firmware EOS
+#!asynOctetSetInputEos("VMC_ETH",0,"\r\n")
+# New Firmware EOS
+asynOctetSetInputEos("VMC_ETH",0,"\r")
 asynOctetSetOutputEos("VMC_ETH",0,"\r")
 
 # These motor records can get their prefix from the environment variable

6.15 Confirm the communication problem is solved

Restart the IOC:

$ ./softioc/xxx.sh run

Move the motor and confirm there isn't a controller error.

The End