Implementing cutting-edge image processing to ensure image based control on ultra-fast processes

Euresys frame grabbers and their CustomLogic capability have enabled a team of researchers from Columbia University, Drexel University, Fermilab, and Lehigh University to develop new capabilities for fusion energy by greatly reducing the time between signal acquisition and control command in a camera-based high-speed tokamak plasma control loop.
In another example of “Vision-in-the-Loop”, plasma control requirements reach unprecedented speed for such complex control logic.

In their paper, the research team described how they processed fast camera data, at rates exceeding 100,000 fps, on in-situ Field Programmable Gate Array (FPGA) hardware for plasma control at the HBT-EP fusion experiment at Columbia University . The purpose is to track magnetohydrodynamic (MHD) instability evolution and generate control requests in real-time. To achieve that the team implemented a Convolutional Neural Network (CNN) model on the frame grabber’s built-in FPGA, allowing them to achieve a trigger-to-output latency of 17.6 µs and throughput up to 120,000 fps.

FIGURE 3 – Data rate versus Computation time of various processes – Fast ML for Science – © N. Tran

THE SELECTED SOLUTION
The hardware platform consisted of:

• The AMETEK Phantom S710™ camera (Up to 7 Gpx/second (87.5 Gbps) of streaming capability e.g. 7,275 fps at 1,280 x 800,
• Euresys Coaxlink Octo frame grabber accommodating outputs of the camera.

In this case, channeling the data to a CPU, or even GPU, for central processing would not have met the responsiveness requirements of the application, which are in the 10 µs range.
The possibility of using Euresys’ CustomLogic offered researchers the unique opportunity of programming their own plasma control algorithms (CNN inference) directly in the Coaxlink Octo frame grabber to achieve latencies commensurate with the plasma dynamics.

Programming the Coaxlink CustomLogic can be done using AMD Vivado™ development tools. The FPGA resources are allocated for three main usages:

• The CoaXPress protocol,
• The internal card management,
• The user defined CustomLogic, in this instance the Neural Network model of the control algorithm and the control output request computation.

The team took on the challenge of implementing a more powerful control algorithm than usual while meeting the resource constraints of the frame grabber FPGA chip. To this end, they have used the open-source High-Level Synthesis for Machine Learning (hs4ml)[iii] package to translate and optimize the model before synthesizing and deploying it to the frame grabber’s FPGA.
The Coaxlink Octo I/Os interface directly with the plasma control system (external hardware in development), and image data are transferred to the host computer via the PCIe bus for post-processing outside of the control loop.

The implementation ultimately achieved a latency of 7.7 µs for the CNN model itself, 17.6 µs total for the acquisition trigger to output control request, and is able to run at pipelined frame rates of up to 120 kfps, meeting the requirements for real-time control of MHD instabilities in the tokamak.

THE TOOLKITS

In addition to AMD Vivado™ tools, the developers are extensively assisted by additional Euresys tools such as:

• General Purpose I/O Interface, enabling easy benchmarking of firmware model latency and throughput via 10MHz RS422 differential IO (shown above).
• Memento Event Logging Tool, providing a precise timeline of time-stamped events, along with context information and logic analyzer view. A valuable assistance during application development and debugging.
• Control/Status interface, allowing the user to read and write registers inside the user logic via the Coaxlink Driver API.
• The Reference Design delivered with the Coaxlink CustomLogic SDK and intended to be used as a template. The reference design exposes all interfaces available to the user.