Prof. Dr. Torsten Höfler, ETH Zurich
Distributed memory programming is the established paradigm used in high-performance computing (HPC) systems, requiring explicit communication between nodes and devices. When FPGAs are deployed in distributed settings, communication is typically handled either by going through the host machine, sacrificing performance, or by streaming across fixed device-to-device connections thereby sacrificing flexibility. In this collaboration between PC² and the Scalable Parallel Computing Lab (SPCL) at ETH Zurich, we address the requirement for a distributed memory programming model for multi-FPGA systems.
We propose the Streaming Message Interface (SMI): a communication model and API which unifies explicit message passing with a hardware-oriented programming model, facilitating minimal-overhead, flexible, and productive inter-FPGA communication. Instead of bulk transmission, messages are streamed across the network during computation, enabling communication to be seamlessly integrated into pipelined designs. Using SMI, programmers are able to implement distributed, scalable HPC programs on reconfigurable hardware, without deviating from best practices for hardware design.
The unique hardware setup of the Noctua cluster at PC² enables us to prototype SMI on a dedicated FPGA interconnect. By utilizing point-to-point QSFP connections between the Nallatech 520N FPGA boards, we were able to implement runtime-configurable routing, with support for arbitrary network topologies. We have already successfully shown low-latency streaming communication between eight FPGAs using SMI, and continue to explore larger systems and network topologies due to the newly installed optical switches in the cluster.
Tiziano De Matteis, Johannes de Fine Licht, Jakub Beraneck, Torsten Hoefler. "Streaming Message Interface: High-Performance Distributed Memory Programming on Reconfigurable Hardware". Accepted at the International Conference for High Performance Computing, Networking, Storage and Analysis (SC'19)