How to Achieve Low-Latency Real-Time Distributed Test Execution

Testing distributed complex systems requires connecting many different components and subsystems together. As vehicles become more autonomous and incorporate additional sensors, solutions that require distributed processing are becoming more common. Modern model-based systems engineering (MBSE) techniques help accelerate development, but only if test facilities can keep up. The future of transportation demands faster and more connected test solutions.

In this blog post, we will explore Genuen’s solution for low-latency, real-time distributed test execution and how it addresses common challenges in real-time networking.

BLOG_How to Achieve Low-Latency Real-Time Distributed Test Execution_Social

 

The Data Exchange Challenge

One of our customers in the hyperloop industry required a distributed real-time solution that involved exchanging data between nodes at a rate of 10 KHz, within a hardware-in-the-loop (HIL) simulation application consisting of 40 nodes. Several nodes were needed to perform real-time calculations on data generated in real-time by other remote nodes in the network, and thus required ultra-low-latency data exchange.

This presented the challenge: how to achieve low-latency, real-time data distribution.

In typical transmission control protocol (TCP) communications, there are several tasks that need to be completed by the CPU in the time that it takes for an application to request data to when it is delivered by the network. This consumes significant processing power in both the transmitting and receiving nodes. Synchronization of distributed nodes usually requires expensive external hardware or is done at a much slower rate via time-based synchronization protocols that can decrease your testing throughput.

 

Real-Time Networking Options

While networking solutions exist for sharing test data, these have significant limitations that prevent seamless real-time data distribution across heterogeneous systems.

Reflective memory (RFM) cards provide a shared memory space between remote nodes but, between needed PCI cards and extra software engineering, add substantial implementation costs. These cards store data in memory that is external to the processing CPU. The data transfer needed from the card’s onboard memory to the host computer's memory introduces latency. The ring network architecture commonly used in RFM applications is fragile, as every node becomes a single point of failure.

Other solutions like Time-Sensitive Networking (TSN) focus on precise clock synchronization, not on the mechanics of data exchange. This centralized approach requires the test engineer to implement custom software data exchange messaging layers. Publish-subscribe protocols like Data Distribution Service (DDS) are decentralized, architecture-independent, and use User Datagram Protocol (UDP) for determinism but sacrifice guaranteed package delivery. Their added protocol overhead prohibits their use in applications that require loop rates faster than 1-2 kHz.

Remote Direct Memory Access (RDMA) allows direct read/write access between host memories via Ethernet. With RDMA, networked computers can exchange data without relying on either systems’ processor, cache, or operating system. In this exchange, a network interface card (NIC) does all the heavy lifting that is typically done by the CPU in TCP-based approaches. However, existing drivers only offer point-to-point read/write operations, leaving the test engineer with the heavy task of implementing the multicasting protocol and the data management stack.

Read about how Genuen is leveraging RDMA technology while enabling scalability of distributed HIL applications.

Despite advances in real-time networking, existing solutions still demand substantial integration effort or special hardware, and most result in vendor lock-in. To enable the next generation of distributed testing, a new approach is needed – one that delivers a universal memory abstraction across distributed systems in real-time.

Learn more about the limitations of today’s XIL systems in solving the problem of high-speed low-latency real-time network communications across multi-vendor systems.

We addressed our hyperloop customer’s challenge by implementing a custom FPGA-based point-to-point communication protocol. Despite successfully meeting the application’s requirements, this solution had some noteworthy limitations. Firstly, the overall system could only support a maximum number of nodes dictated by the capabilities of the data multiplexing hardware structure. Secondly, the physical location of nodes was constrained by the distance specification of the communication solution. Lastly, expanding the system to include new nodes would be a non-trivial task.

 

REPLICA: The Ubiquitous Solution to Sharing Test Data Across Distributed Real-Time Systems

To overcome these limitations for future applications, we created a scalable architecture for low-latency real-time data exchange across distributed testing setups. REPLICA is a revolutionary solution that leverages existing network infrastructure and off-the-shelf hardware, along with a created software stack, to offer local memory access to data from every distributed node. This architecture removes the need for specialized reflective memory cards and improves on state-of-the-art real-time network speed and latency.

By combining the high-level reflective memory concept with RDMA technology, REPLICA delivers a unified view of all distributed test data. It acts as a virtual backplane, connecting heterogeneous systems.

The key benefits of REPLICA include:

  • Ultra-Low Latency – The REPLICA software stack layer offers sub-microsecond latency.

  • Scalability – Add nodes easily by modifying a configuration file. Extensibility is not limited like hardware-based solutions.

  • COTS Hardware – Leverages standard networking equipment. Reduces costs.

  • Low Overhead – Runs independently without burdening CPU resources.

  • Time-Based Synchronization – REPLICA offers Precision Time Protocol (PTP) synchronization capability for real-time loop synchronization.

For test facilities struggling with accelerating amounts of unintegrated data, REPLICA provides the answer. Its universal approach lets teams focus on insights rather than infrastructure.

 

Apply to Participate in the REPLICA Beta Test Program

Think REPLICA is right for your HIL application?  Sign up for the chance to be one of the first beta testers of this innovative data solution.

 

FAQs

How does REPLICA address the challenge of physical location constraints mentioned in the article?
The fact that data exchange happens via a RoCE v2 switch in the converged Ethernet infrastructure removes the usual minimum proximity constraints between nodes imposed by proprietary FPGA-based solutions.  Provided the REPLICA cluster is designed in a manner to guarantee traffic through RoCE v2 multiple-switch hops, the nodes can even be in different buildings as the speed of the light is the data exchange limiting factor.
What specific limitations were encountered with the FPGA-based point-to-point communication protocol mentioned in the article, and how does REPLICA overcome or improve upon them?
FPGA-based solutions for high-speed, real-time networking are usually proprietary.  This limits the interconnection of multiple distributed nodes to only be possible if all systems are provided by the same vendor as the FPGA-based network solution.  Furthermore, physical location of multiple nodes are usually constrained by hardware requirements imposed by the proprietary network.  Having a COTS-based solution like REPLICA not only removes that limitation by allowing multi-vendor, distributed systems to share data within the same real-time network, but it also brings the usual advantages of COTS solutions such as cost-effectiveness, higher scalability, and reduced implementation risk.  Moreover, REPLICA traffic happens within a typical RoCE v2 ethernet infrastructure, which doesn’t impose the same node location constraints typically found with FPGA-based solutions.
Can the REPLICA solution be easily integrated into existing distributed testing setups, and does it support interoperability with various systems and hardware configurations?
Yes. REPLICA can be easily integrated into systems that run real-time Linux operating systems.  The use of NVIDIA NICs (Network Interface Cards) as the hardware front-end allows REPLICA to support any hardware configuration that offers PCIe Gen 3 with a minimum of three lanes. 

 

 

About Genuen

Our goal is to improve time to market without compromising product quality or safety standards. With experience in mission-critical applications and regulatory compliance, Genuen creates custom test systems across the product lifecycle, including hardware-in-the-loop (HIL), fluid power test, and electromechanical test. Headquartered near Kansas City, we have offices across the United States and serve clients in aerospace, transportation, national security, and beyond. The company's Quality Management System (QMS) is certified to ISO 9001.