Testing distributed complex systems requires connecting many different components and subsystems together. As vehicles become more autonomous and incorporate additional sensors, solutions that require distributed processing are becoming more common. Modern model-based systems engineering (MBSE) techniques help accelerate development, but only if test facilities can keep up. The future of transportation demands faster and more connected test solutions.
In this blog post, we will explore Genuen’s solution for low-latency, real-time distributed test execution and how it addresses common challenges in real-time networking.
One of our customers in the hyperloop industry required a distributed real-time solution that involved exchanging data between nodes at a rate of 10 KHz, within a hardware-in-the-loop (HIL) simulation application consisting of 40 nodes. Several nodes were needed to perform real-time calculations on data generated in real-time by other remote nodes in the network, and thus required ultra-low-latency data exchange.
This presented the challenge: how to achieve low-latency, real-time data distribution.
In typical transmission control protocol (TCP) communications, there are several tasks that need to be completed by the CPU in the time that it takes for an application to request data to when it is delivered by the network. This consumes significant processing power in both the transmitting and receiving nodes. Synchronization of distributed nodes usually requires expensive external hardware or is done at a much slower rate via time-based synchronization protocols that can decrease your testing throughput.
While networking solutions exist for sharing test data, these have significant limitations that prevent seamless real-time data distribution across heterogeneous systems.
Reflective memory (RFM) cards provide a shared memory space between remote nodes but, between needed PCI cards and extra software engineering, add substantial implementation costs. These cards store data in memory that is external to the processing CPU. The data transfer needed from the card’s onboard memory to the host computer's memory introduces latency. The ring network architecture commonly used in RFM applications is fragile, as every node becomes a single point of failure.
Other solutions like Time-Sensitive Networking (TSN) focus on precise clock synchronization, not on the mechanics of data exchange. This centralized approach requires the test engineer to implement custom software data exchange messaging layers. Publish-subscribe protocols like Data Distribution Service (DDS) are decentralized, architecture-independent, and use User Datagram Protocol (UDP) for determinism but sacrifice guaranteed package delivery. Their added protocol overhead prohibits their use in applications that require loop rates faster than 1-2 kHz.
Remote Direct Memory Access (RDMA) allows direct read/write access between host memories via Ethernet. With RDMA, networked computers can exchange data without relying on either systems’ processor, cache, or operating system. In this exchange, a network interface card (NIC) does all the heavy lifting that is typically done by the CPU in TCP-based approaches. However, existing drivers only offer point-to-point read/write operations, leaving the test engineer with the heavy task of implementing the multicasting protocol and the data management stack.
Despite advances in real-time networking, existing solutions still demand substantial integration effort or special hardware, and most result in vendor lock-in. To enable the next generation of distributed testing, a new approach is needed – one that delivers a universal memory abstraction across distributed systems in real-time.
We addressed our hyperloop customer’s challenge by implementing a custom FPGA-based point-to-point communication protocol. Despite successfully meeting the application’s requirements, this solution had some noteworthy limitations. Firstly, the overall system could only support a maximum number of nodes dictated by the capabilities of the data multiplexing hardware structure. Secondly, the physical location of nodes was constrained by the distance specification of the communication solution. Lastly, expanding the system to include new nodes would be a non-trivial task.
To overcome these limitations for future applications, we created a scalable architecture for low-latency real-time data exchange across distributed testing setups. REPLICA is a revolutionary solution that leverages existing network infrastructure and off-the-shelf hardware, along with a created software stack, to offer local memory access to data from every distributed node. This architecture removes the need for specialized reflective memory cards and improves on state-of-the-art real-time network speed and latency.
By combining the high-level reflective memory concept with RDMA technology, REPLICA delivers a unified view of all distributed test data. It acts as a virtual backplane, connecting heterogeneous systems.
The key benefits of REPLICA include:
Ultra-Low Latency – The REPLICA software stack layer offers sub-microsecond latency.
Scalability – Add nodes easily by modifying a configuration file. Extensibility is not limited like hardware-based solutions.
COTS Hardware – Leverages standard networking equipment. Reduces costs.
Low Overhead – Runs independently without burdening CPU resources.
Time-Based Synchronization – REPLICA offers Precision Time Protocol (PTP) synchronization capability for real-time loop synchronization.
For test facilities struggling with accelerating amounts of unintegrated data, REPLICA provides the answer. Its universal approach lets teams focus on insights rather than infrastructure.
Think REPLICA is right for your HIL application? Sign up for the chance to be one of the first beta testers of this innovative data solution.