Digital Twin with Xilinx Alveo and Vitis at SPS Drives 2019

https://www.linkedin.com/pulse/digital-twin-xilinx-alveo-vitis-sps-trade-show-giulio-corradi-1f/

After 30 years of SPS IPC Drives, the trade show changed the name into SPS – smart production solutions reflecting the digital transformation that occurs in automation technology.

Digital Twin (DT) in relation to Cyber-Physical-Systems (CPS) are among the main themes of this digital transformation anticipated by the Industry 4.0 initiative.

According to DirectIndustry e-magazine interview to Sylke Schulz-Metzner, Vice President SPS Mesago Messe Frankfurt GmbH, “digital twins is sure to play a large role this year and in the future at SPS…In the future, it will be as difficult to imagine the world without digital twins as it is now to imagine the world without real PLCs or real drives.”.

As the digital twin bridges the physical and the virtual world, where data is transmitted seamlessly allowing the virtual entity representation exists simultaneously with its physical entity. Cyber-Physical-Systems deal on the integration and collaboration of computing, communication, and control, known as the ‘‘3C”. Where communication technology includes operational technology (OT) like field busses and industrial networking as well as information technology (IT) for the enterprise level.

There is no clear boundary between DT and CPS because they require similar infrastructure albeit they have different scope, DT more engineering-focused, CPS more of a discipline.

Xilinx Inc is at the forefront of technology presenting at SPS its demonstrator of Digital Twin and linkage to Cyber-Physical-Systems with its Alveo™boardsand its Vitis™unified software platform, as unique combination of direct Networking access, Modelling Acceleration and High-Performance Control thus able to address the full “3C” requirements as well as the data-hungry need of digital twin avatars.

Xilinx Digital Twin demonstrator presented at SPS 2019 is the result of collaboration with two start-up partners Grovf and Makarenalabs . In the two companies members, most are well below age 30 matching the spirit digital transformation that is bringing new ideas into this long cycle market.

The demonstrator combines 5 technologies:

• Networking for high-performance direct access of data from 100Mbit/s up to 100Gbit/s
• On the fly data filtering, reconciliation and database processing real-time at 12.5Gbyte/s
• Connection and filtering of metadata like the ones from OPC/UA
• Physical engine acceleration for data matching between simulation and physical world
• Game engine linkage of physical simulation for visualization of the virtual world

Some of the software technologies are very advanced paving the way of what will be needed in the near future. Other technologies like physical engine simulators linked to game engines are so new that they never appeared in the field of automation. This transformation is only the beginning of the true CPS era that will allow the virtual representation of the physical body, the machine, connected with the virtual representation of the brain, given by the recent development of A.I.

In Figure 1 a high-level definition of CPS pictures it as a large physical asset of interconnected complex devices and the incarnation of such assets in a virtual world by means of digital twin avatars.

The Digital Twin avatar requires integration with sensory data and information about the physical asset it emulates in the virtual world. Many proponents of DT implicitly assume the digital twin requires new sensors, new data, however, most of the information is already available on the current legacy field bus and industrial Ethernet bus, but it is somewhat filtered or reduced from layer to layer of the automation pyramid as shown in Figure 2. Every higher layer has in general higher bandwidth, but it receives less detailed information as you climb. From a control point of view is good because every piece of information is a valuable aggregation of myriads of information elements that do not need to be exposed. From a Digital Twin, the standpoint is a bit of a challenge taking only the highest level of information, because there is the need of representing the physical level as faithfully as possible and every little detail matter.

Data for digital twin is then readily available if it is intercepted at the proper level without the need of additional sensors. But even if data is intercepted still many challenges are ahead and need to be solved. How to ingest the data, how to reconcile the data, how to take advantage of OPC/UA metadata that is not always readily available, how to execute the digital twin avatar, and where execute its model, how to visualize and optimize it. The number of technologies that must be mastered is large and often intimidating.

In Figure 3 the challenges are displayed as a continuum of data transformation such that the linkage between digital and physical is managed with separation of concerns. The network should have enough bandwidth to process the data and their changes, removing redundancies as the data is transformed. The quality of collected data must be as high as possible mitigating the imprecision and missing values that characterize the real world. The models must have the capability to mimic the real asset in strict real-time. The optimization and solver algorithms must be tightly synchronized with the model and their data such that the very many scenarios required to converge to an optimal model are executed as fast as possible. No matter what strategy is used for the model, analytical based or A.I. or M.L. it requires many runs for converging to the desired accuracy or training the model.

Cloud or on-premise?

Cloud computing is a candidate, but effective digital twin requires the lowest latency in getting factory floor data, as well as a real-time capability at hundreds of microseconds or less to make sure the physical and digital world is always synchronized.

Cloud services are a very good solution in many cases, but not very effective to guarantee real-time response required for machinery digital twin. If you think that a factory floor device sitting in Munich (Germany) pings a cloud service in Frankfurt (Germany) that is around 300Km apart its response time on average can vary from few milliseconds to more than 300ms, clearly unacceptable for the real-time constraints of many machinery applications.

Clearly not all the Digital Twin applications require millisecond timing, but still, such jitter adds uncertainty in processing events and can lead to significant overhead for keeping the systems synchronized. Networking standards like TSN is helping for the next future but today still most cloud network infrastructures do not have provision for it.

Security concerns add another level of criticality to Digital Twin because often factory floor systems are very strategic assets, use legacy protocols not designed for security, and exposing their data to the external world, even with a gateway and firewall that may increase the risk of an information breach.

An on-premise approach is often highly desirable, but it needs the proper network access performances as well as the capability to process the information in real-time.

Demonstrator with Alveo™ U200

In this demonstrator, data have been collected using Xilinx Alveo™ U200 card that host two 100Gb/s interfaces, each comprised of a 4-lane QSFP28, that with a splitter can be transformed into 4x10Gb/s or 4x25Gb/s matching the highest speed today used in the IT and OT networks.

Alveo™ U200 card implements a 16-lane PCI Express® edge connector that performs data transfers at the rate of 2.5 giga-transfers per second (GT/s) for Gen1, 5.0 GT/s for Gen2, and 8.0 GT/s for Gen3 applications. Its memory composed of four independent dual-rank DDR4 interfaces of 16GByte each, provides 64 Gbyte in total with 2400 mega-transfers per second (MT/s), with error-correcting code (ECC) DIMM.

While 10Gb/s and 25Gb/s speed seems at first glance overwhelming because some of the factory floor networks are just today starting to use 1Gb/s data rate, the real digital twin may require information from very many sources that include non-industrial Ethernet sources like video, from subsystems that are on separate subnetwork, from IT Ethernet segments to transfer the virtual data to other part of the enterprise, and finally to get connected to the cloud. Such capacity of the board gives a very long life to the investment if we consider the innovation pace of automation and enough bandwidth to be the centerpiece of the system instead of the bottleneck.

Field busses like EtherCAT® is also announced by its inventor Beckhoff to transition toward 1Gb/s with its EtherCAT G and 10Gb/s with its EtherCAT G10, it will certainly happen thus the Alveo™ boards can be used not just to acquire data but as EtherCAT master for managing the traffic when the transition to 10G will be fully completed.

Demonstrator – Networking Data Parsing

On the Alveo™, each network connector is housed within a single QSFP cage assembly located at the I/O bracket. equipped with a GROVF IP core capable of acquiring data at 100Gb/s and processing the incoming stream using a regular expression. Regular expressions (regex) are specially encoded text strings used as patterns for matching sets of strings. GRegeX is an implementation by GROVF of a standard regular expression algorithm on FPGA chip achieving 12.8 GByte/s throughput with a single IP core. A wide range of supported regular expression functions allows the application of desired rules which can be handled in a chip without reducing the throughput neither involving the CPU. In this way, the data collected from the network are filtered and sorter independently from their constituent part because the regex rules can be made specifically for the specific protocol allowing the degree of independence from the underlining application protocol. EtherCAT, ProfiNET, PowerLink and other industrial network protocol L1 and L2 frames and data can be captured based on regular expression patterns, as well as the IT ethernet packets allowing the full exploitation of the available data in the factory.

Demonstrator – Creating the field information database

Information captured from the network can be formatted as JSON like structures and stored into a non-SQL database like MongoDB®, thanks to the large footprint of the Alveo™ U200board memory (64GBytes) thus that in-memory operations are possible.

Demonstrator – Data filtering and query with MongoDB®

MongoDB® is a document database defined as No-SQL type that stores data in JSON-like structures. Field, range query, and regular expression searches are part of MongoDB that scales acceleration is based on Grovf’s MonetX acceleration platform, which is an FPGA based smart memory extension for near memory data processing. The operating system recognizes MonetX as a standard memory extension which also provides high-performance computing cores API for the host layer. Data can be stored to MonetX memory extension, just like into any other memory connected to the server. Applications than can initiate different processing on the data stored into the MonetX platform directly running on FPGA, where data resides also. MonetX supports many high-performance computing cores such as Regular Expression processing, Search/Sort processing, Data compression/decompression, Statistical Data processing algorithms, etc. MongoDB performance has been boosted 3.5X only using the MonetX acceleration platform as a high bandwidth memory extension for standard server architecture without using any build-in high-performance computing cores in the FPGA. This leads to zero code change in the application (MongoDB) side and provides 3.5X acceleration. More acceleration for the MongoDB and any other application can be achieved using build-in accelerated computing cores in the FPGA residing near memory.

Artavazd Khachatryan commenting the demo said “What we realized is not a demo it is a very efficient IP that occupies about 1/3 of the Xilinx Alveo U200 FPGA card resources making it available for further scaling and achieving greater throughput to support more than one 100G network interfaces with the single chip. GRegeX achieves 12.8 GB/s throughput regardless of the regular expression rule set while software implementation speed decreases when using more complex regex rules such as brackets and repeat symbols“

Demonstrator – The Virtual Model Avatar with Bullet and Unity™

Besides data, the virtual representation requires visualization and physical simulation to be a faithful replica of the physical asset. As much real-time is added to the simulation as close will be virtual to the physical twin behavior.

Visualization and physical simulation are the two other components that play an important role in the virtual twin. Game engine software is often the platform of choice for managing the visual aspect of the virtual twin with as much realism as possible.

For example, Unity™  a game engine developed by Unity Technologies has been adopted by industries outside video gaming, such as film, automotive, architecture, engineering, and construction. The Digital Twin requires visualization of the physical asset thus Unity has been used in combination with Alveo™ for the Xilinx demonstrator for rendering the virtual reality world associated with the digital representation of machinery.

Physical engine software is needed for simulating virtually the real world. Bullet is an open-source physics engine developed by Erwin Coumans to simulate collision detection, soft and rigid body dynamics. Bullet has been used in video games as well as for visual effects in movies and recently used in several projects in the context of Digital Twins for Industry 4.0.

Demonstrator – Linkage with the Digital Twin data acceleration

Xilinx Vitis™ framework has been used to connect the Unity™ and Bullet frameworks to the field data, using Alveo™ as the main accelerator and data-sharing system, requiring the minimum intervention of the host server. The host has been implemented with a Gaming Dell G7 PC just for the rendering, but the primary binding with the virtual avatar is fully executed on the Alveo board.

The Vitis™ unified software platform enables the development of embedded software and accelerated applications on heterogeneous Xilinx platforms including FPGAs, SoCs, and Versal ACAPs.  It provides a unified programming model for accelerating Edge, Cloud, and Hybrid computing applications. It leverages integration with high-level frameworks, develops in C, C++, or Python using accelerated libraries or use RTL-based accelerators & low-level runtime APIs for more fine-grained control over implementation.

In this demonstration, the partner MakarenaLabs created the infrastructure to connect Bullet, Unity, and Alveo using Vitis showing how a soft body, a spring-mass system and a cinematic system can be combined accelerating the execution of their model. 

Demonstrator – Design Productivity with Vitis™

For interacting with the audience, the demonstrator implemented the acceleration in a game where the user can control various parameters like the tension in the spring launching a rigid object to the soft body. A team of 5 engineers worked for assembly the technologies together such that the virtual to physical interaction is possible, the team has been developing the application using Vitis™ and the XRT API.

According to Enrico Giordano (CEO of MakarenaLabs) “Vitis™ has been instrumental for its team with few FPGA engineers to take advantage of the Alveo boards and its acceleration using high-level software Python and C++, without the details of any RTL code.  Many of our software engineers, data scientists, or computer game engineers do not have deep FPGA hardware design experience, thus the level of abstraction provided by Vitis made the endeavor possible “. 

Guglielmo Zanni the Bullet expert said “I am expert in robotics and physical simulation, I started our design using SDx and I needed API consistent with a high-level view of the underlining hardware, Vitis is bringing the level of abstraction and interoperability I was looking for”

Irina Vranescu for the cloud computing and Brixihlida Koçi for the embedded and A.I development said: “With Vitis, we see a more pragmatic approach to compulsory libraries that allows us to rely more on Xilinx for many of the algorithms we need like BLAS, or Solvers, and it gives us the productivity and consistency of a modern development environment without recreating code from scratch.”

Camilla Corradi the Unity expert and computer game engineer said: “I am developing with the most visual and complex part and I need stable interfaces to pass information in real-time to the accelerators, Vitis introduced the unified API that allows me to interoperate with my other colleagues with a clear development contract paradigm.”

Demonstration – The full system 

The full system used the whole technologies available to showcase the real connections and the real traffic delivered by field devices implemented with the Xilinx Cora board running Xilinx PYNQ framework and generating the synthetic data as coming from Drives, PLC and OPC/UA clients and servers. The Xilinx Ultra96 board used to generate traffic from a wireless connection linked to the system with an access point to simulate an even greater CPS system that includes remote and difficult to reach devices.

Alveo™ U200 board has been hosted into an  Akitio®Node Pro that provides a PCIe slot for Thunderbolt 3 computers, in our case Dell G7 laptop computers, such that is possible to handle all the acceleration with minimum impact on the server. The Thunderbolt 3 connection allows to take the device and move it from one system to another. Thunderbolt™ 3 for transfer speeds up to 40 Gbps allowed enough bandwidth to manage all 4 channels of 40Gb/s and the acceleration of Bullet in the Alveo board.

Summary

SPS 2019 has been an exceptional playing field to demonstrate the newest Xilinx technologies that are beneficial to Digital Twin and Cyber-Physical-Systems showcasing high-performance networking tuned to the next development of Industry 4.0. The linkage between the virtual world and the physical world made possible combining technologies like Gaming and Physic Simulation with real-time data was unthinkable a few years ago, but today made possible with accelerators directly connected to a network like the Alveo™ family. Xilinx Vitis made also possible leverage the expertise of non-FPGA developers and engineers providing a familiar Python and C++ framework with the unified XRT API.

All in all, the Digital Twin endeavor is just started, and we will expand in such direction providing even more ability to create actionable virtual-physical realizations.