Performance through TensorRT

A client doing R&D for ADAS had developed a set of neural networks in TensorFlow and PyTorch that they wanted to deploy efficiently into a C++ environment for inferencing on the edge.

Software Supreme was tasked with the integration of two networks – using TensorFlow and LibTorch’s C++ bindings. This solution worked, however both frameworks were fighting over the GPU resources, which led to poor device utilization, so we decided to switch to native TensorRT. To achieve that we implement a few custom layers as TensorRT plugins.. With TensorRT we had full control over the NN execution. We were able to address the initial bottleneck and further increase performance through the custom GPU accelerated layers.

Both networks were integrated and performance exceeded the target specs. One of the networks’ performance increased from 50fps up to 140fps. This allowed both networks to run in parallel each frame, as opposed to running sequentially.