Run RF-DETR in NVIDIA DeepStream on Jetson

This post shows how to run RF-DETR inside NVIDIA DeepStream on a Jetson Orin NX. We cover what DeepStream and RF-DETR are, how to build a TensorRT engine and the parser DeepStream needs to read the model's output, how to run it on RTSP stream, and how to give each class its own box color.

We used NVIDIA's deepstream-import-vision-model agent skill to do most of the import for us. The numbers below come from JetPack 7.2 with DeepStream 9.1 and TensorRT 10.16.

What Is DeepStream?

DeepStream is NVIDIA's video analytics SDK, built on GStreamer. It gives you hardware-accelerated pipeline parts (decode, batching, TensorRT inference, on-screen display, encode) that keep frames in GPU memory the whole way through. The ones you will touch in this post:

nvstreammux: batches N decoded streams into one GPU buffer

nvinfer: runs a TensorRT engine and attaches detection metadata to each frame

nvdsosd: draws boxes and labels from that metadata

On a Jetson it installs straight from NVIDIA's apt repo: sudo apt install deepstream-9.1 on JetPack 7.2 (or deepstream-7.1 on JetPack 6.2).

Keep in mind the inference engine doesn't know or care what your model's output format is; it hands you raw tensors and leaves the interpretation to you. That interpretation lives in a small parser function you write, and it's where most of these projects succeed or fail.

What Is RF-DETR?

RF-DETR ( GitHub ) by Roboflow is a DETR-family detector, released under Apache 2.0. It uses 300 learned queries that look over the image features, and each query predicts one box on its own. So there is no anchor grid and no NMS step afterward. It ships in a few sizes; here I'll use RF-DETR Nano, which takes a 384x384 RGB image and puts out two tensors: 300 boxes (center x, center y, width, height, all scaled 0 to 1) and 300 x 91 class scores run through a sigmoid.

Getting the Weights and ONNX

The rfdetr package downloads pretrained COCO weights the first time you use it, and exports a ready-to-deploy ONNX…